Aws lambda emr

json –bucket event-driven-batch-analytics. New Big Data Architect With Aws Lambda, Emr careers are added daily on SimplyHired. For instance, Hadoop’s core services haven’t gone away but rather have been packaged into services like EMR, Qubole, and Altiscale to decouple running a cluster from … 続きを読む

CloudWatch Logs に蓄積したログをS3に定期的にエクスポートする

‘use strict’; const aws = require(‘aws-sdk’); aws.config.update({region: ‘ap-northeast-1’}); exports.handler = (event, context, callback) => { const s3 = new aws.S3(); const cloudwatchlogs = new aws.CloudWatchLogs(); var getToTS = function() { var now = new Date(); now.setHours(0); now.setMinutes(0); … 続きを読む

EMRからS3にアクセス時のAPI発行回数を調べる

aws cloudtrail create-trail –name az-s3-event-selector –s3-bucket-name az-read-only-events { “IncludeGlobalServiceEvents”: true, “Name”: “az-s3-event-selector”, “TrailARN”: “arn:aws:cloudtrail:ap-northeast-1:********:trail/az-s3-event-selector”, “LogFileValidationEnabled”: false, “IsMultiRegionTrail”: … 続きを読む

[JAWSBigData#11]Cloudera on AWSと Amazon EMRを両方本番運用し 3つの観点から比較してみる …

BigData-JAWS 勉強会#11 発表資料https://jawsug-bigdata.connpass.com/event/77463/ ○ 概要Cloudera on AWSとして、Cloudera社の代表的ツールClouderaDirector/(ClouderaAltus)と、AmazonEMRの特徴を紹介します。Cloudera on AWS/AmazonEMR両方を本番環境で運用し、そこでのアーキテクチャ/エコ … 続きを読む