Skip to content

README.md

Latest commit

 

History

History
64 lines (44 loc) · 1.86 KB

README.md

File metadata and controls

64 lines (44 loc) · 1.86 KB

CloudWatchMetrics

Facilities for automatically sending Hadoop metrics to Amazon CloudWatch, using the metrics and metrics2 interfaces. There are 2 different metrics handlers:

  • CloudWatchContext: send metrics using the old metrics interface.
  • CloudWatchSink: send metrics using the new metrics2 interface.

Once sent, metrics will be available in the CloudWatch UI after a few minutes.

Dependencies

  • AWS SDK >= 1.5.3
  • Guava >= 14.0.1
  • CDH 4.3.1

Options

The following options are available in both interfaces:

  • accesskey: Your personal AWS access key.
  • secretkey: Your personal AWS secret key.
  • region: The AWS region ID.
  • namespace: The CloudWatch namespace. Defaults to "Custom".
  • batch: The maximum number of metrics to send in a single AWS request. Defaults to 5.

Setup

To package the classes and dependencies into a single jar (except the Hadoop classes), run:

mvn package

Update your hadoop-env.sh file and include the following:

export HADOOP_CLASSPATH="/path/to/cloudwatch-metrics-[version].jar"

Restart your services for this change to take effect.

CloudWatchContext

To enable metrics, update hadoop-metrics.properties to use CloudWatchContext, for example:

namenode.class=org.apache.hadoop.metrics.cloudwatch.CloudWatchContext
namenode.period=10
namenode.accesskey=[accessKey]
namenode.secretkey=[secretKey]
namenode.region=us-east-1

CloudWatchSink

To enable metrics, update hadoop-metrics2.properties to use CloudWatchSink, for example:

*.sink.cloudwatch.class=org.apache.hadoop.metrics2.cloudwatch.CloudWatchSink
*.period=10
namenode.sink.cloudwatch.accesskey=[accessKey]
namenode.sink.cloudwatch.secretkey=[secretKey]
namenode.sink.cloudwatch.region=us-east-1
namenode.sink.cloudwatch.namespace=Example