Add EC2 Instance Metadata to metrics sent to CW. (Custom Dimensions)

.gitignore

@@ -1,3 +1,5 @@
 .coverage
 *.pyc
 .DS_Store
+.vscode*
+.vscode/*

README.md

@@ -28,9 +28,10 @@ The default location of the configuration file used by collectd-cloudwatch plugi
  * __flush_interval_in_seconds__ - The flush_interval_in_seconds is used for flush interval, it means how long plugin should flush the metrics to Cloudwatch
  * __whitelist_pass_through__ - Used to enable potentially unsafe regular expressions. By default regex such as a line containing `.*` or `.+` only is automatically disabled in the whitelist configuration.
   Setting this value to True may result in a large number of metrics being published. Before changing this parameter, read [pricing information](https://aws.amazon.com/cloudwatch/pricing/) to understand how to estimate your bill.
- * __push_asg__ - Used to include the Auto-Scaling Group as a dimension for all metrics (see `Adding additional dimensions to metrics` below for details)
- * __push_constant__ - Used to include a Fixed dimension (see `constant_dimension_value` below) on all metrics. Useful for collating all metrics of a certain type (see `Adding additional dimensions to metrics` below for details)
- * __constant_dimension_value__ - Used to specify the value for the Fixed dimension (see `Adding additional dimensions to metrics` below for details)
+ * __dimensions_path__ - Path to file that contains custom dimension definition. (see `Adding EC2 metadata dimensions to metrics` below for details)
+ * __push_asg__ - Used to include the Auto-Scaling Group as a dimension for all metrics (see `Adding simple dimensions to metrics` below for details)
+ * __push_constant__ - Used to include a Fixed dimension (see `constant_dimension_value` below) on all metrics. Useful for collating all metrics of a certain type (see `Adding simple dimensions to metrics` below for details)
+ * __constant_dimension_value__ - Used to specify the value for the Fixed dimension (see `Adding simple dimensions to metrics` below for details)
  * __debug__ - Provides verbose logging of metrics emitted to CloudWatch
 
 #### Example configuration file
@@ -41,6 +42,7 @@ host = "Server1"
 proxy_server_name = "http://myproxyserver.com"
 proxy_server_port = "8080"
 whitelist_pass_through = False
+dimensions_path = "/opt/collectd-plugins/cloudwatch/config/dimensions"
 push_asg = False
 push_constant = True
 constant_dimension_value = "ALL"
@@ -50,7 +52,7 @@ flush_interval_in_seconds = 60
 ```
 
 
-##### Adding additional dimensions to metrics
+##### Adding simple dimensions to metrics
 We support adding both the ASG name to dimensions, as well as a "fixed dimension". Fixed dimensions are an additional value that will be added all metrics.
 
 ###### Example configuration file
@@ -70,6 +72,26 @@ The above configuration will result in all metrics being pushed with "FixedDimen
 
 The above configuration will push the AutoScaling Group name for metrics as well
 
+###### Adding EC2 metadata dimensions to metrics
+User can specify in a dimension file the instance metadata that he/she wants to be pushed to aws cloudwatch along with the metric information. For example, a user can specify region, availability-zone, private-ip, instanceid, and more in the dimension file. In effect those attributes will be pushed along with the metric data to aws cloudwatch giving more clarity and information to the user about the particular metric(s).
+
+Usage:
+Create dimensions file in whatever location you want.
+Recommended Path: /opt/collectd-plugins/cloudwatch/config/dimensions
+
+Sample Dimensions File:
+```
+Host
+PrivateIp
+InstanceId
+Region
+```
+After setup.py is ran there will be a dimensions file located at the dimensions_path file location. By default, dimensions will be Host and PluginInstance. Any key within the [instance dimension document](http://169.254.169.254/latest/dynamic/instance-identity/document) is supported however the first letter of the key will have to be captilized and there should be no spaces at the end of the key.
+
+For example, the key in the identity document the key `imageId` would become `ImageId` in the dimensions file.
+
+If you update a dimension in the dimensions file collectd will need to be restarted. Setup.py must be ran in interactive mode.
+
 ### AWS account configuration
 The account configuration is optional for EC2 instances with IAM Role attached. By default the AWS account configuration file is expected to be stored in: `/opt/collectd-plugins/cloudwatch/config/.aws/credentials`.
 The following parameters can be configured in the above file:

src/cloudwatch/config/plugin.conf

@@ -16,6 +16,9 @@ whitelist_pass_through = False
 # The debug parameter enables verbose logging of published metrics
 debug = False
 
+# The path to the Dimensions file. This path has to be provided if you want extra metadata sent along with metric data to cloudwatch
+#dimensions_path = "/opt/collectd-plugins/cloudwatch/config/dimensions"
+
 # Wheter or not to push the ASG as part of the dimension.
 # WARNING: ENABLING THIS WILL LEAD TO CREATING A LARGE NUMBER OF METRICS.
 push_asg = False

src/cloudwatch/modules/configuration/confighelper.py

@@ -3,6 +3,7 @@
 from configreader import ConfigReader
 from metadatareader import MetadataReader
 from credentialsreader import CredentialsReader
+from dimensionsreader import DimensionsReader
 from whitelist import Whitelist, WhitelistConfigReader
 from ..client.ec2getclient import EC2GetClient
 import traceback
@@ -27,6 +28,7 @@ class ConfigHelper(object):
     _DEFAULT_AGENT_ROOT_FOLDER = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir, './config/') # '/opt/AmazonCloudWatchAgent/'
     _DEFAULT_CONFIG_PATH = _DEFAULT_AGENT_ROOT_FOLDER + 'plugin.conf'
     _DEFAULT_CREDENTIALS_PATH = _DEFAULT_AGENT_ROOT_FOLDER + ".aws/credentials"
+    _DEFAULT_DIMENSIONS_PATH = _DEFAULT_AGENT_ROOT_FOLDER + 'dimensions'
     _METADATA_SERVICE_ADDRESS = 'http://169.254.169.254/' 
     WHITELIST_CONFIG_PATH = _DEFAULT_AGENT_ROOT_FOLDER + 'whitelist.conf'
     BLOCKED_METRIC_PATH = _DEFAULT_AGENT_ROOT_FOLDER + 'blocked_metrics'
@@ -45,6 +47,7 @@ def __init__(self, config_path=_DEFAULT_CONFIG_PATH, metadata_server=_METADATA_S
         self.debug = False
         self.pass_through = False
         self.push_asg = False
+        self.dimensions = []
         self.push_constant = False
         self.constant_dimension_value = ''
         self.enable_high_resolution_metrics = False
@@ -74,7 +77,9 @@ def _load_configuration(self):
         self.config_reader = ConfigReader(self._config_path)
         self.credentials_reader = CredentialsReader(self._get_credentials_path())
         self.metadata_reader = MetadataReader(self._metadata_server)
+        self.dimensions_reader = DimensionsReader(self._get_dimensions_path())
         self._load_credentials()
+        self._load_dimensions()
         self._load_region()
         self._load_hostname()
         self._load_proxy_server_name()
@@ -110,6 +115,23 @@ def _load_credentials(self):
     def _get_credentials_from_iam_role(self):
         """ Queries IAM Role metadata for latest credentials """
         return self.metadata_reader.get_iam_role_credentials(self.metadata_reader.get_iam_role_name())
+
+    def _get_dimensions_path(self):
+        dimensions_path = self.config_reader.dimensions_path
+        self._LOGGER.info("Dimensions Path: " + str(dimensions_path))
+        if not self.config_reader.dimensions_path:
+            dimensions_path = self._DEFAULT_DIMENSIONS_PATH
+        return dimensions_path
+
+    def _load_dimensions(self):
+        """
+        Tries to load dimensions based on the path to the file given in the plugin configuration file. If such file does not exist
+        or does not contain uncommented dimensions, then default dimensions are used.
+        """
+        self.dimensions = self.dimensions_reader.dimensions
+        if not self.dimensions:
+            self.dimensions = None
+            self._LOGGER.info("Dimensions set to None")
 
     def _load_region(self):
         """

src/cloudwatch/modules/configuration/configreader.py

@@ -33,6 +33,7 @@ class ConfigReader(object):
     DEBUG_CONFIG_KEY = "debug"
     PASS_THROUGH_CONFIG_KEY = "whitelist_pass_through"
     PUSH_ASG_KEY = "push_asg"
+    DIMENSIONS_PATH_KEY = "dimensions_path"
     PUSH_CONSTANT_KEY = "push_constant"
     CONSTANT_DIMENSION_KEY = "constant_dimension_value"
     PROXY_SERVER_NAME_KEY = "proxy_server_name"
@@ -48,6 +49,7 @@ def __init__(self, config_path):
         self.pass_through = self._PASS_THROUGH_DEFAULT_VALUE
         self.debug = self._DEBUG_DEFAULT_VALUE
         self.push_asg = self._PUSH_ASG_DEFAULT_VALUE
+        self.dimensions_path = ""
         self.push_constant = self._PUSH_CONSTANT_DEFAULT_VALUE
         self.constant_dimension_value = ''
         self.proxy_server_name=''
@@ -76,5 +78,6 @@ def _parse_config_file(self):
         self.pass_through = self.reader_utils.try_get_boolean(self.PASS_THROUGH_CONFIG_KEY, self._PASS_THROUGH_DEFAULT_VALUE)
         self.debug = self.reader_utils.try_get_boolean(self.DEBUG_CONFIG_KEY, self._DEBUG_DEFAULT_VALUE)
         self.push_asg = self.reader_utils.try_get_boolean(self.PUSH_ASG_KEY, self._PUSH_ASG_DEFAULT_VALUE)
+        self.dimensions_path = self.reader_utils.get_string(self.DIMENSIONS_PATH_KEY)
         self.push_constant = self.reader_utils.try_get_boolean(self.PUSH_CONSTANT_KEY, self._PUSH_CONSTANT_DEFAULT_VALUE)
         self.constant_dimension_value = self.reader_utils.get_string(self.CONSTANT_DIMENSION_KEY)

src/cloudwatch/modules/configuration/dimensionsreader.py

@@ -0,0 +1,41 @@
+from readerutils import ReaderUtils
+from ..logger.logger import get_logger
+
+
+class DimensionsReader(object):
+    """
+    The dimensions file reader class that is responsible for reading and parsing file containing AWS dimensions.
+
+    The credentials file is a simple text file in format:
+    dimension1
+    dimension2
+
+    Keyword arguments:
+    dimensions_path -- the path for the credentials file to be parsed (Required)
+    """
+
+    _LOGGER = get_logger(__name__)
+
+    def __init__(self, dimensions_path):
+        self.dimensions_path = dimensions_path
+        self.dimensions = None
+        try:
+            self.reader_utils = ReaderUtils(dimensions_path)
+            self._parse_dimensions_file()
+        except Exception as e:
+            self._LOGGER.warning("Cannot read AWS dimensions from file. Defaulting to default dimensions.")
+
+    def _parse_dimensions_file(self):
+        """ 
+        This method retrieves values form preprocessed configuration list 
+        in format: 
+        value 
+        value2 
+        """
+        dimensions_list = self.reader_utils.get_dimensions()
+        if not dimensions_list:
+            self._LOGGER.warning("Cannot read AWS dimensions from file. Defaulting to default dimensions.")
+        self.dimensions = dimensions_list
+
+class DimensionsReaderException(Exception):
+    pass

src/cloudwatch/modules/configuration/readerutils.py

@@ -9,6 +9,7 @@ class ReaderUtils(object):
     _LOGGER = get_logger(__name__)
     _COMMENT_CHARACTER = '#'
     _AWS_PROFILE_PATTERN = re.compile("^\s*\[[\w]+\]\s*$")
+    _NONDIMENSIONS_PATTERN = re.compile("^\s")
 
     def __init__(self, path):
         self.path = path
@@ -17,6 +18,9 @@ def __init__(self, path):
 
     def get_string(self, key):
         return self._find_value_by_key(key)
+
+    def get_dimensions(self):
+        return self._find_dimensions()
 
     def get_boolean(self, key):
         value = self._find_value_by_key(key)
@@ -35,7 +39,7 @@ def try_get_boolean(self, key, default_value):
     def _find_value_by_key(self, key):
         config_list = self._load_config_as_list(self.path)
         for entry in config_list: 
-            if not entry or entry[0] == self._COMMENT_CHARACTER or self._AWS_PROFILE_PATTERN.match(entry):
+            if not entry or entry[0] == (self._COMMENT_CHARACTER or self._AWS_PROFILE_PATTERN.match(entry)):
                 continue  # skip empty and commented lines
             try: 
                 entry_key, entry_value = entry.split('=', 1)
@@ -46,6 +50,19 @@ def _find_value_by_key(self, key):
                 self._LOGGER.error("Cannot read configuration entry: " + str(entry))
                 raise ValueError("Invalid syntax for entry '" + entry + "'.")
         return ""
+
+    def _find_dimensions(self):
+        config_list = self._load_config_as_list(self.path)
+        dimensions_list = []
+        for entry in config_list:
+            if not entry or entry[0] == self._COMMENT_CHARACTER or self._NONDIMENSIONS_PATTERN.match(entry):
+                continue # skip empty and commented lines
+            try:
+                dimensions_list.append(entry)
+            except:
+                self._LOGGER.error("Cannot read configuration entry: " + str(entry))
+                raise ValueError("Invalid syntax for dimension entry '" + entry + "'.")
+        return dimensions_list
 
     def _strip_quotes(self, string):
         return re.sub(r"^'|'$|^\"|\"$", '', string)

src/cloudwatch/modules/metricdata.py

@@ -1,6 +1,9 @@
 import awsutils as awsutils
 import plugininfo
 import datetime
+import json 
+import urllib2
+from logger.logger import get_logger
 
 class MetricDataStatistic(object):
     """
@@ -78,6 +81,7 @@ class MetricDataBuilder(object):
     vl -- The Collectd ValueList object with metric information
     adjusted_time - The adjusted_time is the time adjusted according to storage resolution
     """
+    _LOGGER = get_logger(__name__)
 
     def __init__(self, config_helper, vl, adjusted_time=None):
         self.config = config_helper
@@ -123,14 +127,29 @@ def _build_constant_dimension(self):
         return dimensions
 
     def _build_metric_dimensions(self):
-        dimensions = {
-              "Host" : self._get_host_dimension(),
-              "PluginInstance" : self._get_plugin_instance_dimension()
-              }
-        if self.config.push_asg:
-            dimensions["AutoScalingGroup"] = self._get_autoscaling_group()
-        if self.config.push_constant:
-            dimensions["FixedDimension"] = self.config.constant_dimension_value
+        dimensions = {}
+        metadata = self._get_metadata()
+        if self.config.dimensions:
+            for dim in self.config.dimensions:
+                if dim.lower() == 'host':
+                    dimensions["Host"] = self._get_host_dimension()
+                elif dim.lower() == 'plugininstance':
+                    dimensions[dim] = self._get_plugin_instance_dimension()
+                else:
+                    dimensions[dim] = str(metadata.get(dim[:1].lower() + dim[1:]))
+            if self.config.push_asg:
+                dimensions["AutoScalingGroup"] = self._get_autoscaling_group()
+            if self.config.push_constant:
+                dimensions["FixedDimension"] = self.config.constant_dimension_value
+        else:
+            dimensions = {
+                    "Host" : self._get_host_dimension(),
+                    "PluginInstance" : self._get_plugin_instance_dimension(),
+                    }
+            if self.config.push_asg:
+                dimensions["AutoScalingGroup"] = self._get_autoscaling_group()
+            if self.config.push_constant:
+                dimensions["FixedDimension"] = self.config.constant_dimension_value
         return dimensions
 
     def _get_plugin_instance_dimension(self):
@@ -147,3 +166,25 @@ def _get_autoscaling_group(self):
         if self.config.asg_name:
             return self.config.asg_name
         return "NONE"
+
+    def _get_metadata(self):
+        ## regarding commandlist below:
+            ## http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html
+	        ## Due to the dynamic nature of instance identity documents and signatures,
+        ## we recommend retrieving the instance identity document and signature regularly.
+        dimensionsList = []
+        url = "http://169.254.169.254/latest/dynamic/instance-identity/document"
+
+        req = urllib2.Request(url)
+        try:
+            response = urllib2.urlopen(req)
+        except urllib2.URLError as e:
+            if hasattr(e, 'reason'):
+                self._LOGGER.error("We failed to reach a server. Reason: " + e.reason)
+            elif hasattr(e, 'code'):
+                self._LOGGER.error("The server couldn't fulfill the request. Error code: " + e.code)
+        for line in response:
+            dimensionsList.append(line.rstrip())
+        output = "".join(dimensionsList)
+        instanceDimensions = json.loads(output)
+        return instanceDimensions