How to correctly aggregate metrics when dropping host tag?

[syedg1]

Question

I am currently building a Datadog Agent -> Vector -> Datadog API pipeline to drop the host tag before it is sent to Datadog. The motivation behind this is to reduce cardinality and thereby costs. I understand that dropping the host tag imposes the risk of data point duplication in Datadog, which would result in duplicate points of the same time series overwriting original points.

In order to prevent this issue, I am using an aggregation transform to consolidate data points within Vector after the host tag has been dropped. This seems like it should work without incurring any data loss, but the filtered/aggregated metrics are consistently lower than the original metrics with the host tag. I have attached a screenshot of the test metrics.

blue: original metrics, purple: removing host with aggregation, yellow: removing host without aggregation

I would appreciate any guidance on how to aggregate these metrics such that they are accurate despite removing the host tag. Thank you!

Vector Config

# Sources - receive data from Datadog agents
    sources:
      datadog_agents:
        type: datadog_agent
        address: 0.0.0.0:8282
        multiple_outputs: true
        store_api_key: false
        
 # Transforms - delete unnecessary host tags (cardinality reduction)
    transforms:
      remove_host_tag:
        type: remap
        inputs:
          - datadog_agents.metrics
        source: |-
          # Add missing cluster tags using the same logic as datadog-agent.yaml lines 72-83
          # These should come from DD_TAGS but don't due to agent bug: https://github.com/DataDog/datadog-agent/issues/15128

          log("RAW DATA RECEIVED: " + encode_json(.))

          cluster_env = "${CLUSTER_APP_ENVIRONMENT}"
          cluster_group = "${CLUSTER_APP_GROUP}"
          cluster_name = "${CLUSTER_NAME}"
          cluster_region = "${CLUSTER_REGION}"

          if cluster_env != "" {
            .tags.env = cluster_env
          }
          if cluster_group != "" {
            .tags.app_group = cluster_group
          }
          if cluster_name != "" {
            .tags.eks_cluster = cluster_name
            .tags.cluster_name = cluster_name
            .tags.cluster_id = cluster_name
          }
          if cluster_region != "" {
            .tags.region = cluster_region
          }

          if (exists(.tags.host) && exists(.tags.remove_host_tag)) {
            host_value = .tags.host
            del(.tags.host)
            # If the metric is not a distribution, we need to set a tag for uniqueness. Use the vector host, which should have lower cardinality than the datadog host.
            if .kind != "distribution" {
              .tags.vector_host = "${HOSTNAME}"
            }
          }

          # log("UPDATED TAGS: " + encode_json(.tags))

      consolidate_metrics:
          type: aggregate
          inputs:
            - remove_host_tag

# Sinks - forward processed data to Datadog
    sinks:
      datadog_metrics:
        type: datadog_metrics
        inputs:
          - consolidate_metrics
        default_api_key: "${DATADOG_API_KEY}"
        site: "us5.datadoghq.com"

[pront]

Hi @syedg1, you are looking at requests count. This came up here before: #23675 (comment)

The datadog_metrics sink is batching events into one request hence less requests overall.

[jbialy]

@syedg1 I'm wondering if you had any luck with solving your original ask around data point uniqueness. We are experiencing the same problem. We drop tags such as the host, pod uuid, etc to reduce cardinality but by doing so are experiencing data loss on the Datadog side (data points being overwritten). There's an old issue that explains this in great detail: #18123.

[syedg1]

There seems to be a bug in Vector where rate counters are being sent as counters by the Datadog sink. I have opened an issue with them containing more details #23762.

While searching for workarounds, I found some logic in the Datadog sink which handles the aggregation of counter metrics that share the same series and timestamp. Further details can be found in this #13960 and v.0.25.0 Release Logs

The datadog_metrics sink now correctly aggregates counters emitted for the same timeseries within a single second (the timestamp granularity). Previously, it sent them through as-is to Datadog which processed them as last-write-wins.

Based on this, we skipped the aggregation transform and let the Datadog sink within Vector handle the aggregation accordingly. This ensured that the host tag removed metrics accurately represent the original metric and rollup correctly when viewing the metric across different time frames.