Vector Aggregator Discarding Metrics

[syedg1]

Question

We have a DD Agent -> Vector Aggregator -> Datadog metrics pipeline setup. However, we have noticed that there are a lot of discarded events. It looks like all discarded events are coming from component_id: filter_histogram_metrics, component_type: filter.

We have tried tweaking the buffer.max_events, batch.max_events, and request.adaptive_concurrency parameters to see if that would help reduce discarded events, but no luck. Currently, we have only enabled vector as passthrough, as in metrics go through the remap transform, but don't satisfy the condition that removes the tags (see comments in vector config remap transform). This means that we haven't started removing tags yet, however, teams have still been reporting a loss of metrics triggering false alerts.

Can you please assist us in preventing metrics from being dropped?

Vector Discarded Events By Component ID & Component Type

Vector Received Events vs Discarded Events

Vector Config

# Configuration for the vector chart dependency
# All values under this key get passed to the vector chart v0.45.0
vector:
  # Vector configuration for Datadog agents -> Vector -> Datadog pipeline
  # For available options, see: https://github.com/vectordotdev/helm-charts/tree/develop/charts/vector

  logLevel: info

  # Using stateless aggregator to prioritize high availability (faster pod restarts) in tradeoff for some data loss, which would be minimal
  role: Stateless-Aggregator

  # Enable horizontal pod autoscaling with a minimum of 3 replicas to spread across our AZs
  # Overwriting subchart defaults so we have better control over the HPA
  # By default, HPA scale up stabilization window is 0 seconds and scale down stabilization window is 300 seconds
  # https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 10
    targetCPUUtilizationPercentage: 60

  # Tolerate public-subnets nodes
  tolerations:
    - key: nd.group-name
      effect: NoExecute
      value: public-subnets
      operator: Equal

  # Select public-subnets nodes
  nodeSelector:
    nextdoor.com/type: public
    kubernetes.io/os: linux

  # Spread pods across our AZs
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      minDomains: 3  # This ensures that we have public-subnet nodes in each of our 3 AZs
      whenUnsatisfiable: DoNotSchedule  # This forces Spot to spin up nodes in new zones and spread pods evenly across our AZs
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: vector

  # Setting resources for the vector pods
  # The CPU limit is not set to allow the container to use more CPU resources if needed
  # The pods will be in Burstable QOS
  resources:
    requests:
      cpu: 3
      memory: 4Gi
    limits:
      memory: 5Gi

  podMonitor:
    enabled: true

  podDisruptionBudget:
    enabled: true
    minAvailable: 3

  service:

    # Enable topology-aware routing
    annotations:
      service.kubernetes.io/topology-mode: Auto

    # Expose the datadog_agent port for receiving data from your Datadog agents
    ports:
    - name: datadog-agent
      port: 8282
      targetPort: 8282
      protocol: TCP

  # Custom configuration for the vector chart: https://vector.dev/docs/reference/configuration/
  customConfig:
    # Global options - required when using customConfig
    data_dir: /vector-data-dir

    # Sources - receive data from Datadog agents
    sources:
      datadog_agents:
        type: datadog_agent
        address: 0.0.0.0:8282
        multiple_outputs: true
        store_api_key: false

    # Transforms - delete unnecessary host tags (cardinality reduction)
    transforms:
      remove_host_tag:
        type: remap
        inputs:
          - datadog_agents.metrics
        source: |-

          cluster_env = "${CLUSTER_APP_ENVIRONMENT}"
          cluster_group = "${CLUSTER_APP_GROUP}"
          cluster_name = "${CLUSTER_NAME}"
          cluster_region = "${CLUSTER_REGION}"

          if cluster_env != "" {
            .tags.env = cluster_env
          }
          if cluster_group != "" {
            .tags.app_group = cluster_group
          }
          if cluster_name != "" {
            .tags.eks_cluster = cluster_name
            .tags.cluster_name = cluster_name
            .tags.cluster_id = cluster_name
          }
          if cluster_region != "" {
            .tags.region = cluster_region
          }

          # Currently, no metrics contain the remove_host_tag, so this block does not actually run
          # This was done to ensure the vector aggregator works without data loss prior to us dropping tags
          if (exists(.tags.host) && exists(.tags.remove_host_tag)) {
            host_value = .tags.host
            del(.tags.host)
            # If the metric is not a distribution, we need to set a tag for uniqueness. Use the vector host, which should have lower cardinality than the datadog host.
            if .kind != "distribution" {
              .tags.vector_host = "${HOSTNAME}"
            }
            log("Forwarding processed metrics: " + encode_json(.))
          }

    # Sinks - forward processed data to Datadog
    sinks:
      datadog_metrics:
        type: datadog_metrics
        inputs:
          - remove_host_tag
        default_api_key: "${DATADOG_API_KEY:-dummy-api-key}"
        site: "datadoghq.com"
        buffer:
          - type: memory
            max_events: 200000  # number of events to buffer in memory, larger values take more memory, but more resilient to data loss due to network issues
            when_full: drop_newest
        batch:
          max_events: 10000  # set lower than buffer.max_events to prevent buffer from being full when retrying failed batches
        request:
          concurrency: "adaptive"
          rate_limit_duration_secs: 1
          rate_limit_num: 50
          retry_attempts: 15
          retry_max_duration_secs: 1800
          retry_initial_backoff_secs: 1
          timeout_secs: 5
          adaptive_concurrency:
            decrease_ratio: 0.7
            ewma_alpha: 0.4
            initial_concurrency: 2
            rtt_deviation_scale: 2.5

  # Environment variables for Datadog API key and cluster info
  env:
    - name: DATADOG_API_KEY
      valueFrom:
        secretKeyRef:
          name: vector
          key: api-key
          optional: true
    - name: CLUSTER_NAME
      valueFrom:
        configMapKeyRef:
          name: cluster-config
          key: cluster-name
    - name: CLUSTER_APP_ENVIRONMENT
      valueFrom:
        configMapKeyRef:
          name: cluster-config
          key: cluster-app-environment
    - name: CLUSTER_APP_GROUP
      valueFrom:
        configMapKeyRef:
          name: cluster-config
          key: cluster-app-group
    - name: CLUSTER_REGION
      valueFrom:
        configMapKeyRef:
          name: cluster-config
          key: cluster-region

Vector Logs

In one of our high traffic regions, we are seeing these logs

2025-10-01T19:06:32.331265Z  WARN sink{component_kind="sink" component_id=datadog_metrics component_type=datadog_metrics}:request{request_id=43925}: vector::sinks::util::retries: Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased. internal_log_rate_limit=true
2025-10-01T19:06:32.331357Z  WARN sink{component_kind="sink" component_id=datadog_metrics component_type=datadog_metrics}:request{request_id=43924}: vector::sinks::util::retries: Internal log [Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable. Alternatively `request.timeout_secs` can be increased.] is being suppressed to avoid flooding.

In some other relatively lower traffic regions, we are seeing logs like this...

2025-10-01T21:11:57.779294Z ERROR source{component_kind="source" component_id=datadog_agents component_type=datadog_agent}:http-request{method=POST path=/api/v2/series}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=2791 reason="Source send cancelled." internal_log_rate_limit=true
2025-10-01T21:55:53.149487Z ERROR source{component_kind="source" component_id=datadog_agents component_type=datadog_agent}:http-request{method=POST path=/api/v2/series}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=3338 reason="Source send cancelled." internal_log_rate_limit=true
2025-10-01T22:32:29.237871Z  WARN sink{component_kind="sink" component_id=datadog_metrics component_type=datadog_metrics}:request{request_id=10803}:http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2025-10-01T22:32:29.238945Z  WARN sink{component_kind="sink" component_id=datadog_metrics component_type=datadog_metrics}:request{request_id=10803}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: Failed to make HTTP(S) request: connection closed before message completed internal_log_rate_limit=true
2025-10-01T23:01:36.497269Z ERROR source{component_kind="source" component_id=datadog_agents component_type=datadog_agent}:http-request{method=POST path=/api/v2/series}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1842 reason="Source send cancelled." internal_log_rate_limit=true

[syedg1]

These are intentional discarded events, we started monitoring based on unintentional discarded events to view metrics loss and noticed a reduction in unintentional discarded events when decreasing the batch.max_bytes.

However, we do see some unintentional discarded events line up with the "source send cancelled" logs. Does this happen because the vector buffer is full?