[receiver/kafkametrics] Add kafka.cluster.id resource attribute (Kafka cluster UUID)

[shashank-reddy-nr]

Component(s)

No response

Is your feature request related to a problem? Please describe.

The Kafka Metrics receiver emits broker, topic, partition, and consumer-group metrics, but it does not emit any stable, globally unique identifier for the Kafka cluster the metrics came from.

Today the only cluster-level dimension is kafka.cluster.alias, which is:

• optional and disabled by default,
• user-supplied free-form text, and
• not guaranteed to be unique or even present.

That leaves a real gap:

1. No reliable cluster key for correlation: There is no source-of-truth identifier on the emitted telemetry that ties these metrics to a specific cluster. Anything built on top (dashboards, topology/relationship building, joining the receiver's metrics with other Kafka signal sources) has to fall back on broker addresses or a hand-maintained alias.

2. Broker addresses are not a cluster identity: Self-hosted clusters very commonly reuse identical broker endpoints across different networks/VPCs/environments (e.g. kafka-broker-1:9092). Two distinct clusters can present the exact same bootstrap addresses, so broker address cannot be used to disambiguate them.

3. Correlating producers ↔ topic ↔ consumers requires a shared cluster key:
   To build a producer-app → topic → consumer-app topology in a backend, every signal source (this receiver, infra agents, APM agents) needs to agree on a single stable cluster key. There is currently no such key emitted by the receiver.

The Kafka-native cluster UUID solves all three: it is generated once at cluster creation, never changes, is globally unique per cluster, and is already returned in every standard broker MetadataResponse — so it's the natural, zero-cost cluster identity that's missing from the receiver's output.

Describe the solution you'd like

Emit the Kafka cluster UUID as a new resource attribute kafka.cluster.id.

• Source: it is already present in the metadata the receiver fetches. The franz-go admin client exposes it as kadm.Metadata.Cluster (Client.Metadata / Client.BrokerMetadata). The broker scraper already issues a metadata request (ListBrokers → internally Metadata), so the cluster ID comes back in the same response — no extra connection, no admin tooling, no new credentials, no customer configuration. (Cluster ID is Kafka-native since KIP-78 / Kafka 0.10.1.)

• Modeling: a resource attribute (constant per cluster), placed next to the existing kafka.cluster.alias, in the same kafka.cluster.* namespace. It is set once per scrape in the broker, topic, and consumer scrapers — it is not a per-datapoint metric attribute, because the value does not vary by broker/topic/partition.

• Default: disabled by default (opt-in via resource_attributes.kafka.cluster.id.enabled: true), matching the existing kafka.cluster.alias behavior, so it does not change existing users' output. When enabled, the topic/consumer scrapers issue one lightweight, topic-less metadata request to obtain it; when disabled, no extra request is made.

Example metadata.yaml entry:

resource_attributes:
  kafka.cluster.id:
    description: The unique ID (string) of the Kafka cluster, as reported in cluster metadata.
    enabled: false
    type: string

This is purely additive — no existing metrics or attributes are changed or removed. It also lets the receiver's metrics share a single stable cluster key with other Kafka signal sources, enabling cluster-scoped correlation and topology without payload changes or extra pipelines.

Describe alternatives you've considered

• kafka.cluster.alias (existing): user-supplied, optional, free-form, and not guaranteed unique — fine as a friendly label, but unusable as a reliable correlation key.

• Broker address as the cluster key: breaks for self-hosted clusters that reuse identical broker endpoints across VPCs/environments; addresses do not uniquely identify a cluster.

• Embedding an identifier in message payloads/headers (e.g. a producer-service header): adds bytes to every Kafka message and has collision edge cases (e.g. the same producer/topic name across different clusters can't be disambiguated). Rejected as too costly and not robust.

• A separate correlation pipeline (e.g. a dedicated service joining traces + logs): roughly doubles pipeline cost and adds operational complexity disproportionate to the problem.

The native cluster UUID avoids all of these: it's already in the metadata response, requires no payload or config changes, and is the only identifier that distinguishes clusters with identical broker endpoints.

Additional context

• The cluster UUID is the same identifier used by other Kafka instrumentation ( OTel or Proprietary APM agents(ex: New Relic APM Agents) that read it from the broker's metadata response), so emitting it here lets metrics from the OTel Collector and from agent-based sources be correlated on one shared key — enabling a producer-app → Kafka topic → consumer-app relationship graph keyed on the cluster.

• Implementation is small and contained: read kadm.Metadata.Cluster and set it via the generated resource builder in the three franz-go scrapers (broker_scraper_franz.go, topic_scraper_franz.go, consumer_scraper_franz.go). The receiver is franz-go only, so there is no Sarama path to touch.

• Verified end-to-end against a self-hosted Kafka cluster: with the attribute enabled, every emitted resource carries kafka.cluster.id equal to the broker's actual cluster UUID; with it disabled (default), it is absent and all other metrics are unchanged.

• Happy to open a PR - a working implementation already exists locally.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/kafkametrics: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.