[receiver/hostmetrics] Emit unique `service.instance.id` attribute per resource for interoperability with Prometheus as OTel backend

[aknuds1]

Component(s)

receiver/hostmetrics

Is your feature request related to a problem? Please describe.

The hostmetrics receiver emits resource attributes without service.instance.id or service.name. This prevents interoperability with Prometheus as OTel backend. For reference, per the Prometheus OpenTelemetry guide Prometheus treats service.instance.id and service.name as identifying resource attributes. The documentation states:

If a resource lacks both service.instance.id and service.name attributes, no corresponding target_info series is generated.

This means that when hostmetrics receiver metrics are sent to Prometheus via OTLP, all resource attributes are silently dropped — no target_info metric for encoding resource attributes is generated at all. The effect is that all resource context (hostname, OS info, etc.) is lost without warning.

The OTLP to Prometheus compatibility spec further mandates that service.instance.id MUST map to the Prometheus instance label and service.name to job. Without these, metrics from different hosts are indistinguishable — the instance label gets an empty value, collapsing all hosts into a single target.

The combined effect is:

1. No target_info metric generated: All resource attributes are lost
2. No instance label: Host metrics from different machines cannot be distinguished
3. No host-to-process correlation: Cannot correlate host-level metrics with process-level metrics from the same machine

To work around the receiver not sending identifying resource attributes recognized by Prometheus, users currently have to add a resourcedetectionprocessor to the OTel Collector pipeline to emit these attributes.

Describe the solution you'd like

I propose adding a deterministic service.instance.id resource attribute to all scrapers in the hostmetrics receiver:

• For host-level scrapers (cpu, disk, filesystem, load, memory, network, nfs, paging, processes, system): a UUID v5 derived from the hostname using the OTel namespace UUID (4d63009a-8d0f-11ee-aad7-4c796ed8e320), following the semantic conventions recommendation for deterministic ID generation
• For the process scraper: a unique UUID v5 per process, derived from the base hostname UUID combined with the process PID — ensuring each monitored process is a distinct Prometheus target

Describe alternatives you've considered

• resourcedetectionprocessor — works but requires extra pipeline config, doesn't provide per-process uniqueness
• Random UUIDs — not deterministic, change on collector restart, would create new Prometheus time series on every restart
• Collector-level service config — no per-process support for the process scraper

Additional context

Additional context

• Follows OTel semantic conventions for deterministic service.instance.id generation using UUID v5.
• Reference: I previously opened [receiver/hostmetricsreceiver] Add service.instance.id resource attribute #45548 to propose the same change. @braydonk suggested I open an issue for discussion instead, so doing that here.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/hostmetrics: @dmitryax @braydonk @rogercoll

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[lmolkova]

I think this is a part of general issue we've been discussing in the context of postgres (and DB/messaging/etc) receivers in #45736, #45345, #45347.

Telemetry coming from collector receivers + processor should have service.name and service.instance.id set to the best of collector knowledge. This is a requirement for SDKs.

The resource detection may be imperfect, not configured, but there should be some not terrible defaults.

I think it's important to have some common guidance for collector components on how to achieve it.

Possible options:

1. what happens today: receivers do whatever their component owners think is reasonable, resource detection processor may or may not override it. It seems resourcedetectionprocessor does, but k8s attrs does not (k8sattributeprocessor does not override attributes even if user explicitly configured annotation #45736). Users are responsible to configure everything properly and it depends on the combination of receivers and resources detectors they use
2. receivers, when it makes sense, set service.instance.id, service.name, maybe some other attrs but resource detectors override them if they can by default.
3. receivers don't set defaults, but some central normalization component sets required attributes for all of them (only?) when the resource detection is not configured
4. Users are required to configure resource detection of some sorts, without it some basic things like host metrics are not expected to work, and perhaps there is even a warning when resource detection is not configured

I don't have enough context in the collector to understand what's possible and technically sound, but from the end user perspective, it seems reasonable to expect service.instance.id and service.name on telemetry they export by default.

[Aneurysm9]

• For host-level scrapers (cpu, disk, filesystem, load, memory, network, nfs, paging, processes, system): a UUID v5 derived from the hostname using the OTel namespace UUID (4d63009a-8d0f-11ee-aad7-4c796ed8e320), following the semantic conventions recommendation for deterministic ID generation

should this be FQDN to avoid collision among hostnames from different domains?

• For the process scraper: a unique UUID v5 per process, derived from the base hostname UUID combined with the process PID — ensuring each monitored process is a distinct Prometheus target

is just the PID sufficient? if the hostname UUID is deterministic then UUIDs derived from it and PIDs will eventually get reused, either on reboot or when the PID rolls over.

[dmitryax]

For reference, per the Prometheus OpenTelemetry guide Prometheus treats service.instance.id and service.name as identifying resource attributes. The documentation states:

If a resource lacks both service.instance.id and service.name attributes, no corresponding target_info series is generated.

This effectively introduces a requirement for OpenTelemetry to associate every resource with a service entity. Do we have this requirement documented anywhere on the OTel side? This is the first time I’ve seen it, and it hasn’t been discussed within the OTel Entities SIG. cc @jsuereth

The issue is that a service entity is an application-level concept. Metrics produced by infrastructure receivers (e.g., host metrics, Kubernetes cluster metrics) describe infrastructure, not applications.

For example, a Kubernetes namespace (another entity with its own identifying attributes) has metrics associated with it. Which service should it be associated with? A namespace may host a dynamic set of services that change over time. Creating a synthetic service for a namespace is not useful and would only add confusion.

A similar situation for process entities. There is typically a one-to-many relationship between a service and the processes it runs on. If we create a synthetic service entity for each process, it will be difficult to associate it later with the real service reported by the application.

[dmitryax]

Instead of relying on service entity to be present, prometheus exporter should probably read the entities from the resource and use their identity attributes for target_info

[aknuds1]

Instead of relying on service entity to be present, prometheus exporter should probably read the entities from the resource and use their identity attributes for target_info

@dmitryax Prometheus' OTLP endpoint will indeed use entities when they are GA. But even after entities are GA, how does the OTLP endpoint identify resources for which entities are not included in the payload? AFAIK, entities will be optional.

[dmitryax]

AFAIK, entities will be optional

Entities on resource won't be optional. If they aren't present it's a backward compatibility concern. What does prometheus currently do for this problem?

[aknuds1]

Entities on resource won't be optional. If they aren't present it's a backward compatibility concern. What does prometheus currently do for this problem?

Can you elaborate on what you mean @dmitryax? I don't see how it can be enforced that entities are sent with OTLP payloads, considering that the Entity Data Model is supposed to be backwards compatible, and strictly additive to the Resource model AFAIK. Prometheus isn't currently Entity Data Model aware, as it's still under development AFAIK, and you have to import an experimental package to inspect them in OTLP payloads.

[lmolkova]

I think there are two problems here

1. Long term: Prometheus will need to adjust and leverage entities, but currently it uses resources - https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#resource-attributes and can't use entities since nothing emits them

2. Short term: what should users do until entities land and prometheus compatibility spec changes to use them? Should collector provide some guarantees that service.instance.id is set (this is probably a good guarantee to provide for host metrics).
   Coming back to my previous point - today users are responsible to configure specific combination of receivers and resources detectors and there is no consistent or documented happy path.

[dashpole]

I think @dmitryax is correct that the service resource attributes should not be applied to infrastructure, like a host. But once we are talking about a process on the host, it seems like that should be considered an application, and receivers can generate service.instance.id as described in the service conventions. The root of the issue is that Prometheus' job and instance are domain-agnostic identifiers, but the service namespace is application-specific.

For metrics about a host that are not associated with a process, we should consider changing the mapping for job and instance to better-handle infrastructure namespaces by adding fallbacks in prometheus exporters. I think the requirements for instance might be slightly looser than those of service.instance.id (since prometheus uses IP + port and it is common for IP + port to be re-used within a k8s cluster), and could be as simple as using host.id when it is present.

I suspect OBI has faced a similar question of how to set service.* for processes discovered on a host, and could possibly have some lessons learned here that would be helpful. @grcevski @mariomac