Rule proposal: Only encouraged resource attributes found

[jpkrohling]

Proposal

Rule ID: MET-nnn

Description: Metrics contain only resource attributes that are useful when aggregating events.

Rationale: Metrics are aggregations of events over time. Resource attributes contain a mix of identifying attributes and metadata attributes (something that is being handled at the entities SIG). For metrics, it's recommended to keep only the resource attributes that would be used in aggregations, either at the storage level (typically as a time-series database) or query (like in columnar stores). For example, the following resource attributes would certainly not provide value for metrics payloads:

• process.runtime.description
• process.command_args
• process.command_line
• host.ip (this contains a list of IPs from the host that generated the telemetry)
• host.mac (this contains the list of mac addresses)
• os.description (when this is collected, there are typically other metadata emitted that can be used in this place)

Here's an example of resource attributes from the OTel Demo, Ad Service

It's worth noting that those attributes might make sense for spans and logs, just not so much for metrics.

Target: Metric

Criteria: Resource attributes for metrics do not include the following attributes:

• process.runtime.description
• process.command_args
• process.command_line
• host.ip (this contains a list of IPs from the host that generated the telemetry)
• host.mac (this contains the list of mac addresses)
• os.description (when this is collected, there are typically other metadata emitted that can be used in this place)

Impact: Normal

[EzzioMoreira]

Sometimes we encounter questions about the purpose of the generated telemetry. I believe some labels are important for governance, and in my opinion, k8s.namespace.name would be one of them, among others. In our daily work, we often encounter questions about the purpose of telemetry—observability vs. governance.

[tsloughter]

Isn't it that you want to ensure they aren't promoted to metric attributes, ot that you don't want them as resource attributes? I'm not sure what other metric backends do but in the case of Prometheus this is what I'd most want to check for users' metric data. And in that case I'd want to keep even more out of the metric attributes but still in the resource attributes than you have listed. As long as pod uid or service id are there you can join with the resource attributes that include the rest of the attributes like k8s.pod.ip.

This way you still have access to the other attributes in case you want to spend the cost of a join to access them, like host.ip -- maybe you are investigating increased latency and have a theory that it is actually the whole host seeing increased latency -- but in your regular dashboards you just query based on the attributes promoted to the metrics.

[jpkrohling]

No, I really really want to avoid that being generated in the first place.

The host.ip is an array of IPs containing all the IPs from all interfaces in the host. Same for host.mac . The full os.description is also useless, as other similar but less verbose attributes are also typically sent when this is used. In practice, they provide zero value to the absolute majority of our users today: if I really need to know the list of IPs of the host, can I trust that the resource attributes value represents the state of the host when the telemetry has been generated, or does it represent the state of the host when the process started? Am I ever going to aggregate based on the command arguments?

I have additional arguments for not using this at the storage (time series or columnar stores), but the main point is really that those specific values are DEFINITELY not useful as resource attributes for metrics.

[tsloughter]

Ok, true, there are other values to know what node something is happening on. But my point was more general (and in part a question) about if there is need for filtering down the resource attributes on metrics if they aren't being promoted to metric attributes.

I'd have expected a check against them being promoted, which is common and I've seen it used to simply promote every resource attributes instead of the necessary few. Not something that can really be checked but I've also seen this done on other signals, Datadog is putting all resource attributes on each span for example.

[mmanciop]

I feel this is a more specialized version of "metric cardinality should be sensible". In and of itself, the spirit of the rule seems fine to me, but it leaves a lot open to interpretation. Also, I am not sure whether we should really focus it on resource attributes, because one can blow up cardinality really easily with metric data point attributes too (for example when the instrumentations cannot give you a sensible http.route).

[jpkrohling]

I kept the focus of this very narrow because it's not only about the cardinality: these attributes are clearly garbage, and should be removed no matter the cardinality implications, as they cause an overhead on everything they touch, especially network.

You are right though that we could/should look at common offenders at the metric and data point attributes and have a strong opinion about them as well.

[jpkrohling]

Folks, I feel like this has been well-received, and unless you have concerns, I'm opening a PR to include this.

[mmanciop]

I would prefer this rule to be rephrased as a matter of reducing the risk of changing "LabelSet" (to borrow a Prometheus terminology) for metrics, rather than being phrased as a judgment on the usefulness of specific attributes, which may not be objectively rendered by tooling, especially when the perceived usefulness is something situational that can very well change over time.

Although, maybe we could morph this in a "Hey use entities for OS level stuff" as that is probably a better way of conveying rich information about, say, the operating system, and it should not have side-effects on metric cardinality or the stability of the LabelSet.