profiles: Profiles reinvents the attributes

[bogdandrutu]

See

opentelemetry-proto/opentelemetry/proto/profiles/v1development/profiles.proto

Line 170 in 56bc2ce

repeated KeyValueAndUnit attribute_table = 6;

In otel attributes are not associated with a unit, and this is an inconsistency (already raised in meetings and slack) with the rest of the ecosystem. We should remove unit and have a global discussion about adding this to all the attributes if necessary otherwise define a new concept for it.

Also, why do you need unit in the first place? Attributes are not measurements.

[aalexand]

Indeed, attributes are supposed to be categorical values, but they can still have units.

Some examples (not an exhaustive list):

• Memory size values: e.g. breaking down allocation metrics by the size of the allocation.
• Duration values: e.g. breaking down CPU metrics by the duration of the request in which the CPU sample occurred.
• Simple count values: e.g. the number of in-flight RPC requests when the CPU sample occurred.

The values are typically binned (log2, fixed histogram, ...) to keep the cardinality limited, but that's orthogonal.

Knowing the attribute unit is important for properly scaling the value in display and for supporting magnitude-aware units in filtering.

google/pprof#213 is the original PR in pprof that added the unit support for tags/labels (which is the pprof speak for attributes).

[bogdandrutu]

Simple count values: e.g. the number of in-flight RPC requests when the CPU sample occurred.

I understand this decision was made by profiling in the past, but do you have a rationale on this and when do we record the metrics as metrics vs as attributes in profiles? It will become confusing for some people not familiar with profiling when to use one vs the other.

[aalexand]

I think I already explained the motivation in my message.

[bogdandrutu]

I don’t see where you explain why recording metrics as profiles attributes is needed (I do understand why you need unit if you do that), that may be my limited ability to understand…

[aalexand]

They are not metrics, they are categorical variables as other attributes. I gave examples.

[felixge]

cc @open-telemetry/profiling-maintainers @open-telemetry/profiling-approvers @open-telemetry/profiling-triagers

[jsuereth]

TL;DR: I don't think this issue is an issue, at least not as worded.

@bogdandrutu Profiling is inherently calculating aggregate data, but NOT the way we consider metrics in OTLP today. There are two important distinctions:

• Metrics tend to aggregate data over a span of time. Profiles tend to aggregate data over a a span of code. That is the implicit / "default" aggregations you want to perform are different.
• Profiles are heavily multi-variate - I.e. there's lots of related "metric points" that share attributes and timestamps and are reported together.

I believe you could force fit profiles into metrics, but you'd just be building an intractable system that could not work in real life.

Note: I suspect this type of data (multivariate, high cardinality) to be something we'll need to support (e.g. RUM work). OTEL metrics do NOT support this well today, and we see countless examples of people using logs with attributes (that have units) to model this.

In my view there's a lot of things to consider here.

SHOULD we bring units to other attributes in OTLP?

Yes. We've been asked to do so in semantic conventions a few times already. I think it's inevitable we figure out how to support units on attributes. For a while I believed we could 100% use the side-channel of schema_url to communicate units. I stil lthink we should rely on this for metrics/spans/logs for now, BUT I've come around to the idea that we'll need to eventually have support for this inline.

For profiling specifically, we don't have "translation compatibility" with PPROF if we do not. While we are not making OTLP be exactly transmissible as PPROF, we still need to have a clear way to communicate PPROF data to systems which rely on it today, and that includes passing unit with the data vs. a side channel.

CAN we bring units to other attributes in OTLP?

If we consider the entire ecosystem, this is likely a breaking change for some subset of APIs, which require stability. I believe we COULD find a path forward for this but:

• I believe the schema_url reliance for passing units will alleviate some pressure to add this generally. I do NOT think this workaround is good enough for profiling.
• I think we'll need to sort out a path forward over time, and this will be a major effort. Now is not the time to tackle this, we need to focus on "stable by default" and pushing major breaking changes through our ecosystem purely for "consistency" doesn't seem like a good idea. YES we should aim for consistency, but not at the cost of stability. We need to balance these two considerations (remember our principles).

Is this a blocker concern for profiling?

No. Profiling already has key differences to account for its data, e.g. the dictionaries.

I would like to see us have a stronger plan around RUM / client-side data going forward outside of hiding data in logs events where we can't interact with it semantically (e.g. "simple" spatial aggregation), but for now this is the world we've built.

[bogdandrutu]

Personal opinion is that the main reason to bring profiling into OpenTelemetry is to make it compatible and inter-operate with the other signals already existing in OpenTelemetry (if that is not the case, let's document the motivation to bring profiling into OpenTelemetry, or link if that is already somewhere). With that mindset, I believe it is very important to not have these kind of "one-ofs".

I may be wrong in my reason though.

[aalexand]

One option to discuss could be whether units can be a part of the semantics convention for each attribute.

The caveats there are:

• In pprof the unit captures both the nature of the unit (e.g. memory -> bytes) and the magnitude (e.g. kbytes vs mbytes vs bytes). Semantics conventions could capture the former but not the latter since it's dynamic and not 1:1 with the attribute name.
• Requiring units to be mapped in semantic conventions would mean that not every pprof profile can be converted into an OTel profile since some attributes may carry the unit in the pprof profile but the semantics conventions either do not describe that attribute or don't document the unit for it or the unit is not compatible / not the same as in the pprof profile.

[christos68k]

I looked at the eBPF profiler code and unless I missed something, the profiler never sets unit in KeyValueAndUnit, which makes pprof the only (currently active) use-case (as @jsuereth outlined).

In which case, can't we come up with a scheme that maintains pprof convertibility whilst switching to OTel attributes and avoids side-channel transmission? (e.g. encode the unit as part of the key and make it clear it's pprof-specific).

[jsuereth]

One option to discuss could be whether units can be a part of the semantics convention for each attribute.

Yeah, you can see this addressed in my comment. I think we should do this, but it still doesn't alleviate the full issue for profiling, it just helps other signals in OTEL where, e.g. we're trying to use "js objects" and "python dictionaries" for attributes, and adding something beyond key-value is awkward.

Personal opinion is that the main reason to bring profiling into OpenTelemetry is to make it compatible and inter-operate with the other signals already existing in OpenTelemetry

I don't see how having "unit" on attribute prevents this. This is ignoring the forest for the trees.

The important compatibility we need here is the following:

• We correlate metrics/logs/spans with profiles by resource context - e.g. I'm getting latency spikes. Let me go filter profiles for this to see what might be happening.
• We have an open standard exchange format, where we can use the observability-data-plane we set up for getting profiling data from where it's generated to our backends.

We still have some important unanswered questions (although @open-telemetry/profiling-maintainers can correct me if we do have answers now)

• What traces exist at the time a profile was collected? Can we do examplars in a similar fashion to how do that for metrics?
• How will Context interact with profiles? e.g. can I tag profiles via baggage to identify A vs. B paths or red/blue testing performance of paths?
• What (spatial) aggregation might we need for profiles in our backends.

There are MUCH HARDER issues to solve here than "do attributes have the same values?" or "can I put profiler attributes into log attributes". I also think they're much more important problems to focus on.

TL;DR: do I agree with this statement: "bring profiling into OpenTelemetry is to make it compatible and inter-operate with the other signals already existing in OpenTelemetry" YES.

Do I think resolving this issue brings us closer to that goal? no

[florianl]

The important compatibility we need here is the following:

• We correlate metrics/logs/spans with profiles by resource context - e.g. I'm getting latency spikes. Let me go filter profiles for this to see what might be happening.

The reference based attributes change (#733) aligns the understanding of resources in Profiling, with the understanding of resources in Logs, Metrics and Traces. In addition, open-telemetry/opentelemetry-specification#4905 clarifies the understanding for eBPF use cases - not just for Profiling but also for OBI. These two changes are essential, I think, to allow cross signal correlation.

We still have some important unanswered questions

• How will Context interact with profiles? e.g. can I tag profiles via baggage to identify A vs. B paths or red/blue testing performance of paths?

There is work in progress to share context. For bpf to bpf cases (like OBI to profiler), there is open-telemetry/opentelemetry-specification#4855. And for SDK to others (like profiling), there is open-telemetry/opentelemetry-specification#4719.

Profiling does not rely on KeyValueAndUnit attributes to describe the Resource of profiled elements. Consequently, the correlation between Profiles and other OTel signals, which is based on Resources, remains unaffected. But there are use cases, where attaching a unit to an attribute is significant - like bytes vs kBytes or GBytes for heap information. My personal preference would be to identify these attributes during the upcoming Alpha phase of Profiling and work on improving the Semantic Convention for these attributes.

[tigrannajaryan]

My opinion on this is the following:

• It would be very useful to have units for attribute values for other signals, e.g. for spans or logs. For example we have a http.client.request.body.size metric, which is in bytes and the unit specified is By. We also have http.request.body.size http span attribute, which is expected to be in bytes by convention, but there is no place in OTLP to record the unit explicitly. I don't see why we have the unit for metric values by not for numeric attribute values. The unit would make visualization in backends better. Having the unit would also make things like span-to-metric conversion better supported, where the unit would be carried over properly.
• I do not think we should block profiles progress until we add units to attributes in other signals (or decide that we can't/won't). Profiles should be able to go ahead. However we need to be prepared that while profiles signal is experimental we may break it for the purpose of aligning it to the rest of Otel. This may happen if we for example decide that units must be added to regular attributes. In that case we will likely get rid of KeyValueAndUnit and just use KeyValue. This of course can only happen before profiles are declared stable. I am speculating here, I do not know if anyone has an actual appetite to look into adding units to all attributes. I personally don't have time to do it, but if someone is willing to work on it I will review it.
• Consistency with other signals is important, but I do not think it is an absolute requirement for Otel. We want new signals to be possible to add Otel and we can't foresee all future signal requirements. In this case the profile signal needs units because profile representations in the industry have units (e.g. pprof does). We always said in the past we want OTLP to be able to losslessly represent other formats of the same signal that industry already has.

[felixge]

Based on @tigrannajaryan 's comment above, we're proposing to not block the alpha maturity graduation of profiling (see open-telemetry/sig-profiling#64) on this issue.

However, we consider making a decisions around this to be a pre-requisite for moving profiling to beta stage or beyond.

Does that work for you @bogdandrutu ?