Missing runtime metrics

[morningspace]

Description

Some runtime metrics, e.g.: memStats.Alloc, memStats.Sys, are missing.

Steps To Reproduce

• By checking the code for deprecated runtime, we found there're no runtime metrics for memStats.Alloc, memStats.Sys.
• Also in the new runtime metrics implementation, the code to collect the metrics is different. Then, what would be the equivalents to the missing runtime metrics in the new implementation?

Expected behavior

Can the missing runtime metrics be added? If a PR is allowed, we'd be happy to contribute.

[ysomad]

any updates?

[dmathieu]

The non-deprecated runtime uses runtime/metrics to retrieve metrics data.
The metrics we exposed are documented in semantic conventions.

It looks like you're looking for go.memory.allocated and go.memory.used.

The current runtime does expose those two metrics on its latest version, even though it uses the runtime/metrics package, not runtime.MemStats.

[morningspace]

Thanks @dmathieu for your reply!

• It was mentioned in semantic conventions that go.memory.allocated is the memory allocated to the heap by the application, while in runtime.memStats, it has both memStats.HeapAlloc and memStats.Alloc available. So, I assume go.memory.allocated maps to memStats.HeapAlloc, not memStats.Alloc.
• For memStats.Sys, that's the memory obtained from system. When I check the code in non-deprected runtime, I found it actually checks go heap memory defined here, which I'm not sure if it is identical to memStats.Sys.

[dmathieu]

cc @dashpole who led this.

[dashpole]

Based on https://github.com/prometheus/client_golang/blob/76b74e25d5660965000a74cf2e918c217ed76da2/prometheus/go_collector.go#L26, this is the mapping from memStats to go runtime metrics:

• memStats.Alloc is the same as the /memory/classes/heap/objects:bytes runtime metric.
• memStats.Sys is the same as the /memory/classes/total:bytes runtime metric.

/memory/classes/total:bytes isn't useful on its own, as it includes released memory. For the new runtime metrics, we provide the go.memory.used metric, which excludes released memory.

You can see golang/go#67120 for why we don't provide live + unswept heap memory via /memory/classes/heap/objects:bytes:

live+unswept heap memory isn't a terribly useful metric since it tends to be noisy and misleading, subject to sweep scheduling nuances. The heap goal is a much more reliable measure of total heap footprint.

In keeping with that, we would recommend using the go.memory.gc.goal metric, which measures the heap goal.

[morningspace]

Hi @dashpole, thanks for your reply and sorry for the delay. I spent a few hours today to revisit the materials mentioned in the above thread. Here's my findings and feel free to correct me if any.

/memory/classes/total:bytes isn't useful on its own, as it includes released memory ...

While I can understand go.memory.used is useful as it excludes released memory, according to golang/go#67120, I don't think /memory/classes/total:bytes is not that useful. It is part of the "Proposed initial metrics" w/ the following rational:

This metric is necessary for tuning GOMEMLIMIT. It's also useful for identifying "other" memory, and together with /memory/classes/heap/released:bytes, what the runtime believes the physical memory footprint of the application is.

You can see golang/go#67120 for why we don't provide live + unswept heap memory via /memory/classes/heap/objects:bytes

According to golang/go#67120, I think the reason that /memory/classes/heap/objects:bytes not included in the "Proposed initial metrics" is different. It's because this metric can be derived from total allocations and frees, which makes it redundant, as its value is essentially a subtraction of the two other metrics.

Re: the discussion "live+unswept heap memory isn't a terribly useful", it actually refers to /gc/heap/frees:bytes, as this metric, combining with /gc/heap/allocs:bytes, can be used to compute the total amount of live + unswept heap memory. But since it includes unswept memory, it recommends to use /gc/heap/goal:bytes, i.e.: go.memory.gc.goal, to measure total heap footprint.

With that, I'm still thinking there are some enhancement chances for the new runtime metrics. WDYT?

Also, I'm wondering when the deprecated runtime metrics will be completely dropped given there are enhancement chances as well for the deprecated ones. It looks currently the feature flag OTEL_GO_X_DEPRECATED_RUNTIME_METRICS is still enabled by default, which means it will use the deprecated runtime metrics by default. BTW: I am working on a more detailed investigation of potential gaps for the deprecated runtime metrics, which I will share here a bit later.

[dashpole]

This is definitely the right time to make changes to the new runtime metrics. I don't think there is a rush with disabling the old metrics, and we won't remove them for a while. In particular, we could probably add some metrics which are disabled by default, with options to enable them. A few follow-up questions:

This metric is necessary for tuning GOMEMLIMIT

I believe you would want to compare /memory/classes/total:bytes - /memory/classes/heap/released:bytes (i.e. go.memory.used) to GOMEMLIMIT, right? From my understanding, /memory/classes/total:bytes on its own wouldn't be useful in that case.

It's also useful for identifying "other" memory

This seems like the primary use for it, but i'm fuzzy on the actual value here. It would just let us calculate the "released" memory, which from reading doesn't sound that useful.

It's because this metric can be derived from total allocations and frees, which makes it redundant, as its value is essentially a subtraction of the two other metrics.

Do you think we should expose it anyways? Its possible it could be an opt-in metric. But given the go maintainers recommended using /gc/heap/goal:bytes instead, it seems like it could mislead users.

BTW: I am working on a more detailed investigation of potential gaps for the deprecated runtime metrics, which I will share here a bit later.

Looking forward to it! We really appreciate the feedback.

[morningspace]

Thanks @dashpole !

I like the idea of "add some metrics which are disabled by default, with options to enable them", so that's the end user who determines whether the metric is needed. A few more comments after you as below:

you would want to compare /memory/classes/total:bytes - /memory/classes/heap/released:bytes (i.e. go.memory.used) ...

Just to confirm, by reading the code, it looks go.memory.used has two variations with different attributes: go.memory.type = stack vs. other. And, other equals to /memory/classes/total:bytes - /memory/classes/heap/released:bytes - /memory/classes/heap/stacks:bytes, not /memory/classes/total:bytes - /memory/classes/heap/released:bytes. So, it appears there's no equivalent.

This seems like the primary use for it ...

I'd think this might be one candidate of opt-in metrics, because the total memory usage /memory/classes/total:bytes includes heap, plus other runtime components like stacks, metadata, and cache, which might contribute to the application’s memory footprint to some extent, depending on the nature of user's application and its workload.

Do you think we should expose it anyways? ...

I agree that it could be an opt-in metric. Strictly, /gc/heap/goal:bytes and /memory/classes/heap/objects:bytes are different. The first one represents the target heap size at which the garbage collector will trigger the next cycle, while the second one shows the current live heap memory and reflects the real-time memory usage. In general, /memory/classes/heap/objects:bytes should stay below /gc/heap/goal:bytes. If it frequently approaches or exceeds the goal, it indicates the application is under memory pressure, potentially triggering frequent garbage collections.

Does it make sense?

[dashpole]

So, it appears there's no equivalent.

If you do not group by the go.memory.type label, you would get /memory/classes/total:bytes - /memory/classes/heap/released:bytes, which is what we want. It is common to do this for metrics where the total is a sum of parts (heap vs other in this case).

[dashpole]

If we want to add /memory/classes/heap/objects:bytes, we would need to propose it here: https://github.com/open-telemetry/semantic-conventions/blob/106f880ccbf26443f115ce1f48c236ec6a0b6f1b/docs/runtime/go-metrics.md?plain=1#L4