Resource labels and custom metrics labels are missing from tessera metrics

[cmurphy]

Description

Tessera metrics are missing three of the six resource labels described here:

https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-otel#detect-resource-attributes

The defined-by-us metric attribute service.version does not appear in these metrics. Though it looks like service.name is working, that might be getting populated by the PodMonitoring config. Adding other attributes like Hostname does not populate the label in the metrics explorer.

The attributes set at the metric-level by tessera itself are working.

Consequences

This means it is not possible to filter or aggregate tessera metrics by any of these attributes, which means it is impossible to tell when a metric arises from one shard or another or one revision or another:

There are also an extreme number of "TimeSeries could not be written...Points must be written in order. One or more of the points specified had an older start time than the most recent point." spamming the logs, which is another potential effect of this misconfiguration.

Mitigation

I can get the service_instance_id and service_namespace attributes to appear by setting OTEL_RESOURCE_ATTRIBUTES in the deployment. But I did not have luck setting k8s.namespace_name or getting namespace, job, or instance to appear as resource attributes.

Next steps

I'm filing this issue because the documentation on exporting custom metrics from GKE to Managed Prometheus is murky and does not seem straightforward.

We might need to set up Managed Collection for finer-grained attribute labeling.

It might be possible to coax the PodMonitoring resource to do this advanced metric relabeling.

[jku]

There are also an extreme number of "TimeSeries could not be written...Points must be written in order. One or more of the points specified had an older start time than the most recent point." spamming the logs, which is another potential effect of this misconfiguration.

marked this as blocking 470 based on this

[jku]

So I am very much a newbie here but: it seems like
Kubernetes Service Account "collector" needs to be connected to a GCP SA like iam.gke.io/gcp-service-account for monitoring.metricWriter and it does not seem to be -- at least I can't find that in the annotations

[jku]

So I am very much a newbie here but: it seems like Kubernetes Service Account "collector" needs to be connected to a GCP SA like iam.gke.io/gcp-service-account for monitoring.metricWriter and it does not seem to be -- at least I can't find that in the annotations

documenting findings mostly to myself:

• it seems When running on GKE, the collecting Prometheus server automatically retrieves credentials from the environment based on the node's service account meaning the workload is done with node default KSA that uses the GSA sigstore-staging-node-sa@projectsigstore-staging.iam.gserviceaccount.com so does get the appropriate permissions in GCP

• I think there might still be a problem here: I'm not convinced the node KSA has access to the metadata needed to add the resource labels. I think the we need a pod level identity for that -- in other words, we should not be using the default node KSA, we should make sure the collector pods have a service account annotation

• maybe something like this in argocd:

     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: collector
       namespace: gmp-system
       annotations:
         iam.gke.io/gcp-service-account: sigstore-staging-prometheus-sa@projectsigstore-staging.iam.gserviceaccount.com
  

EDIT: It seems like the node KSA is only used to get the metricWriter role, and the collector otherwise uses the normal gmp-system/collector KSA so I think this is not the issue... It really should work

[jku]

Something else to maybe test:
PodMonitoring docs say container metadata label is only populated if the port is referenced by name because port numbers are not unique across containers.

We refer to metrics port with a number

[haydentherapper]

Merged https://github.com/sigstore/public-good-instance/pull/3244 to see if that helps

[jku]

I don't see any change: possibly the metric would now have the container label... but the main problem listed in this issue is preventing that too.

Maybe we should get support to look at these issues, at least I can't figure them out:

• why the tessera metrics don't get resource labels like job instance and namespace
• why do we have loads of "Failed to export self-observability metrics to Cloud Monitoring: Points must be written in order. One or more of the points specified had an older end time than the most recent point" errors. These errors are for GKE/k8s internal metrics (all seem related to the metrics collection and exporting itself, and seemingly don't work at all: at least I can't find them in metrics explorer)

I'm not sure if these two are related but I believe the number of self-observability errors shot up around the time these tessera metrics were added.

[jku]

I'm doing one more try here. First some things where I'm not sure I understand the original report:

Tessera metrics are missing three of the six resource labels described here:

https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-otel#detect-resource-attributes

This list is a reasonable one but it is for opentelemetry collector specifically... We don't have an otel-collector AFAICT

We might need to set up Managed Collection for finer-grained attribute labeling.

But we are already using the managed collection method, right? This is why gmp-system has the collector pods.

It might be possible to coax the PodMonitoring resource to do this advanced metric relabeling.

So my assumption is that the metrics "export" is done by node exporter and that node exporter is not even supposed to add resource labels like this: I expected the managed collection to add labels like job, instance on it's own but somehow it does not. The podmonitoring honorLabels variable would affect what is done with conflicting labels from the scraped data... but as I said I'm not sure we have anything in the scraped data, and we don't set honorLabels so there should be no problems). reference: https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#reserved-labels

I just don't understand why managed collection does not add the labels... and maybe even more strangely I don't understand how the k8s native labels (like pod_name, container_name) don't show up

[jku]

Maybe this is something:

$ kubectl describe podmonitoring log2025-alpha2-rekor-tiles \
  -n log2025-alpha2-rekor-tiles-system

...

  Target Labels:
    Metadata:
      pod
      container
      top_level_controller_name
      top_level_controller_type

...

Is Target Labels supposed to be there -- it's not the default and we never set it (at least based on last-applied-configuration annotation ) so I think it has to be set by managed prometheus in a webhook?

assuming that's a bug in managed prometheus, we could try to set targetLabels: {} to maybe prevent the incorrect (?) value from being set in the webhook... It's unclear if targetLabels is supposed to be an "append" or a "replacement" but maybe it's worth a try.