receive: Some metrics for the tenant are being calculated incorrectly

[c32exe]

Thanos version used: 0.39.1

Object Storage Provider:

What happened:
Hello!
Thanos is deployed in k8s cluster and receives metrics via the remote_write protocol from Prometheus and vmagents.
I'm using the --receive.split-tenant-label-name="tenan_tname" option to separate tenant metrics. However, the http_requests_.+ metrics collected from Thanos distributors do not reflect tenants passed via labels. These metrics are correctly attributed to tenants passed through the http header. For tenants specified via labels, however, these metrics are instead aggregated under the default_tenant.

Metrics like thanos_receive_write_.+ correctly include the tenant extracted from the label, rather than the default tenant.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

[GiedriusS]

It's by design - these metrics reflect the HTTP request level and on the HTTP request level the tenant is the default one (or whatever else) if you are using the split tenant label name functionality.