feat: Allow utf-8 label names in write path

[jake-kramer]

Utf-8 label names are valid in the write path if the config client-capability.allow_utf_8_label_names=true is set, and the client capability allow-utf8-labelnames=true is passed in the Accept header for API calls.

Legacy label name validation and sanitization will continue to be enforced on write path, otherwise.

Follow up to #4442

@simonswine @bryanhuhta

[simonswine]

The write path should not use allow-utf8-labelnames=true. The capability is a read path only capability.

When profile producers (SDK, profilecli upload, Alloy pyroscope.write) support utf8 label names, they should:

Send the labels in utf8, if they have any.

Then there are two options:

• Backend support utf8 label names/has enabled support: All is fine
• Backend doens't support: The first request fails and the client falls back to filtering/translating label names.

So basically you should create a tenant override maybe IngestionAllowUtf8LabelNames, around here:

pyroscope/pkg/validation/limits.go

Line 65 in 019bac2

IngestionRelabelingDefaultRulesPosition RelabelRulesPosition `yaml:"ingestion_relabeling_default_rules_position" json:"ingestion_relabeling_default_rules_position" category:"advanced"`

Then the ingest path should not consider client flags, they should just call limits.IngestionAllowUtf8LabelNames(), and the act accordingly.

[jake-kramer]

The write path should not use allow-utf8-labelnames=true. The capability is a read path only capability.

When profile producers (SDK, profilecli upload, Alloy pyroscope.write) support utf8 label names, they should:

Send the labels in utf8, if they have any.

Then there are two options:

• Backend support utf8 label names/has enabled support: All is fine
• Backend doens't support: The first request fails and the client falls back to filtering/translating label names.

So basically you should create a tenant override maybe IngestionAllowUtf8LabelNames, around here:

pyroscope/pkg/validation/limits.go

Line 65 in 019bac2

IngestionRelabelingDefaultRulesPosition RelabelRulesPosition `yaml:"ingestion_relabeling_default_rules_position" json:"ingestion_relabeling_default_rules_position" category:"advanced"`

Then the ingest path should not consider client flags, they should just call limits.IngestionAllowUtf8LabelNames(), and the act accordingly.

Adding a metric to determine scope and impact of removing write path sanitization by default #4501