prometheus exporter reporting metrics with dots in them

[chrisleavoy]

Component(s)

exporter/prometheus

What happened?

Description

Upgraded to 0.124.1 and started receiving errors about the collector exposing invalid metric names via the internal health exporter to prometheus:

Steps to Reproduce

      "telemetry":
        "metrics":
          "readers":
          - "pull":
              "exporter":
                "prometheus":
                  "host": "0.0.0.0"
                  "port": 8888

Expected Result

target_info{service_instance_id="c1562f2a-14c3-47cd-b264-ac0acb57864a",service_name="otelcontribcol",service_version="0.124.1-dev"} 1

Actual Result

target_info{service.instance.id="c1562f2a-14c3-47cd-b264-ac0acb57864a",service.name="otelcontribcol",service.version="0.124.1-dev"} 1

Note . are used in labels instead of _!

If you curl localhost:8888/metrics you get the correct behavior, but when scrape modePrometheusProto is used, invalid metric names are produced.

utf-8 mode in Prometheus 3.0 is experimental now, so in theory, . are accepted. However, inconsistently translating metrics between the different scrape protocols is tortuous for end users, because PrometheusProto is the new default when native_histogram feature flag is set in Prometheus.

I wrote an example scrape to demonstrate I wasn't hallucinating and it was, in fact, an issue with the otel-collector not presenting the same metric and label names based on which scrape protocol was used (text vs. protobuf):

See https://gist.github.com/chrisleavoy/da17ed3d555ecf7bd929b47ecd880897

Collector version

0.124.1

Environment information

No response

OpenTelemetry Collector configuration

Log output

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

Hey 👋

I totally understand the confusion here, but the collector's internal metrics are exposed through otel-go-sdk's prometheus exporter. Here is the codebase: https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus

With that said, I totally agree that the SDK's exporter and the Prometheus exporter, the component in collector-contrib, should be aligned. @MrAlias @XSAM @dashpole @pellared @dmathieu, we might want to talk about this :)

[chrisleavoy]

Ah, that'd explain why i couldn't find the code that reported these. I lost half a day on this issue, lol.

[chrisleavoy]

I tried escaping=underscores; with proto and it seemed to be ignored. Is there a workaround? I'd really like to keep proto enabled for the native histogram support, but id rather not enable UTF8 until its supported in Mimir/Grafana.