The target allocator needs to handle target labels in a way that is compatible with both prometheus and prometheus-operator. Without relabel filtering, this only involves calculating target identity from labels, but even this can be subject to drift from upstream Prometheus semantics. This has been a source of many subtle bugs over the years, and even refactoring to improve the situation is hampered by fear of inadvertently introducing more such bugs.
We should improve test coverage to the level where major changes to the target allocator logic are possible without fear of breakage. Unfortunately, there isn't an official conformance test suite we can use here - the relabelling logic is an implementation detail of prometheus. But their code is public, so we should be able to at least cover what they do.
My rough plan for this is the following:
-
Run the target allocation pipeline against a suite of conformance tests generated from the results of promtool check service-discovery
-
Run the same suite as an integration test combining the target allocator and prometheus receiver in-process. This can actually verify the shape of the data.
-
Add an e2e test in a kind cluster which uses the same configurations to scrape the same data using both Prometheus itself and the target allocator + otel collector setup exporting to Prometheus, and verify that the target labels are the same.
Issues blocked on this:
The target allocator needs to handle target labels in a way that is compatible with both prometheus and prometheus-operator. Without relabel filtering, this only involves calculating target identity from labels, but even this can be subject to drift from upstream Prometheus semantics. This has been a source of many subtle bugs over the years, and even refactoring to improve the situation is hampered by fear of inadvertently introducing more such bugs.
We should improve test coverage to the level where major changes to the target allocator logic are possible without fear of breakage. Unfortunately, there isn't an official conformance test suite we can use here - the relabelling logic is an implementation detail of prometheus. But their code is public, so we should be able to at least cover what they do.
My rough plan for this is the following:
Run the target allocation pipeline against a suite of conformance tests generated from the results of
promtool check service-discoveryRun the same suite as an integration test combining the target allocator and prometheus receiver in-process. This can actually verify the shape of the data.
Add an e2e test in a kind cluster which uses the same configurations to scrape the same data using both Prometheus itself and the target allocator + otel collector setup exporting to Prometheus, and verify that the target labels are the same.
Issues blocked on this: