Bug: Continuous Test Fails to Read Histogram Data After Successful Write

[AneelkumarAmmanagi]

What is the bug?

I have followed these steps :-
[https://github.com/grafana/mimir/blob/main/docs/sources/mimir/send/native-histograms/_index.md]

How to reproduce it?

Directly running in locally with go .

What did you think would happen?

buckets problem

What was your environment?

Running after building binary file

Any additional context to share?

No response

Issue :-
ts=2025-03-04T10:31:32.446552Z caller=write_read_series.go:224 test=write-read-series method=WriteReadSeriesTest.writeSamples level=warn timestamp="2025-03-04 10:07:20 +0000 UTC" num_series=10000 type_label=histogram_float_gauge msg="Failed to remote write series" status_code=400 err="server returned HTTP status 400 Bad Request and body \"send data to ingesters: failed pushing to ingester mimir-ingester-zone-c-1: user=anonymous: err: 0 observations found in buckets, but the Count field is 1: histogram's observation count should equal the number of observations found in the buckets (in absen\" (truncated to 256 bytes)

ts=2025-03-04T10:38:49.883736Z caller=write_read_series.go:421 test=write-read-series method=WriteReadSeriesTest.runInstantQueryAndVerifyResult level=warn query=sum(mimir_continuous_test_histogram_float_gauge_v2) ts=1741082980000 results_cache=true type=histogram_float_gauge msg="Instant query result check failed" err="histogram at timestamp 1741082980000 (2025-03-04 10:09:40 +0000 UTC) has sum 11339.746003 while was expecting 11339.746003"

[narqo]

@zenador you may some context of native histograms besides krajo. Can you help checking where this could possibly come from?

[AneelkumarAmmanagi]

While running the continuous-test setup in the cluster using the following command:

./cmd/mimir/mimir -target=continuous-test \                          
  -tests.write-endpoint="" \
  -tests.read-endpoint="" \
  -tests.write-read-series-test.histogram-samples-enabled=true

I encountered the following error:

ts=2025-02-20T08:57:33.74343909Z caller=write_read_series.go:313 test=write-read-series method=WriteReadSeriesTest.runRangeQueryAndVerifyResult level=warn query=sum(mimir_continuous_test_histogram_int_counter_v2) start=1740041840000 end=1740041840000 step=20s results_cache=true type=histogram_int_counter msg="Range query result check failed" err="expected 1 series in the result but got 0"

After this, I switched to local testing using mimir-distributed and followed the steps outlined in the Grafana Mimir Native Histograms Documentation.

However, I encountered another error while running the write-read-series test:

ts=2025-03-04T10:31:32.446552Z caller=write_read_series.go:224 test=write-read-series method=WriteReadSeriesTest.writeSamples level=warn timestamp="2025-03-04 10:07:20 +0000 UTC" num_series=10000 type_label=histogram_float_gauge msg="Failed to remote write series" status_code=400 err="server returned HTTP status 400 Bad Request and body \"send data to ingesters: failed pushing to ingester mimir-ingester-zone-c-1: user=anonymous: err: 0 observations found in buckets, but the Count field is 1: histogram's observation count should equal the number of observations found in the buckets (in absen\" (truncated to 256 bytes)

Additionally, during the instant query verification, the following error occurred:

ts=2025-03-04T10:38:49.883736Z caller=write_read_series.go:421 test=write-read-series method=WriteReadSeriesTest.runInstantQueryAndVerifyResult level=warn query=sum(mimir_continuous_test_histogram_float_gauge_v2) ts=1741082980000 results_cache=true type=histogram_float_gauge msg="Instant query result check failed" err="histogram at timestamp 1741082980000 (2025-03-04 10:09:40 +0000 UTC) has sum 11339.746003 while was expecting 11339.746003"

These errors indicate issues related to histogram bucket observations and mismatches in expected versus actual results.

[AneelkumarAmmanagi]

@narqo Is there any document available that outlines the steps to run the continuous test? If so, could you share it with me?