fix: handle the opLatencies histogram MongoDB 8.3 added - #1356
Merged
Conversation
MongoDB 8.3 added a histogram to serverStatus().opLatencies; on 8.2 the
subtree has only latency, ops and queryableEncryptionLatencyMicros. Two
things about it were unrecognised:
- the field is spelled "histogram", while isHistogramPath only matched
the plural "histograms";
- its buckets are keyed by "micros", while isHistogramBucketSlice only
accepted "lowerBound".
So the array fell through to processSlice, which labels members by name,
stateStr or host -- none of which a bucket has. Every bucket then produced
the same series and the registry rejected all but the first:
collected metric "mongodb_ss_opLatencies_commands_histogram_micros"
was collected before with the same name and label values (x64)
That is issue #1285, and it also broke TestDiagnosticDataCollector and
TestCollectorWithCompatibleMode on the mongo:latest CI leg once the tag
rolled to 8.3.8.
Bucket detection now accepts either bound key and the bound becomes a
label, as it already did for lowerBound, so each bucket gets its own
series and the bound stops being exported as a measurement of its own.
The label is named after the field it came from -- micros rather than
lower_bound -- because the two are not documented as meaning the same
thing and I could not run 8.3 to check.
TestFCVCollector: the version table had no case for 8.3, so it fell
through to default and expected FCV 8.3, while the 8.3.8 CI cluster
reports 8.0. Folded 8.3 in with the existing 8.2 case.
Closes #1285.
ademidoff
requested review from
4nte and
JiriCtvrtka
and removed request for
a team
September 1, 2026 11:00
JiriCtvrtka
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1285.
The
mongo:latestCI leg started failing when the floating tag rolled to MongoDB 8.3.8. Three tests, two root causes — one of them a real exporter bug that also affects production.opLatencies.histogramproduced colliding seriesMongoDB 8.3 added a histogram to
serverStatus().opLatencies. On 8.2 that subtree has onlylatency,opsandqueryableEncryptionLatencyMicros; I confirmed this against a live 8.2.12:Two things about the new array went unrecognised:
histogram, whileisHistogramPathmatched only the pluralhistograms;micros, whileisHistogramBucketSliceaccepted onlylowerBound.So it fell through to
processSlice, which labels array members byname,stateStrorhost— none of which a bucket has. Every bucket then produced an identically-labelled series and the registry rejected all but the first:That is exactly #1285, which reports the same errors from a production exporter against 8.3.1.
Bucket detection now accepts either bound key, and the bound becomes a label as it already did for
lowerBound. Each bucket gets its own series, and the bound stops being exported as a measurement of its own.One judgement call: the label is named
micros, after the field, rather than folded into the existinglower_bound. The two are not documented as meaning the same thing, and I could not start MongoDB 8.3 locally to check (see below), so I would rather not assert an equivalence I have not verified. Happy to switch tolower_boundif you know it is the inclusive lower bound.The singular spelling is now gated by
--collector.diagnosticdata-histogramslike the plural one. That is a behaviour change worth naming: these bucket series are no longer exported by default. They are not a loss, since today they are dropped by the registry anyway — the only thing they produce is one error per bucket per scrape.TestFCVCollectorhad no case for 8.3exporter/feature_compatibility_version_collector_test.gomaps the server version to the FCV it expects. The table stopped at8.2, so 8.3 fell through todefaultand expected FCV8.3, while the CI cluster reports8.0. Folded8.3in with the existing8.2case.For the record, since the direction of a
CollectAndComparediff is easy to get backwards:comparecallsdiff.Diff(got, want), and godebug marks lines present in the first argument with-. So in8.0is what the 8.3.8 cluster actually reported. I checked that orientation empirically rather than by reading: running this test against a fresh single-node 8.2 gives-…{version="8.2"} 8.2/+…{version="8.0"} 8, and 8.2 is unmistakably that server's real FCV.That also shows the existing
8.2 → 8.0entry is calibrated to the CI cluster topology, not to a fresh node — a fresh 8.2 node reports FCV 8.2. Worth knowing before anyone tries to run this test outside the compose cluster.Verification
What I could check locally:
TestOpLatenciesHistogramBucketsDoNotCollidefeeds the 8.3 shape throughmakeMetricsWithHistogramsinto a pedantic registry. Againstmain'sexporter/metrics.goit fails with the same error text and the same metric names as CI; with the fix it passes.TestDiagnosticDataCollectorandTestAllDiagnosticDataCollectorMetricspass against a live 8.2.12 replica set, as do the existingTestHistogramMetrics*tests for thelowerBoundshape.go build,go vet,make formatandgolangci-lint run --new-from-rev=mainare clean.What I could not check: no MongoDB 8.3 build starts on this machine. Every 8.3 tag from
8.3.0-rc5to8.3.8refuses:The Docker VM kernel here is 7.0.12; GitHub runners are below that threshold, which is why CI runs it fine. So the 8.3 half of this rests on the CI log plus the synthetic unit tests, not on a live 8.3 server. The
mongo:latestleg of this PR's own CI is the real check.