Skip to content

fix: handle the opLatencies histogram MongoDB 8.3 added - #1356

Merged
ademidoff merged 1 commit into
mainfrom
fix/mongodb-8.3-metrics-and-fcv
Sep 3, 2026
Merged

fix: handle the opLatencies histogram MongoDB 8.3 added#1356
ademidoff merged 1 commit into
mainfrom
fix/mongodb-8.3-metrics-and-fcv

Conversation

@ademidoff

Copy link
Copy Markdown
Member

Closes #1285.

The mongo:latest CI leg started failing when the floating tag rolled to MongoDB 8.3.8. Three tests, two root causes — one of them a real exporter bug that also affects production.

opLatencies.histogram produced colliding series

MongoDB 8.3 added a histogram to serverStatus().opLatencies. On 8.2 that subtree has only latency, ops and queryableEncryptionLatencyMicros; I confirmed this against a live 8.2.12:

$ mongosh --eval 'Object.keys(db.serverStatus().opLatencies.commands)'
[ 'latency', 'ops', 'queryableEncryptionLatencyMicros' ]

Two things about the new array went unrecognised:

  • the field is spelled histogram, while isHistogramPath matched only the plural histograms;
  • its buckets are keyed by micros, while isHistogramBucketSlice accepted only lowerBound.

So it fell through to processSlice, which labels array members by name, stateStr or host — none of which a bucket has. Every bucket then produced an identically-labelled series and the registry rejected all but the first:

gathering metrics failed: 64 error(s) occurred:
* collected metric "mongodb_ss_opLatencies_commands_histogram_micros" { untyped:{value:8}} was collected before with the same name and label values
* collected metric "mongodb_ss_opLatencies_commands_histogram_count" { counter:{value:1813}} was collected before with the same name and label values
...

That is exactly #1285, which reports the same errors from a production exporter against 8.3.1.

Bucket detection now accepts either bound key, and the bound becomes a label as it already did for lowerBound. Each bucket gets its own series, and the bound stops being exported as a measurement of its own.

One judgement call: the label is named micros, after the field, rather than folded into the existing lower_bound. The two are not documented as meaning the same thing, and I could not start MongoDB 8.3 locally to check (see below), so I would rather not assert an equivalence I have not verified. Happy to switch to lower_bound if you know it is the inclusive lower bound.

The singular spelling is now gated by --collector.diagnosticdata-histograms like the plural one. That is a behaviour change worth naming: these bucket series are no longer exported by default. They are not a loss, since today they are dropped by the registry anyway — the only thing they produce is one error per bucket per scrape.

TestFCVCollector had no case for 8.3

exporter/feature_compatibility_version_collector_test.go maps the server version to the FCV it expects. The table stopped at 8.2, so 8.3 fell through to default and expected FCV 8.3, while the CI cluster reports 8.0. Folded 8.3 in with the existing 8.2 case.

For the record, since the direction of a CollectAndCompare diff is easy to get backwards: compare calls diff.Diff(got, want), and godebug marks lines present in the first argument with -. So in

-mongodb_fcv_feature_compatibility_version{version="8.0"} 8
+mongodb_fcv_feature_compatibility_version{version="8.3"} 8.3

8.0 is what the 8.3.8 cluster actually reported. I checked that orientation empirically rather than by reading: running this test against a fresh single-node 8.2 gives -…{version="8.2"} 8.2 / +…{version="8.0"} 8, and 8.2 is unmistakably that server's real FCV.

That also shows the existing 8.2 → 8.0 entry is calibrated to the CI cluster topology, not to a fresh node — a fresh 8.2 node reports FCV 8.2. Worth knowing before anyone tries to run this test outside the compose cluster.

Verification

What I could check locally:

  • The new unit tests reproduce the CI failure with no server at all. TestOpLatenciesHistogramBucketsDoNotCollide feeds the 8.3 shape through makeMetricsWithHistograms into a pedantic registry. Against main's exporter/metrics.go it fails with the same error text and the same metric names as CI; with the fix it passes.
  • No regression on 8.2. TestDiagnosticDataCollector and TestAllDiagnosticDataCollectorMetrics pass against a live 8.2.12 replica set, as do the existing TestHistogramMetrics* tests for the lowerBound shape.
  • go build, go vet, make format and golangci-lint run --new-from-rev=main are clean.

What I could not check: no MongoDB 8.3 build starts on this machine. Every 8.3 tag from 8.3.0-rc5 to 8.3.8 refuses:

MongoDB cannot start: Linux kernel versions 6.19 and newer has a known
incompatibility with this version of MongoDB. See SERVER-121912

The Docker VM kernel here is 7.0.12; GitHub runners are below that threshold, which is why CI runs it fine. So the 8.3 half of this rests on the CI log plus the synthetic unit tests, not on a live 8.3 server. The mongo:latest leg of this PR's own CI is the real check.

MongoDB 8.3 added a histogram to serverStatus().opLatencies; on 8.2 the
subtree has only latency, ops and queryableEncryptionLatencyMicros. Two
things about it were unrecognised:

  - the field is spelled "histogram", while isHistogramPath only matched
    the plural "histograms";
  - its buckets are keyed by "micros", while isHistogramBucketSlice only
    accepted "lowerBound".

So the array fell through to processSlice, which labels members by name,
stateStr or host -- none of which a bucket has. Every bucket then produced
the same series and the registry rejected all but the first:

  collected metric "mongodb_ss_opLatencies_commands_histogram_micros"
  was collected before with the same name and label values   (x64)

That is issue #1285, and it also broke TestDiagnosticDataCollector and
TestCollectorWithCompatibleMode on the mongo:latest CI leg once the tag
rolled to 8.3.8.

Bucket detection now accepts either bound key and the bound becomes a
label, as it already did for lowerBound, so each bucket gets its own
series and the bound stops being exported as a measurement of its own.
The label is named after the field it came from -- micros rather than
lower_bound -- because the two are not documented as meaning the same
thing and I could not run 8.3 to check.

TestFCVCollector: the version table had no case for 8.3, so it fell
through to default and expected FCV 8.3, while the 8.3.8 CI cluster
reports 8.0. Folded 8.3 in with the existing 8.2 case.

Closes #1285.
@ademidoff
ademidoff requested a review from a team as a code owner September 1, 2026 11:00
@ademidoff
ademidoff requested review from 4nte and JiriCtvrtka and removed request for a team September 1, 2026 11:00
@ademidoff
ademidoff merged commit b5c5b00 into main Sep 3, 2026
12 checks passed
@ademidoff
ademidoff deleted the fix/mongodb-8.3-metrics-and-fcv branch September 3, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collected Metric was collected before with the same name and label values with MongoDB 8.3.1

2 participants