OpenTelemetry metrics: SQL thread saturation, query latency, write queue depth

[asg017]

🤖 Claude-generated PR description

Adds OpenTelemetry metrics to the phase-1 stack, alongside the spans from #2862 and #2863. Stacked directly on #2863.

Why metrics belong in phase 1

Spans answer "why was this request slow?" — but the most operationally important Datasette question is a level, not an event: am I saturating my SQL threads? num_sql_threads defaults to 3, and when they're full requests queue silently — no error, no slow query to point at, just a db.query span whose duration quietly includes the wait. datasette.sql.threads.queue_depth sustained above zero is the one number worth alerting on, and no amount of span work surfaces it.

Metrics also survive sampling: an operator keeping 1% of traces still keeps 100% of every histogram, and a rate (how often are queries killed at the time limit?) cannot be recovered from sampled spans at all.

Like the span work, this uses opentelemetry-api only — core records measurements, but without an SDK MeterProvider installed every one is a no-op. A plugin (or opentelemetry-instrument) supplies the provider and exporter; a working /-/metrics Prometheus plugin exists and runs against this branch, which is what makes e.g. Fly.io's free managed Prometheus+Grafana a three-line fly.toml deployment story.

The metrics

name	kind	what it answers
db.client.operation.duration	histogram (s)	query latency distribution, split by database, read/write, and error.type
datasette.write.queue_wait	histogram (s)	how long writes sat behind the single write thread
datasette.sql.queries.interrupted	counter	queries killed at the configured time limit (expected short-budget timeouts, e.g. facet suggestion, are excluded — same semantics as the span errors in #2862)
datasette.sql.threads.limit	gauge	the num_sql_threads ceiling
datasette.sql.threads.queue_depth	gauge	the alerting metric — reads waiting for a free thread
datasette.sql.queries.pending	gauge	reads submitted and not yet complete, per database
datasette.write.queue_depth	gauge	writes queued per database
datasette.connections.open	gauge	open SQLite file connections, per database

Gauges observe live Datasette instances via a weak registry (callbacks never take request-path locks). The internal database is deliberately included — permission checks hit it on essentially every request.

Histogram buckets are explicit, on purpose

All histograms declare unit="s" with explicit boundaries (0.0001 … 10 seconds). Without this, OpenTelemetry's default millisecond-scale buckets put every SQLite-speed measurement in a single bucket and histogram_quantile() returns noise while count/sum/min/max all stay correct — which is exactly why the bug is easy to ship without noticing. The bucket spread is pinned by a test.

Relationship to the hand-rolled tracer

This PR no longer depends on the tracer-removal PR (#2864, now closed): the metric recording call sites in datasette/database.py sit inside the OTel span blocks, which remain nested inside the existing trace() wrappers. Nothing about ?_trace=1 or trace_debug changes here, and removing the hand-rolled tracer stays available as a future standalone PR.

Commits

1. Add OpenTelemetry metrics for SQL thread pool saturation and query latency — the instruments, recording call sites, and the live-instance gauge registry.
2. Register metrics and give histograms bucket boundaries suited to seconds — the metric registry entries, shared bucket boundaries, and a cog-generated "Metric reference" section in the internals docs.
3. Check metric attributes in the registry conformance test — every emitted metric's attributes are pinned against the registry, including a real (unexpected) interrupt via a 5ms time limit.
4. Document metric exemplars — spans and metrics correlate out of the box when both are enabled; documents what exemplar support exists and where.
5. Apply black to the metrics additions.

Docs

docs/internals.rst gains a metric reference table (cog-generated from the registry, so it can't drift) and exemplar documentation. The demos/otel/ additions from an earlier version of this branch were dropped along with the demo PR #2894.

🤖 Generated with Claude Code

https://claude.ai/code/session_012U7coQfVu8nK2R4q2mCULA

[codecov]

Codecov Report

❌ Patch coverage is 0% with 126 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (1e44c3d) to head (9e5a402).

Files with missing lines	Patch %	Lines
datasette/telemetry.py	0.00%	75 Missing ⚠️
datasette/telemetry_registry.py	0.00%	26 Missing ⚠️
datasette/database.py	0.00%	23 Missing ⚠️
datasette/app.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                         @@
##           asg017/otel-phase1-2-request-span   #2897    +/-   ##
==================================================================
  Coverage                               0.00%   0.00%            
==================================================================
  Files                                     75      75            
  Lines                                  12625   12738   +113     
==================================================================
- Misses                                 12625   12738   +113     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.