feat(browser): opt-in OpenTelemetry OTLP metrics export; v1.13.0 - #30
feat(browser): opt-in OpenTelemetry OTLP metrics export; v1.13.0#30harper-joseph wants to merge 2 commits into
Conversation
Add an opt-in metrics path. When `metrics.enabled` and `metrics.otlpEndpoint` are set, the worker folds each stats window — the same delta counts and raw per-render timing samples `logStats` already computes — into OpenTelemetry counters, histograms, and gauges and pushes them over OTLP/HTTP. Design notes: - Off by default. The OpenTelemetry SDK is dynamically imported only when enabled, so a disabled worker loads no OTel code and pays zero cost. - Single integration point: `RenderWorker.logStats` calls `metrics.record()` once per window (reusing the existing aggregation — no new hot-path work). - Histograms record the raw ms samples (not the pre-aggregated percentiles) so fleet-wide quantiles can be re-derived correctly across workers/pods. - Export is periodic and bounded (fire-and-forget): a slow/unreachable collector never blocks, delays, throws into record(), or fails a render, and the shutdown flush is capped so it can't delay container exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces opt-in OpenTelemetry metrics export over OTLP/HTTP for the render worker. It adds OpenTelemetry dependencies, implements a lazy-loaded metrics recorder module to track various performance and resource utilization metrics, and integrates it into the worker lifecycle. The feedback highlights an improvement opportunity in Worker.ts to call this.logStats() during the destroy() sequence, ensuring that any metrics accumulated since the last interval are captured and exported before shutdown.
| // throws). Cap the wait so a down collector's flush retries can't delay container exit; a | ||
| // dropped final window is an acceptable trade at shutdown. AbortController cancels the cap | ||
| // timer once the flush wins so it doesn't linger on the event loop (mirrors the drain above). | ||
| if (this.metrics) { |
There was a problem hiding this comment.
During graceful shutdown or uncaught exceptions, destroy() is called to clean up resources. However, any metrics accumulated since the last 60-second logStats() interval are currently lost because logStats() is not called before shutting down the metrics provider. Calling this.logStats() at the start of destroy() ensures that the final window's metrics are captured, logged, and successfully exported via OpenTelemetry. Wrapping it in a try-catch block ensures that any unexpected errors during stats collection (e.g., during an uncaught exception) do not block the critical browser cleanup process.
| if (this.metrics) { | |
| try { | |
| this.logStats(); | |
| } catch (err) { | |
| logger.warn({ err }, 'failed to log final stats during destroy'); | |
| } | |
| if (this.metrics) { |
… down metrics destroy() shut down the metrics provider without folding in the stats accumulated since the last 60s logStats() tick, so a graceful shutdown/deploy dropped up to a full window of counts/timings from the export. Call logStats() once at the start of destroy() (guarded, before this.metrics is cleared) so the final window is logged and recorded into the OTel instruments ahead of the flush. Addresses gemini-code-assist review on PR #30. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Addressed the review finding in 1ccc4e0.
(AI-generated: authored by an agent addressing the gemini-code-assist comment.) |
Summary
Adds an opt-in OpenTelemetry metrics path to
@harperfast/prerender-browser(v1.13.0). Whenmetrics.enabled+metrics.otlpEndpointare set, the worker folds each stats window into OTelcounters/histograms/gauges and pushes them over OTLP/HTTP to a collector (Grafana Alloy). Off by
default — a disabled worker loads no OpenTelemetry code and behaves exactly as before.
Why
The render fleet already emits rich per-window stats via
logStats, but only as pino logs. Thislets those numbers land in Grafana Cloud as real Prometheus metrics (via the Alloy stack already in
the cluster) so we get fleet-wide dashboards/alerting — including correct fleet-wide latency
quantiles, which per-worker pre-aggregated percentiles can't give.
Chosen over a logs→metrics or scrape approach because instrumenting the shared library (a) versions
the metric definitions with the code, (b) benefits every customer (kohls/stbhb/mcy…) on their next
bump, and (c) works with the multi-process fork model (each worker pushes independently). The fleet
is CPU-constrained (Guaranteed 14-core pods, saturated at peak), so the export is engineered to cost
effectively nothing on the render path (see below).
Design / where to look
src/metrics.ts(new) — lazy-imports the OTel SDK (dynamicimport()), so nothing loadsunless a worker enabled metrics. Counters + explicit-bucket histograms (
render.duration,render.phase.duration) + observable gauges. Export is periodic and bounded/fire-and-forget:a slow or down collector can never block, delay, throw into
record(), or fail a render.src/Worker.ts— single integration point:logStats()callsmetrics.record()once perwindow, reusing the aggregation it already computes (no new hot-path/per-render work).
destroy()flushes on shutdown with a 2s cap so a down collector can't delay container exit.
src/index.ts— constructs the recorder only when enabled (the sole thing that loads OTel);init failure logs and continues without metrics. Redacts
metrics.headersvalues from the config log.src/settings.ts— newmetricsoption + resolution; defaults to disabled.Attention / self-review notes (⚠️ cross-model CLI review couldn't run locally)
The Codex/Gemini CLIs weren't usable in my environment (Codex auth token expired; Gemini has no API
key), so the mandated two-model review did not run — please lean on the AI PR reviewer + a human
pass. Areas I'd focus a reviewer on:
AggregationTypeviews,resourceFromAttributes,PeriodicExportingMetricReader) — 2.0 changed these; verified against the installed type defs and unit tests.record()is guarded, shutdown flush is capped.)@opentelemetry/{api,sdk-metrics,resources,exporter-metrics-otlp-proto}. The 3npm audithighs (undici,ws,js-yaml) are pre-existing (undici direct dep; ws/js-yaml via mqtt/puppeteer), not introduced here.Testing
test/metrics.test.ts(record + shutdown never throw, incl. collector-unreachable andall-zero/null-gauge windows) and settings coverage.
node --testgreen (15/15).tscbuild, eslint, and prettier all clean. Verified the compileddist/keeps the importsdynamic (no static
@opentelemetryinindex.js/Worker.js) — disabled path loads nothing.renderOnce.test.tsneeds a real Chrome (not installed in my sandbox) and is unaffected by this change.Follow-ups (not in this PR)
metrics.enabled/otlpEndpointfrom env (after this releases as a tarball).enable one in the Grafana
k8s-monitoringHelm release (routing to the existing Grafana Cloudremote_write) is prepared and will be handed to devops.
🤖 Generated by an agent (Claude). The cross-model CLI review step was skipped due to local tooling auth — please ensure a review runs before merge.