feat: continuous coverage evaluation + calibration-triggered retraining

[gkneighb]

Problem

continuous_training_loop retrains every RETRAINING_INTERVAL_SEC (default 1800s) regardless of model quality or workload-distribution change. Two things follow:

1. Calibration is only computed inside train(). _calculate_metrics_on_test (training_server.py:717) runs once per retrain, populates ttft_coverage_scores / tpot_coverage_scores / ttft_quantile_loss_scores / tpot_quantile_loss_scores, and the /metrics endpoint serves whatever value was computed at the last retrain. Between retrains, /metrics is stale by construction.
2. No mechanism reads the calibration signal back into the training cadence. All four metric deques are populated and exposed but never consumed for any decision in train() or continuous_training_loop.

So under workload drift the system has a window of RETRAINING_INTERVAL_SEC during which (a) predictions are served from the stale model, (b) /metrics shows pre-drift coverage, and (c) the operator has no signal anything is wrong.

Empirical characterization

I prototyped the proposed change against a synthetic-drift harness (training_server in isolation, no router/vLLM) — a workload script drives /add_training_data_bulk with a baseline regime that flips to a drift regime at a configurable time, and a separate script polls /metrics every 5s. Three experiments under RETRAINING_INTERVAL_SEC=300, MIN_SAMPLES_FOR_RETRAIN=1000:

(A) Production-like timing, 3× slope drift. Drift fires at t=600s; first post-drift retrain at t=910s (310s detection lag). Coverage drops from 88.5% pre-drift to 76.8% post-drift, violation rate from 11% to 23%. Between t=600 and t=910, /metrics shows healthy 89% coverage — drift is silent.

(B) 10× slope drift, 20s retrain interval. Coverage moves only ~5pp (88% → 81%) but quantile loss explodes 69× (0.55 → 39.27) and decays gradually. Coverage is a weaker drift signal than QL for sharp shifts because the test buffer is FIFO size 1000 and binary-above/below loses magnitude.

(C) Same as A but with continuous coverage evaluation + calibration trigger (described below). Drift-to-first-detection drops from 310s to 74s (4×). 9 retrains in the post-drift window vs A's 1.

Proposal

Two changes, both contained to training_server.py:

1. continuous_coverage_loop — a background thread that runs every COVERAGE_EVAL_INTERVAL_SEC (default 30s), evaluates _calculate_metrics_on_test against the currently loaded model + test buffer, and appends to the existing *_coverage_scores and *_quantile_loss_scores deques. No training, just evaluation. This alone makes /metrics fresh between retrains.

2. Calibration trigger on continuous_training_loop's inter-retrain wait. Track EMA of |coverage - quantile_alpha| and quantile_loss / qloss_baseline. When either exceeds its threshold for K consecutive evaluations, signal a threading.Event the training loop checks during its sleep — fires train() immediately instead of waiting out RETRAINING_INTERVAL_SEC.

New env vars matching existing style:

LATENCY_COVERAGE_EVAL_INTERVAL_SEC      (default: 30; 0 disables)
LATENCY_CALIBRATION_COVERAGE_THRESHOLD  (default: 5.0 pp deviation from quantile_alpha)
LATENCY_CALIBRATION_QLOSS_RATIO         (default: 3.0× rolling baseline)
LATENCY_CALIBRATION_TRIGGER_K           (default: 2 consecutive bad evals)
LATENCY_CALIBRATION_EMA_ALPHA           (default: 0.3)

Default RETRAINING_INTERVAL_SEC could be raised once calibration-triggered retraining is in (e.g. to 7200s) since drift events bypass the schedule.

Open design questions for review

• Dual-signal vs coverage-only. Experiment B showed coverage moves ~5pp under 10× drift while QL moves 69×. C also showed coverage and QL diverging during recovery — model accuracy improves (QL ↓) while calibration degrades (coverage ↓) as predictions converge on the drift-regime mean. A coverage-only trigger underweights both ends. The proposal above uses OR-composition of two signals; happy to argue otherwise if there's a preference.

• Warmup grace period. In experiment C the trigger fired twice during cold-start (t=46s, t=106s) on transient coverage swings while the model was still converging on its first thousand samples. Easy mitigations: skip triggers for the first WARMUP_RETRAINS retrains, relax the threshold for the first WARMUP_SAMPLES, or both. Worth picking one before code lands.

• Threshold tuning under different quantile_alpha. Defaults above are for the 0.9 baseline. For 0.99 quantile the natural variance of quantile_coverage is much smaller — threshold should likely scale.

• /metrics cardinality. Adding a quantile_coverage_deviation_ema and qloss_ratio_ema gauge per model (TTFT/TPOT) is 4 extra series. Trivial but flagging it.

Things I considered and am explicitly leaving out

• RandomDropDeque (lines 63-88). Defined but never used — actual buckets and test buffer use plain deque(maxlen=...) which is FIFO. So the "buffer dilutes new-regime data" argument isn't real today. If it ever gets wired in, it deserves a separate issue.

• Feature-distribution drift detection (1D KS-test on incoming samples). Would catch upstream regime change before it shows up as calibration drift. Separate issue — needs a snapshot reference distribution and lifecycle.

• Test-buffer size and refresh rate. Experiment C showed coverage continues degrading for ~500s post-drift even with aggressive triggered retraining, because the test buffer takes that long to fully refresh (FIFO size 1000 @ 2/s test arrivals). A larger buffer slows refresh; a smaller one is noisy. Worth a separate discussion.

Happy to send a PR for the two-change proposal once the design direction lands. Prototype implementation (~80 lines) is on a local branch; can post a draft PR or wait for feedback first depending on preference.

[Whatsonyourmind]

This is a clean writeup, and your A/B experiments already contain the key result: coverage is a low-information drift signal and quantile loss is a high-information one. Worth grounding why, because it settles the dual-signal question and the threshold-scaling question together.

Why coverage barely moves. "Was this point inside the interval?" is one bit per observation. Empirical coverage at target level c has variance c(1−c)/n, so a few-pp shift sits under the sampling noise until n is large — low statistical power by construction. Pinball/quantile loss keeps the residual magnitude, so it reacts to the same drift with far more leverage (your ~5pp vs ~69× is exactly this asymmetry). So: make quantile loss the primary trigger; keep coverage as a slow guardrail, not the detector.

Threshold scaling across quantile_alpha (your (b)). Because the band is c(1−c)/n, a fixed absolute-pp threshold is wrong across quantiles. One SE for c=0.9 ≈ 0.30/√n; for c=0.99 ≈ 0.0995/√n — about 3× tighter. Set the trigger in standard-error units (fire at k·√(c(1−c)/n)) instead of a fixed pp, and it auto-scales with the quantile. The warmup/grace period in (c) then falls out of the same formula — gate triggering on a minimum effective n so the √n denominator isn't tiny at cold start.

The reframe I'd actually push. The 310s detection lag is only a problem because the interval is static between retrains. Adaptive conformal (ACI, Gibbs & Candès 2021) adjusts the level online — α_{t+1} = α_t + γ·(α_target − err_t), with err_t = 1 if the last observation fell outside the interval — so the interval self-widens under drift between retrains and holds long-run coverage with a single scalar (γ ≈ 0.005–0.05 trades reactivity vs jitter). Layer that on the interval and keep scheduled retraining for the point model: you get correct coverage during the lag window instead of racing to detect-and-retrain. It's a few lines on top of the deques you already populate, and your synthetic-drift harness is the right rig to show coverage-vs-time staying flat through the regime flip.

Happy to sketch the ACI update against _calculate_metrics_on_test / continuous_training_loop if useful.

[Whatsonyourmind]

Quick correction/refinement to my own comment above — rather than leave the coverage-vs-quantile-loss claim as an assertion, I simulated it, and the result is more nuanced than "make quantile loss the primary trigger, coverage the guardrail."

Setup: a perfectly-calibrated τ=0.9 predictor (q̂ = z_0.9, y∼N(0,1)), then inject drift and measure each statistic's per-observation signal-to-noise z = |Δmean|/sd, which sets samples-to-detect ∝ 1/z². 5M samples/scenario, fixed seed.

• Pure location shift (+0.1σ): coverage 0.900→0.881, quantile loss 0.1756→0.1764. Coverage hits a 3σ alarm in ~2.7k samples; quantile loss needs ~520k. Coverage wins by ~190× here.
• Dispersion shift (scale ×1.2): coverage 0.900→0.857, quantile loss 0.176→0.216. Quantile loss wins by ~1.9×.

So the magnitude-sensitivity argument is real, but it only makes quantile loss the better detector for dispersion/tail drift. For a location shift, coverage is the matched statistic — it's literally the fraction-below-q̂ that a shift moves — and quantile loss's high variance makes it the weaker detector. The large asymmetry only holds for dispersion-dominant drift, not as a general rule.

Revised takeaway: monitor both. They're sensitive to orthogonal failure modes (coverage → location, quantile/magnitude loss → dispersion). If you must pick one primary trigger, pick it by the drift mode you expect; don't demote coverage by default. The standard-error-units thresholding and the ACI reframe both still stand — ACI is orthogonal (it keeps coverage valid between retrains regardless of which detector fires).

Repro (numpy):

import numpy as np
rng = np.random.default_rng(20260617); N = 5_000_000; tau = 0.90
qhat = float(np.quantile(rng.standard_normal(20_000_000), tau))  # z_0.9
pin = lambda y, q: (tau - (y < q).astype(float)) * (y - q)
yb = rng.standard_normal(N); cb, lb = (yb <= qhat).astype(float), pin(yb, qhat)
def scn(y):
    c, l = (y <= qhat).astype(float), pin(y, qhat)
    zc = abs(c.mean()-cb.mean())/c.std(); zp = abs(l.mean()-lb.mean())/l.std()
    return (3/zc)**2, (3/zp)**2  # samples-to-3σ: coverage, pinball
print("loc +0.1σ:", scn(rng.standard_normal(N)+0.10))
print("scale x1.2:", scn(rng.standard_normal(N)*1.20))

[gkneighb]

Thanks for running the simulation -- it pushed me to stop arguing from summary stats and actually test the claim on the real predictor instead of a toy. I froze a trained XGBoost p90 model (baseline-calibrated), used it as a fixed oracle over baseline + three drift regimes, and computed per-observation SNR / samples-to-3sigma for both the coverage indicator and pinball loss -- your exact metric, 20k samples/phase. The three regimes isolate the failure modes:

• location: shift intercepts only (mean moves, spread fixed)
• slope: scale slope coefficients (my original "drift")
• dispersion: scale the noise sigma only (spread grows, mean fixed) -- the zero-location counterpart to your N(0,1)x1.2

regime	target	coverage	pinball	samples-to-3sigma (cov / pin)	winner
location +0.5sd	ttft	87.7->76.8%	0.546->0.623	135 / 11,148	coverage 83x
	tpot	88.9->77.2%	0.107->0.131	116 / 2,801	coverage 24x
slope x1.3	ttft	87.7->26.6%	0.546->9.36	5 / 23	coverage 4.9x
dispersion x1.2	ttft	87.7->84.3%	0.546->0.649	1,033 / 8,986	coverage 8.7x
	tpot	88.9->85.0%	0.107->0.125	753 / 2,643	coverage 3.5x

The surprise is the dispersion row. Your toy has pinball winning pure dispersion ~1.9x; here coverage wins it 3.5-8.7x. The marginals nearly match your toy (coverage 88.9->85.0% ~ your 0.900->0.857; pinball +17% ~ your +23%), so it isn't a setup bug -- the direction flips through the variance denominator. Coverage is a bounded Bernoulli indicator (sd <= 0.5 always); pinball inherits the heavy-tailed latency magnitudes (the workload's Pareto-alpha1.3 / lognormal features), so its sd is large enough to swamp the noise-induced mean shift. On heavy-tailed inference latencies, coverage's bounded variance is robust where pinball's magnitude-sensitivity becomes an SNR liability -- across all three regimes, not just location.

Caveat I can't fully rule out: my model is slightly underfit (baseline TTFT coverage 87.7%, not 90%), and an underfit model has heteroscedastic tail residuals that also inflate pinball's variance. Your toy assumed a perfectly-calibrated thin-tailed predictor, so part of the gap may be fit quality rather than the feature tails. I don't think a better fit reverses location/slope, and the heavy-tail mechanism is real, but the honest scope is "this predictor + this heavy-tailed workload," not a universal law. I'd happily rerun against a tighter-fit model or your exact threshold settings.

Where this leaves your proposals: the SE-units thresholding is orthogonal and still right -- whatever the primary detector, a fixed-pp trigger is wrong across quantile_alpha. And ACI is still the most valuable reframe regardless of which signal fires, since it keeps coverage valid during the detection lag. The one thing I'd revise from "monitor both, pick primary by expected drift mode": on a heavy-tailed workload, coverage looks like a strong default primary across modes, with pinball as the complement for the well-calibrated/thin-tailed regime -- rather than the other way round. The repro harness (three drift regimes + the frozen-oracle eval) is small; happy to share it so you can poke at the fit confound directly.

[Whatsonyourmind]

That's a great experiment, and it isolates exactly the right variable. The dispersion flip you saw is real and it's the tails, not the underfit — I tested your caveat directly by removing fit quality from the picture: hold the predictor perfectly calibrated (q̂ = the true baseline τ-quantile over 20M draws) and vary only the residual tail-heaviness.

Symmetric Student-t, dispersion = spread ×1.2 (mean fixed), samples-to-3σ:

residual	coverage n	pinball n	winner
Normal	598	323	pinball 1.9×
t₈	703	502	pinball 1.4×
t₅	800	699	pinball 1.1×
t₄	853	912	coverage 1.1×
t₃	969	1547	coverage 1.6×

The Normal row reproduces my toy (pinball ~1.9×); the winner crosses over around ν≈4–5, and past that coverage wins — with a perfectly-calibrated predictor. So the underfit isn't necessary for the flip; it can only amplify it. Your heavy-tail mechanism is the cause: pinball's SNR denominator is sd(magnitude), which grows without bound as the tail fattens, while the coverage indicator is a bounded Bernoulli (sd ≤ 0.5) however heavy the tail.

It survives right-skew too (lognormal, log-scale σ grown ×1.2, median fixed):

lognormal σ	coverage n	pinball n	winner
0.3	598	427	pinball 1.4×
0.6	598	535	pinball 1.1×
1.0	605	754	coverage 1.2×

Same crossover as the tail weight grows — so on a Pareto-α≈1.3 / lognormal latency workload you're well past it, which is why your real-model dispersion row landed coverage 3.5–8.7× even though my thin-tailed toy had pinball winning that regime.

So your revised takeaway is the right one, and I'd state the rule tail-aware: coverage is the robust default primary on heavy-tailed latency workloads — it wins location and dispersion there; pinball only overtakes on dispersion when residuals are near-thin-tailed (ν≳5 / small σ). Location stays coverage-dominated everywhere (104–619× across the t-sweep) — it's the matched statistic for a shift below q̂. Both orthogonal points are unchanged: SE-units thresholding (a fixed-pp trigger is wrong across quantile_alpha), and ACI keeping coverage valid during the lag regardless of which detector fires.

Repro (numpy, ~10s):

import numpy as np
rng = np.random.default_rng(20260618); N=5_000_000; CAL=20_000_000; tau=0.9
def std_t(df,n): x=rng.standard_t(df,n); return x/np.sqrt(df/(df-2))
def draw(df,n): return rng.standard_normal(n) if np.isinf(df) else std_t(df,n)
def s2(q,b,s):
    f=lambda y:((y<=q).astype(float),(tau-(y<q).astype(float))*(y-q))
    cb,pb=f(b); cs,ps=f(s)
    nc=(3/(abs(cs.mean()-cb.mean())/cs.std()))**2
    npb=(3/(abs(ps.mean()-pb.mean())/ps.std()))**2
    return round(nc), round(npb)
for df in [np.inf,8,5,4,3]:
    q=float(np.quantile(draw(df,CAL),tau))
    print(df, "cov/pin:", s2(q, draw(df,N), draw(df,N)*1.2))  # dispersion x1.2

Still happy to sketch the ACI update against _calculate_metrics_on_test / continuous_training_loop whenever you want to wire it in — that's the piece that turns this from detect-and-race into valid coverage through the lag.

[gkneighb]

This is the clean control I couldn't run -- holding calibration perfect and sweeping nu isolates it completely. The nu~=4-5 crossover settles it: the flip is the tail, not the fit, and real latency workloads (Pareto-alpha~=1.3 / lognormal) sit well past it. I'll adopt the tail-aware rule: coverage as the robust default primary on heavy-tailed workloads -- winning location everywhere and dispersion past the tail crossover -- with pinball only overtaking on near-thin-tailed dispersion. SE-units thresholding and ACI both stand orthogonally.

And yes -- I'd like to take you up on the ACI sketch. Wiring alpha_{t+1} = alpha_t + gamma*(alpha_target - err_t) into continuous_training_loop / _calculate_metrics_on_test so the interval stays valid through the lag is the natural next step, and it's the piece that turns this from detect-and-race into continuous validity. Happy to prototype it against the synthetic-drift harness and share results the same way.

[gkneighb]

Prototyped the ACI layer. Short version: it does exactly what you described -- holds coverage through the detection lag at a small width premium. Offline demo, one-sided upper ACI on TTFT (the SLO-relevant direction), XGBoost p90 to mirror the predictor, synthetic location drift, static model retrained at the end of a lag window to model the scheduled-retrain recovery.

Mechanics (per request): score s = y - qhat(x), interval qhat(x) + c_t with c_t = Quantile_{1-alpha_t}(recent scores), update alpha_t <- clip(alpha_t + gamma*(0.1 - err_t), 0, 1) where err_t = 1[y > qhat+c_t]. Base model untouched.

Coverage through the lag (drift active, pre-retrain):

	lag-window coverage	long-run
static p90	45.0%	70.1%
ACI	~90%	90.0%

The static interval collapses to 45% until the retrain fires; ACI holds target throughout, at ~3.2 ms added width (~= the 1.5sd shift it's compensating). The 45% checks out analytically too (a +1.5sd residual shift drops p90 coverage to =Phi(1.28-1.5)=41%).

gamma tradeoff (reactivity vs jitter):

gamma	transient recovery (first 200 reqs)	settled width-jitter
0.005	84.5%	0.22 ms
0.020	88.0%	0.39 ms
0.050	87.0%	1.16 ms

gamma~=0.02 is the sweet spot -- best transient recovery, modest jitter; 0.05 recovers no faster but ~3x the jitter.

Integration sketch (where it'd land in the server): ACI state {alpha_t, score_buffer, gamma} per target on the predictor; step alpha_t in continuous_coverage_loop/evaluate_current_coverage (it already re-evaluates on fresh test data each interval); apply c_t in predict(); expose alpha_t + ACI-adjusted coverage in /metrics alongside the static value. It composes with the calibration-triggered retraining rather than replacing it -- ACI keeps coverage valid between retrains, retraining still sharpens the point model.

Caveats: offline + synthetic single-drift, one-sided. I'd want to validate against the test-buffer cadence and a couple of drift shapes before wiring it in. If the shape looks right to you, I'll turn it into a PR against the loop -- and your earlier offer to sketch it against _calculate_metrics_on_test would be the perfect review checkpoint.

[Whatsonyourmind]

Nice — reran your mechanics end-to-end and the shape holds on my side too:

	lag-window coverage
static p90 (analytic Φ(1.28−1.5))	0.414
ACI γ=0.005	0.890 (transient 0.86)
ACI γ=0.02	0.897 (transient 0.89)
ACI γ=0.05	0.890 (transient 0.88)

Same γ≈0.02 sweet spot you landed on — best transient recovery, and 0.05 buys no extra recovery for ~3× the jitter. The update direction, the [0,1] clip, and the one-sided err_t = 1[y > qhat+c_t] against target 0.1 are all correct (miss → α_t down → 1−α_t quantile up → interval widens). Base-model-untouched + compose-don't-replace is exactly right.

One integration gotcha worth baking into the PR — rebase the score buffer when the retrain fires. At retrain qhat jumps to the new calibrated quantile, but the buffered scores were computed against the old qhat, so they're stale by ~the shift. Leave them in and c_t keeps adding the shift the new qhat already absorbed → transient over-coverage. The failure mode is sharp:

post-retrain first-100 coverage:  no-reset (stale buf) = 1.000   vs   reset = 0.950   (target 0.90)

So on the retrain hook, recompute the buffer's scores against the new qhat (or just clear it and let it refill). Cheap, and it stops ACI from fighting the retrain it's meant to complement.

Two smaller notes:

• L vs γ vs drift timescale. In my run most of the recovery comes from the score-quantile itself recalibrating as post-drift residuals enter the buffer; α_t fine-tunes on top. So buffer length L sets the coarse reactivity and γ the fine — worth picking L relative to your continuous_coverage_loop test-buffer cadence rather than independently.
• Free drift signal. α_t drifting off 0.1 (or c_t widening past baseline) is a calibrated drift detector — when ACI is widening hard, that's the regime flip the issue set out to catch. Exposing α_t/c_t in /metrics alongside static-vs-ACI coverage gives you valid-coverage-through-the-lag and the detector in one signal, closing both halves of feat: continuous coverage evaluation + calibration-triggered retraining #19.

Shape looks right — turn it into a PR. Happy to review against _calculate_metrics_on_test / continuous_coverage_loop / predict() once it's up; the retrain-rebase and the /metrics exposure are the two spots I'd look at hardest.

[gkneighb]

Turned the ACI prototype into a PR: #33 (stacked on #32, the continuous-coverage loop it builds on). Opt-in (ACI_ENABLED=false), one-sided upper ACI in a small aci.py, with the integration hooks against continuous_coverage_loop / predict() / the retrain path, plus alpha_t/c_t in /metrics.

Your buffer-rebase catch is in and verified -- a live-server smoke confirmed offset_ms -> 0 and alpha_t -> target exactly when a retrain fires, so no stale over-coverage. Unit tests cover the state machine; the offline demo + your reproduction cover the mechanics.

Two things I flagged for your review in the PR, both from this thread: the batched alpha_t update (per coverage-eval interval vs the per-request prototype -- does the cadence look right?), and that /metrics coverage is still the static, pre-offset value (good as a drift signal, but I can add ACI-adjusted coverage too if you'd prefer). #33 is in draft until #32 lands; happy to take it out of draft whenever you want to dig in.