NSP clock conversion is wrong by ~170-280 ms under CENTRAL (no fallback, dishonest uncertainty)

[cboulay]

Summary

Clock synchronisation is dramatically worse when reading through Central than when CereLink owns the device. The device-to-host conversion exceeds the harness's 100 ms tolerance by nearly 3x.

Evidence

Same hardware, same session code, soak harness --intense:

Metric	CENTRAL CLIENT	STANDALONE (--native)
clock_uncertainty_ns	10,000,850 (10 ms)	102,750–142,400 (~0.1 ms)
device_to_monotonic vs monotonic()	-280.84 ms (FAIL)	+2.35 / +2.71 ms (PASS)

Roughly 100x worse uncertainty, and a conversion error large enough to fail the harness check outright:

[FAIL] clock/device_to_monotonic: -280.84 ms vs monotonic

Timestamps themselves are fine in both modes — ts/monotonic shows 0 regressions over 4000 packets and ts/nanosecond-scale reports a median inter-packet delta of 33,320 ns, so this is the offset/uncertainty estimation rather than tick-to-ns conversion.

Scope

Independent of instrument count and of the channel-addressing work — it reproduced before and after #195, and on instrument 0 (HUB1), which is otherwise fully functional under Central. So it is not related to #196.

integration.ClockProbeTest.ProbesMaintainUncertainty is intermittent on this bench, which is consistent with a noisy/degraded estimate under Central.

Why it matters

A 280 ms conversion error makes device timestamps unusable for aligning against host-side events in CENTRAL CLIENT mode, which is the only mode available while Central is running.

[cboulay]

Root-caused and addressed in #195 (2d0381c, c3d1bb4). The original diagnosis in this issue is wrong in an important way, so recording the corrected one.

It is not a general Central degradation

Hubs read through Central are fine. Their ~10 ms uncertainty is rtt/2 honestly reporting Central's queue latency (measured probe RTT ~19 ms), and their conversions land at +6 to +10 ms. The −280 ms came from the NSP specifically, for a device-cadence reason.

The NSP's probe replies are quantised, not wrong

It transmits in ~8192-sample blocks and latches NPLAYREP header.time to those boundaries. Consecutive probes 600 ms apart returned timestamps differing by exact multiples of 273.067 ms (= 8192 samples at 30 kHz), and two probes inside one block returned an identical timestamp.

With 80 probes accumulated the spread was 269.7 ms — one whole block. It does not shrink with more samples, because the error is bounded by the block period rather than random, so no averaging or max-picking rescues it.

Central's queue is not the cause: RTT ~19 ms bounds any forward/return asymmetry bias at ~10 ms, nowhere near 280 ms.

Two real defects

1. CENTRAL clients had no fallback. STANDALONE handles a bad-probe device two ways — peers publish offsets to shared memory and are voted on (an NSP explicitly borrows a hub), and DeviceSession feeds a data-packet floor. Under Central neither existed: no peer CereLink process publishes anything, and addDataPacketSample was never called from the CLIENT path (data=0 in every debug line). So the NSP was left with the unusable probes.

Fixed by (a) probing every present instrument and applying the same vote/borrow rules, via a new generic pre-filter packet observer on ShmemSession — the instrument filter itself is untouched, so demux keeps a single rule; and (b) feeding the data floor once per batch from the CLIENT loop, mirroring DeviceSession.

2. Reported uncertainty was not a bound on the error. bestProbe() returned rtt/2, and the data branch returned a flat ONE_WAY_DELAY_ESTIMATE_NS. The NSP advertised ~8 ms confidence for an estimate 170 ms wrong, and a caller checking clock_uncertainty_ns had no way to tell. Both now report max(constant, spread/2) over the glitch-filtered set.

Verification

Same NSP, same moment — the two estimators are not close:

estimator	samples	spread
probes	80	269.7 ms
data floor	320	< 1.4 ms

Conversion quality:

configuration	result
NSP, STANDALONE	unc 700,000 ns, delta +1.7 ms
HUB1, STANDALONE	unc 102,670 ns, delta +0.9 ms
NSP + 2 hubs, CENTRAL	delta +8.4 ms (was −169.8 ms) — via cross-instrument consensus
NSP alone, hubs off, CENTRAL	unc 3,794,263 ns, delta +4.4 ms — via the data floor

The last row is the case with no peer to borrow from, which is the one this issue's fallback work exists for. Note the uncertainty there reports 3.79 ms rather than the old flat 0.7 ms, because the data spread through Central really is wider than direct — the estimate now says so.

Remaining

• CERELINK_CLOCK_DEBUG instrumentation in clock_sync.cpp is deliberately retained for testing and must be removed before Address each instrument independently in CENTRAL CLIENT mode #195 merges.
• The probe path's max(rtt/2, spread/2) is still not directly observed, because on every configuration tested the data floor is selected ahead of unreliable probes. That is the desired outcome, but it means the probe branch's new uncertainty is reasoned rather than measured.