fix(macos): one transient empty read discarded the whole freshness verdict - #164
Conversation
…rdict `verify_macos_share_freshness` is a release-gated self-test, and it was non-deterministic. Measured on an Apple Silicon host, one machine, one VM, three runs: - issued as separate ssh connections it reproduced staleness and its `after` read was correct -- the "verified" verdict - run as a tight twelve-iteration loop the `before` read came back ALREADY CURRENT every single time -- the "no longer need" verdict, 12 for 12 - on the real `up --macos` reconcile path, twice, the `after` read came back EMPTY -- and the guest was demonstrably alive, since a manual `cat` of that same probe from inside it returned the correct current value moments later The third outcome is what `make e2e` hit. A check whose answer depends on scheduling teaches the operator to ignore it, so this makes the two value-bearing reads survive a transient empty answer. ONLY on empty, and that limit is the point. A NON-EMPTY but WRONG `after` IS the SELF-TEST FAILED verdict -- the finding this tripwire exists to produce. Retrying that would hand a broken guest three chances to eventually agree and convert "the mitigation no longer works on this guest OS" into a clean pass: a check that cannot fail, which is the #131/#136/#137 shape exactly. The new loop breaks on the first non-empty answer whatever it says. Three attempts and 0.2 s are borrowed, not invented -- three matches `_MACOS_SWEEP_TRIES`, 0.2 s matches the guest-poll backoff already used here. Worst case this adds 0.8 s to `up --macos`. The `before` read is deliberately NOT retried: its expected value is the stale one, so extra attempts would bias the run toward "no longer need" for a reason that is this function's own latency. THE CAUSE OF THE EMPTY READ IS NOT ESTABLISHED and nothing here claims it is. Every reproduction that read the probe successfully differed from `up` in ways never narrowed down -- most obviously that `up` runs a full-share sweep immediately before this function, which none of them did. This defends against the observation, not against a theory. Also, two messages that sent an investigator the wrong way: - the empty-`after` warn said "the guest stopped answering mid-check". The guest was answering. Naming a cause that was never measured cost a real debugging session chasing gvproxy. It now states what was observed, and the warm-read warn likewise stops reporting an empty answer and a wrong answer as the same observation. - `tests/e2e_macos_vm.sh`'s `*)` arm reported "no verdict in the up output at all" for four outcomes, three of which DO print "Could not verify the shared-file refresh: ...". It now surfaces that line and claims silence only when there is none. Still a `fail` -- this is a release gate. And the tripwire's own two `python3` probes are now PATH-pinned like the sweep's. #162 pinned the mechanism and its commit message said "The sibling's pin has the same hole" -- this is the sibling. The disclaimer carries over unsoftened: it stops a PATH-resolved `python3`, not a shell function or alias. tests/42 grew a section for the retry and its limit. The arm that matters is the wrong-but-non-empty one: its mode list is one longer than the run needs, with the CORRECT value parked where only a retry could reach it, so if a future change ever retries a non-empty answer that arm flips to "verified" and fails loudly. Two fixture defects surfaced while writing it and are fixed here: `survives rcfail rcfail rcfail rcfail` named a mode that was never a case arm, so it fell through and asserted non-zero-return protection against a stub returning ZERO; and a mode list that ran out used to fall back to `fresh`, letting an under-counted arm test a healthy guest while claiming otherwise. Exhausting the list is now recorded and checked once at the end. Verified: `make offline-tests` ALL GREEN (30 scripts), `make shellcheck` clean, `bash -n augur` OK. Not verified here: the live macOS E2E, which needs an Apple Silicon host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🧭 Egress drift reminderThis PR changes the egress core ( If this change alters the egress posture, please:
Non-blocking reminder. It does not verify correctness (that is the egress fail-closed E2E + |
Live E2E on Apple Silicon: 28 passed · 0 failed · 1 skipped
This closes the one item both PRs listed as "not verified here". About the earlier red runAn intermediate run failed After That underlying seam ordering is a real latent defect and gets its own PR: the |
1 participant
Fixes the
make e2efailure seen while validating #163 — which turned out to beunrelated to that PR and to be a real defect in
verify_macos_share_freshness.What was measured
One Apple Silicon host, one VM, three runs — three different outcomes:
beforeafterShared-file refresh verifiedno longer need→ E2E skipsup --macosreconcile, twiceCould not verify …→ E2E failsThe guest was demonstrably alive in the third case: a manual
catof that sameprobe from inside it returned the correct current value moments later.
So a release-gated self-test was non-deterministic, and one transient empty read
discarded the whole verdict. A check whose answer depends on scheduling teaches
the operator to ignore it.
The fix, and its limit
The two value-bearing reads (
warm,after) now survive a transient emptyanswer — 3 attempts, 0.2 s apart, both numbers borrowed rather than invented
(
_MACOS_SWEEP_TRIESand the guest-poll backoff already used here). Worst casethis adds 0.8 s to
up --macos.Important
Only on empty, and that limit is the whole point. A NON-EMPTY but WRONG
afteris theSELF-TEST FAILEDverdict — the finding this tripwire existsto produce. Retrying it would hand a broken guest three chances to eventually
agree, converting "the mitigation no longer works on this guest OS" into a clean
pass: a check that cannot fail, which is the #131/#136/#137 shape exactly. The
loop breaks on the first non-empty answer whatever it says.
beforeis deliberately not retried — its expected value is the stale one, soextra attempts would bias the run toward
no longer needfor a reason that isthis function's own latency rather than the platform's behaviour.
The cause of the empty read is not established, and nothing here claims it is.
Every reproduction that read the probe successfully differed from
upin waysnever narrowed down — most obviously that
upruns a full-share sweep immediatelybefore this function, which none of them did. This defends against the
observation, not against a theory.
Two messages that sent an investigator the wrong way
afterwarn said "the guest stopped answering mid-check". It wasanswering. Naming a cause that was never measured cost a real debugging session
chasing gvproxy. It now states what was observed. The warm-read warn likewise
stops reporting an empty answer and a wrong answer as the same observation.
tests/e2e_macos_vm.sh's*)arm reported "no verdict in the up output atall" for four outcomes, three of which do print
Could not verify the shared-file refresh: …. It now surfaces that line and claims silence only whenthere is none. Still a
fail— this is a release gate.PATH pin
The tripwire's two
python3probes are now pinned like the sweep's. #162 pinnedthe mechanism and its own commit message said "The sibling's pin has the same
hole" — this is the sibling. The disclaimer carries over unsoftened: it stops a
PATH-resolved
python3, not a shell function or alias.Tests
tests/42gained a section for the retry and its limit (12 arms). The onethat matters is the wrong-but-non-empty arm: its mode list is one longer than the
run needs, with the CORRECT value parked where only a retry could reach it — so if
a future change ever retries a non-empty answer, the arm flips to "verified" and
fails loudly.
Mutation-tested. Removing the
breakfrom the retry loop makes 7 arms fail,including
a wrong but non-empty answer is reported as BROKEN. The arms haveteeth; they are not decoration.
Two fixture defects surfaced while writing them and are fixed here:
survives rcfail rcfail rcfail rcfailnamed a mode that was never acasearm. It fell through to the
freshfallback, so an assertion aboutnon-zero-return protection was made against a stub returning zero.
fresh, letting an under-counted arm testa healthy guest while its name claimed otherwise. Exhausting the list is now
recorded and checked once, as the last assertion in the file.
Verified / not verified
Verified here:
make offline-testsALL GREEN (30 scripts),make shellcheckclean (including the strict pass, which covers
tests/e2e_macos_vm.sh),bash -n augurOK, and the mutation test above.Not verified here: the live macOS E2E on Apple Silicon. This host is itself a
VZ guest, so nested virtualisation is unavailable.
AUGUR_TEST_LIVE=1 make e2estill needs a human — that is the run that will confirm the
up --macosreconcilepath now reaches a verdict.
One thing this does not fix
A subagent reported that of four full-suite runs on the unmodified base, one
came back
1/30 script(s) FAILEDand it did not reproduce in three further runs.The script was not identified. I could not reproduce it either. Flagging it
because "
make offline-testsmust end ALL GREEN" is currently not a fullytrustworthy gate, and a rare red here would be easy to misattribute to this
retry. Worth its own issue.
🤖 Generated with Claude Code