a run that never reaches the app under test still reports "no violations" and exits 0.
seen while working on #80. a launch flake left the emulator on the home screen, and sanderling then fuzzed the launcher for 200 steps in 65 seconds with nodes=2 per snapshot, finished, and reported a clean run. exit 0, no violations, a trace full of nothing.
nothing in the runner's own output says the run was worthless. the numbers that would have given it away are all present and none of them is checked: 200 steps in 65s is ~0.3s/step against a normal 2.2s, the hierarchy held 2 nodes throughout, and the app's package never appeared. a person reading "run complete: 200 steps / no violations" has no way to tell this from a healthy clean run.
the folio ci gate would have caught this specific case, because folio-run.sh greps the trace for AddTransactionScreen and fails when it is absent. but that is folio's own gate doing it, per spec and per platform. the runner reports a clean bill of health for a run that observed nothing, and every other spec has to reinvent the same check or inherit the same blind spot.
this is the same shape as the replay-ui vacuity problem: exit 0 means "no property returned false", not "anything was checked". that one got fixed by counting judged and declined per property in the job summary. the run-level version wants something similar but cheaper, and the honest signal is probably not a single number.
candidates, none of them obviously right, which is why this is an issue rather than a patch:
- the app under test never being the foreground package for the whole run is unambiguous and platform-specific
- a hierarchy that never exceeds a handful of nodes across every step
- steps that are an order of magnitude faster than the run's own median, which suggests nothing was ever waited for
- the trace never containing more than one distinct screen
worth deciding deliberately rather than bolting on a heuristic. the failure mode to avoid is a check that fires on a legitimately small app and trains people to ignore it.
a run that never reaches the app under test still reports "no violations" and exits 0.
seen while working on #80. a launch flake left the emulator on the home screen, and sanderling then fuzzed the launcher for 200 steps in 65 seconds with nodes=2 per snapshot, finished, and reported a clean run. exit 0, no violations, a trace full of nothing.
nothing in the runner's own output says the run was worthless. the numbers that would have given it away are all present and none of them is checked: 200 steps in 65s is ~0.3s/step against a normal 2.2s, the hierarchy held 2 nodes throughout, and the app's package never appeared. a person reading "run complete: 200 steps / no violations" has no way to tell this from a healthy clean run.
the folio ci gate would have caught this specific case, because folio-run.sh greps the trace for AddTransactionScreen and fails when it is absent. but that is folio's own gate doing it, per spec and per platform. the runner reports a clean bill of health for a run that observed nothing, and every other spec has to reinvent the same check or inherit the same blind spot.
this is the same shape as the replay-ui vacuity problem: exit 0 means "no property returned false", not "anything was checked". that one got fixed by counting judged and declined per property in the job summary. the run-level version wants something similar but cheaper, and the honest signal is probably not a single number.
candidates, none of them obviously right, which is why this is an issue rather than a patch:
worth deciding deliberately rather than bolting on a heuristic. the failure mode to avoid is a check that fires on a legitimately small app and trains people to ignore it.