record what the model picker did, and make both policies see the same actions - #74
Merged
Conversation
An LLM-in-the-loop evaluation has to report tokens per action and cost per defect, and the client discarded both counters. Served model is recorded separately from the requested one because a router can substitute a differently-priced variant. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
llm-calls.jsonl carries the prompts as sent, the candidate list as the model saw it, the screenshot reference, the raw response, tokens, latency and how the step ended. It sits beside trace.jsonl rather than inside it because every trace line already carries a full hierarchy and both the replay server and the campaign summarizer scan all of them; folding prompts in would grow the lines those readers parse for data neither reads. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
It lags the runner's current step whenever a transitional tree caused an observation to be skipped, which is exactly when the model is shown an older screen than the step it is choosing for. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
The strict echo-skip left only a logger.Warn, so a step the guard discarded was indistinguishable in the trace from a picker that legitimately declined. Any yield or actions-per-hour figure computed from model traces mixed the two. Every path that ends a step without a model-chosen action now records its own outcome. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
A step could carry a next_action that the foreground guard or an apply error stopped from running, and nothing said so. An executed-action count read off trace.jsonl included actions that acted on nothing. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Steps to first violation with clean runs right-censored at the budget, since per-run yield is a binary at 11 to 45 percent and separating two arms on it would need roughly 80 runs per arm. Kaplan-Meier, log-rank, Wilcoxon rank-sum with Vargha-Delaney A12, Holm within each family. A hand-rolled log-rank that is subtly wrong is a silent-wrong-number generator and would be believed, so every statistic is validated against a published worked example with the source named in the test: R survdiff on aml, Freireich 6-MP, Hollander and Wolfe 1973 for the rank sum, printed p.adjust output for Holm. Two could not be: the k>2 log-rank, guarded by calibration instead, and the tie-corrected variance, checked against an exact permutation variance. Failed and timed-out runs are excluded as missing data and counted by reason, never treated as censored observations, which would bias the result. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Candidates takes the label source as an argument rather than storing it, which is what keeps the asymmetry structural: the seeded picker selects by index and never calls Candidates, so the mode cannot reach it. That asymmetry is load-bearing, because it makes the two seeded cells of the factorial a manipulation check with identical draw streams. The identifier ladder deliberately has no text rung. A fallback that reached for text would silently turn one arm back into the other. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Recorded for seeded runs too, unlike model and instructions. Without it the two seeded cells are indistinguishable in the artifact and the manipulation check cannot be grouped. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Unknown values are rejected at parse time rather than falling back to the default, matching the generator check: a campaign that completes with the wrong arm and a correct-looking output directory is worse than one that fails. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
The dedup key was the rendered description, which embeds the label, so two distinct controls sharing a visible label collapsed to one entry and the survivor carried the first one's action. The second control was not mislabelled, it was absent from the candidate list, so no policy could reach it. Two scrollable containers collapsed the same way, leaving the second unscrollable. The key is now the executable Action struct itself plus whether the model supplies the typed text, so a new Action field cannot silently fall out of it. Descriptions may now repeat; the numbering disambiguates and the echo guard is index-anchored, not description-anchored. This also makes the label source a pure observation-channel change. It was not one before: the label fed the dedup key, so the two arms of the labelling factor enumerated different-sized candidate lists, in both directions depending on the screen. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
applyAction could return nil without calling the driver, so the trace showed an action that looked executed and acted on nothing. Six paths did it: a tap, double-tap or long-press whose coordinates do not resolve and which carries no selector, a long-press whose selector is stale, an empty key press, and a zero-duration wait. It now reports whether it dispatched, and the runner records the reason and clears lastAction so the verifier never attributes the next state to an action that did not run. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Descriptions can now repeat after candidates dedup by what they execute. The guard is index-anchored, so this pins that a repeated string cannot make it misfire in either direction. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
A step where the policy declined has no action, and a step whose action was never dispatched did nothing. Both were being counted as actions by everything downstream. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Defects per thousand actions counted every step, including steps that chose nothing and steps whose action was never dispatched. The inflation is policy-dependent, so it does not cancel between arms: on the fixture campaign the model arm's yield was reported at 60.3 per thousand against a true 120.7, because half its steps did nothing. A runs.jsonl without the count is refused by name and line rather than read as zero actions, which would report every per-action rate wrongly. The report also carries steps beside actions now, so the gap is visible rather than folded into a denominator. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
The authored descriptor path had no parity guard and diverged from the wire format on almost every verb. A Wait lost its duration and was skipped as a zero-duration wait. A Scroll lost its endpoints and its 250ms. A target that resolved to nothing became a tap at the origin, a phantom focus tap, or a swipe to (0,0) instead of being dropped. An authored target object with no x property panicked the whole run at candidate enumeration: ToInteger was called on a nil goja.Value. A target on the screen origin is still kept, so the drop rule cannot swallow it. Builtins were never affected. They serialize through the same path the seeded arm uses, which the existing policy parity test covers. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
serializeAction sent the container's own point as both endpoints, so an
authored Scroll({in, direction}) reached the driver as a drag from a point to
itself and did nothing, on the seeded arm. The wire now carries the selector
and leaves the drag to the runner, which sizes it from the container's bounds
and has always had tested support for it that nothing could produce.
No rng runs in the serializer, which lowers an already-drawn action, so the
draw stream does not move. Builtin scrolls compute both endpoints and their
bytes are unchanged.
Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
Compares the recorded driver calls across 13 authored shapes. The builtin path had a parity guard and the authored path had none, which is why it drifted on almost every verb. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
idPrefix: is id: with starts-with in place of equality, so a list whose
rows are named <role>_<record id> is reachable by the durable half. The
Android package prefix is skipped the same way id: skips it.
Routing both prefix kinds through matchAttr also makes the object form
work: {descPrefix: ...} matched nothing on the native side while the web
runtime honoured it.
The DOM has no package prefix, so the native rule reduces to [id^=]. Both prefix kinds now go through the one key table, which drops the separate descPrefix branch that string and object selectors each carried.
Binding it at module scope bound it to whenever the module was first imported, so a test file that imported the runtime before setting SANDERLING_SEED froze the seed at zero for every file after it. The bundler still replaces the expression with a literal.
Selector matching is written once per runtime: internal/hierarchy over the dump, web-runtime.ts over the DOM. Nothing made the two agree, and a selector that resolves on one and not the other is silent, since an empty match yields no action and the run still passes.
The object form fell through to the raw attribute map, which carries no
id or desc key on any platform, so {id: "save"} matched nothing while
"id:save" matched. The repo's own web spec uses the object form thirty
times. Both forms now resolve through one switch.
Adds the accepted-key list and UnknownSelectorKeys with it, since the
same silence hides any mistyped key. A key some element carries is always
accepted, so raw driver attributes stay reachable.
a multi-key object selector concatenates its parts into one compound,
and a type selector is valid only at the head of one, so {id, tag} built
'[id="amount"]input' and querySelectorAll threw. whether a spec got an
exception or an element depended on the order its author wrote the keys
in.
internal/hierarchy writes the attribute a selector matches on only where
the producer stated the flag, so a state emitted as null is one no
selector can ask about: {clickable: false} and {enabled: false} matched
nothing at all against a web dump while matching on android, which
states every flag both ways. only secure stays three-valued.
the fixture differs one state at a time: a disabled button and an aria-disabled role control, a box ticked by script with no checked attribute beside one cleared by script that has it, and a select whose first option is selected without the markup saying so anywhere. half the states are asked inside one container, because a state the whole page has an opinion about answers with most of the document and a want list nobody can check by reading.
…cers the target enumeration carries none of the three, so they reach a spec through the ax handle alone, and a selector naming one of them resolves against that same reading. the shadow fixture holds the focused control inside its shadow root, where document.activeElement names the mount element and only a producer that descends finds the field.
the head renders nothing, so the hierarchy dump drops it and so does the enumeration the picker walks, but a selector still resolved into it: a whole-page findAll answered with <head> and <title> here and with neither on the goja host, which is a divergence the moment a state selector asks a question every element has an answer to.
adb falls through to the local server when ADB_SERVER_SOCKET is unset, and claims the only device attached there. That could be a personal handset, and a run installs the app, clears its state and fuzzes it. Every recipe that touches a device now resolves the target through _require-device, which only picks on its own when a single local emulator is all adb sees.
a compound object selector dropped text and matched on the other keys
alone, so {testTag: "Row", text: "Alice"} selected every row carrying
the tag where internal/hierarchy selects the one row the author named.
matching more than the spec said is silent: the find lands on a row
nobody wrote and every property over it still passes.
text is answered against the element the way the boolean states are,
since css cannot ask what an element's text says and the xpath that can
cannot ask about the rest, and the innermost rule now holds over what
the whole selector matched, where internal/hierarchy holds it. a
text-only selector still compiles to the same innermost xpath.
object keys iterate in insertion order, so the order the author wrote them in decided what a compound selector meant. the innermost rule is pinned over the whole selector's matches: a row whose badge carries the class and the text both is dropped, one whose badge carries the text alone is kept, and a state key is anded before either.
one page, both resolvers, text written before and after the key beside
it. the object form now encodes its keys in the order the filters state
them rather than the order a map iterates, so both orders are asked.
the row and the badge under it share a class so the innermost rule has
something to drop, and {text, clickable} pins that text is anded before
that rule runs: the innermost element carrying "January" is the option,
and the select is the only element that is both.
the object selector section said every pair must match without saying where the innermost rule then lands.
className is an accepted selector key that no producer writes: android
reports the view class, ios the element type and the chrome dump
el.className, all of them under `class`. With no alias onto that key the
selector matched NOTHING here on every platform while the web runtime
resolved it against the live DOM, so {className: "status"} named the row
and the badge on one host and no element at all on the other.
The failure is silent: the key is accepted, so no unknown-key error
fires, and a property over the element that was never found passes
having checked nothing.
one page, both resolvers, the two names for the one attribute. class is asked beside className so the pair is pinned to the same elements rather than each to itself: the row and the badge under it both carry it.
this host answers both names against the live DOM and internal/hierarchy now aliases the second onto the first, so a name dropped from the table here would match nothing on web while the dump still answers it.
the key was already typed on the spec surface and already resolved on web, and the alias table said nothing about which attribute it reads.
label and accessibilityLabel aliased onto accessibilityText alone, which only the ios sidecar writes, and alias expansion is ONE level: the hop from accessibilityText to content-desc was never taken, so both keys matched nothing on android and on the chrome dump, which write the fact under content-desc. ariaLabel and contentDescription aliased onto nothing at all and matched nothing anywhere. The web runtime resolves all four against the live DOM, so a selector naming a field this way found it on one host and no element at all on the other. The keys are accepted, so no unknown-key error fires, and a property over the element that was never found passes having checked nothing. Each name lists both keys rather than chaining through accessibilityText: transitive expansion would silently widen every existing key at once.
Compose for Web writes a test tag as data-testid, which is what the web runtime resolves both names against. testTag aliased onto the three identifier keys and not that one, and testID aliased onto nothing at all, so a tag the web runtime found on every row of a list named no element here and every property over it passed vacuously.
…e DOM identifier, accessibilityIdentifier, accessibilityText and elementType are the names ios writes four facts under, and internal/hierarchy aliases each onto the key the other producers write. This table listed none of them, so each fell through to a raw attribute lookup and built [accessibilityIdentifier="summary_card"], which no element carries. Every one of them resolved against the dump on the goja host and named nothing here. The keys are accepted, so no unknown-key error fires, and a property over the element that was never found passes having checked nothing.
…rives Both facts are derived from the live element rather than written by the markup, and matching them as attributes built [editable="true"], which no page carries. Both resolve against the dump on the goja host, so a spec naming a field or a scroll container that way found it there and no element at all here, with no unknown-key error to say so. Each reads the same function the fact is derived with, so a selector cannot name an element this host calls something else: the handle, the picker's target list and the editable selector all go through isEditable, and scrollable reads the overflow test collectTargets reads. scrollable false names nothing rather than every element that does not scroll: both producers state the fact only where it holds, the way an element that is no field at all answers to neither value of secure.
one page, both resolvers, the ten names that resolved on one host only. each alias is asked beside the key it resolves through, so the pair is pinned to the same elements rather than each to itself. the page grows a container that overflows its box and a neighbour that does not, because scrollable is derived from the box: without one the only scrolling element on the page is the document root, whose answer moves with the window.
Every native dump writes the rectangle out as a string under bounds, and no DOM element carries an attribute of that name, so the key resolved against the dump and matched nothing on web on every page there is. It is accepted, so no unknown-key error said so, and no mapping can be invented for it: there is no DOM fact to map it to. Off the accepted list the web runtime raises the unknown-key error instead of matching nothing in silence, and the key still resolves wherever a producer writes it, through the escape hatch every other raw attribute already uses: a key some element carries is a key that can match, on both sides.
the table listed neither name for the accessible label that a web page writes, nor the key a web test tag lands on, and said nothing about elementType. editable and scrollable are boolean states like the rest, and scrollable is the one of them the platforms state only where it holds. bounds is a raw driver attribute rather than an accepted key.
it described the alias table as it stood before the label and test-tag names reached the keys android and web write, and said nothing about expansion being one level deep, which is why each name has to list every key rather than hop through another alias.
hintText and placeholderValue are the accessible-name ladder, derived from the live element, and compiling them to [placeholder="..."] made them name the wrong field or none at all. A field labelled by an aria-label or a bound <label> carries no placeholder, so it resolved against the dump on the goja host and reached nothing here; one carrying both answered to its placeholder here where the dump answers to its aria-label, which lands a find on an element nobody named. Both keys read the same fieldHint elementHandle and the hierarchy dump (internal/driver/chrome/driver.go) derive the fact with, so a selector cannot name a field this host calls something else. An empty hint names nothing rather than everything that is no field: both producers write the fact only where the ladder answered. placeholder stays the attribute the markup writes, which is what the dump carries under that name too, so a field whose hint is something else still answers to it on both hosts.
TapSelector is a third resolver, and it built [placeholder="..."] for hintText and placeholderValue too. Now that both matchers read the accessible-name ladder, that CSS names a field whose hint is its aria-label and whose placeholder happens to carry the value, which is an element neither matcher named. No CSS says what the ladder says, so both keys fall through to a match that reaches nothing and the step fails naming the selector, the way every other derived key in this file already does. A selector reaches here only where the dump resolved it to no coordinates at all.
The page gains four fields that differ one rung at a time: a bound label, a placeholder, a placeholder an aria-label outranks, and the name the form gives the field. Only the placeholder rung was reachable before, so hintText and placeholderValue named a field on the goja host and no element at all on web for the other three, and named the field here and nothing there for the rung the ladder passed over. placeholder was measured empty on both hosts because nothing on the page carried the attribute, which said nothing about it. It now names the field the markup wrote it on and not the field whose hint is its aria-label. The third resolver reads the same selectors: what TranslateStringSelector builds for a hint key has to match nothing over CDP rather than the field carrying the value as a placeholder.
…ribute The web section said the hintText key does not read the ladder on both hosts and told authors to select such a field by attrs.hintText instead. Both hosts read it now, so that instruction is gone rather than left standing beside a newer sentence. placeholder is stated as the attribute the markup writes and nothing more, the tap path is stated as failing by name where no CSS says what the ladder says, and the alias table gains the row it was missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
second batch of experiment instrumentation, plus the fixes that came out of actually running it.
the model picker now records everything per step in llm-calls.jsonl: prompts as sent, the candidate list the model actually saw, screenshot reference, raw response, tokens, latency, and a typed outcome. before this a guard-skipped step left only a log line, so it was indistinguishable from a picker that declined, and any yield number off model traces mixed the two. llmclient parses usage now so tokens per action and cost per defect are computable at all.
new analyze tool does the survival stats: steps to first violation with clean runs right-censored at the steps they actually ran, kaplan-meier, log-rank, gehan-breslow generalized wilcoxon with a censoring-aware a12, exact sign test on the seed-paired contrast, holm. every statistic is validated against a published worked example with the source named in the test. two couldn't be validated that way and say so in the report rather than getting a fixture made up from their own output.
--label-source picks how candidates are named for the model, which is the other factor in the factorial. seeded picker can't see it by construction, and there's a test driving 300 draws per verb to prove the draw stream doesn't move.
then the bug class, which is most of the diff. a bunch of places where something didn't happen and nothing said so.
applyAction could return nil without dispatching, six different ways. steps looked executed and acted on nothing. candidates deduped on the rendered description, which embeds the label, so two controls sharing a visible label collapsed and the second was unreachable by any policy. the authored action path had no parity guard and had drifted from the seeded arm on nearly every verb: authored Scroll serialized the container's point as both endpoints so it was a drag from a point to itself, authored Wait lost its duration, an authored target with no x property panicked the whole run. authored leaves that sample drew from an rng that only exists inside the picker's walk, so every draw collapsed to the first item. that's a hard error now, naming the offending leaf. analyze counted declined and skipped steps as actions, which is policy-dependent bias that doesn't cancel between arms: on the fixture campaign the model arm's yield read 60.3 per thousand against a true 120.7.
folio-web's data-* extractors read keys the web runtime didn't publish, so ledgerTxnCount and ledgerBalance sat at 0 forever. worth noting it broke both directions: the old committed runs report someTransactionExists violated against an app whose ledger was showing $81,963. a blind harness gave us a vacuous pass and a false alarm at the same time.
a run against a spec that registers no properties is refused now instead of driving the app and reporting a green run that judged nothing. --allow-no-properties opts out by name. bundle-check gained the same gate.
the rest came from running the folio-web job rather than from reading code. document.activeElement stops at a shadow boundary, so a compose-for-wasm app reported focus on div#app forever and the new confirmFocus guard could never be satisfied: every InputText step aborted the run after three tries. fixing the shadow descent wasn't enough, because compose never focuses the semantics node at all, it proxies keystrokes through a hidden 1px backing input that is a sibling of the a11y tree. focus is attributed to the field the caret sits in now, in both hosts. folio-web goes 209 steps and convicts on the submit property.
replay-ui's spec was reading dataset-style keys after the runtime switched to raw markup names, so three properties went vacuous and exactlyOneStepIsSelected reported false against a ui that was fine.
tag: selectors matched by substring on the go side, so tag:li resolved to .
a handle answered editable off isContentEditable, which is inherited, so every span inside a contenteditable div called itself typeable. the browser parity test only compared clickable, which is why a unit test caught it and the real browser didn't. it compares editable now too. worth knowing: the handle exposes 16 facts and that test compares 2 of them, so most of this class is still unguarded.
binary and required-flag resolution ranged over maps and returned at the first failure, so omitting three flags told you about one, picked at random, and two identical runs printed different errors. that's why a sweep test reached master as a flake instead of a clean red. four tools plus both sweeps now name all of them in flag order.
new confusion-matrix tool cross-tabulates what the checker said against a blind reviewer's verdict, one cell per implementation. an implementation that failed to build, has no usable run, or no filed verdict is listed as missing data rather than counted as a clean cell. it is undocumented on purpose, same as the other ten tools under cmd/internal-tools.
correcting the earlier version of this description: it said builtin web typing named fields by css class and that elementHandle hardcoded clickable true. both were already fixed on this branch, in 9b9c7cf and 7066b23. nothing is knowingly left broken now.
one thing still open, reported not fixed: the hintText selector key resolves off the derived attribute on goja and off [placeholder] in the web runtime, so {hintText: "Email"} matches on one host and not the other. selector_parity_test.go covers no hintText case, which is why nothing catches it.
second review round on this branch, most of it fallout from the fixes above rather than new work.
analyze was censoring clean runs at the budget instead of at the steps they actually ran, so the flattened rank tests read a censored run as an event. two arms where nothing was ever detected could report a12 1.0 and p near zero while the log-rank four lines above in the same report said p 1.0. unpaired is gehan-breslow with the conditional variance now, paired is an exact sign test, both checked against published worked examples and a permutation p.
a folio login was writing the account email and password in cleartext into llm-calls.jsonl, 166 times in one run. redacted at every render site now, and an element reports whether it masks what is typed into it.
the g4 driver gate had gone blind on android, because redaction removed the value it was reading, and separately it was dying on any input typed at coordinates, which scored 129 of 485 recorded traces as fail. both fixed and verified on a real emulator and a real simulator with a negative control, since the gate passing green was the thing that was wrong.
21 selector keys either matched nothing on one host or matched different elements on each, and none of them errored, because the key was accepted. a property built on one just passed while testing nothing. all closed except the substring vs exact rule, which is #92.
just test could not reach a remote adb server at all, and once that was fixed it would install and fuzz whatever single device happened to be attached, which here is a physical phone. it refuses to guess now.