Improve element-ref robustness on dynamic pages (beyond the stale-ref recovery in #625)

[lmorchard]

Context

Element refs (E1, E2, …) are positional counters minted fresh on every snapshot; they are not stable across snapshots and can go stale between the model reading a snapshot and the action firing. On a churny SPA this surfaces as Invalid element reference errors.

Surfaced by a Zoo eval (Magento guest-checkout): stale-ref rate was ~3% of actions (~1 per checkout) — occasional, but before #625 a single stale ref was deterministically fatal: with no snapshot refresh on the error, the model kept acting on a frozen snapshot, confabulated refs, and hit the consecutive-error budget. Evidence + forensic write-up live in the eval repo at docs/dev-sessions/2026-07-27-vwa-checkout-stale-ref/.

What #625 already does

#625 adds loop-level recovery: on a stale/invalid-ref error, force a snapshot refresh so the model gets live refs and retries. Validated 11/11 on the checkout task (each recovering from its stale ref). This stops the cascade but (a) costs a model round-trip per event and (b) doesn't reduce the underlying rate.

Possible general improvements (leverage order)

1. Browser-layer self-healing (highest leverage) — on an invalid ref, re-resolve the element by its stored identity before erroring to the model. Pilo already captures {role, name} per ref (__piloIdentityMap) and sets data-pilo-ref on the DOM. If the same element still exists, act transparently — most stale refs become silent successes, no LLM round-trip.
2. Nearest-valid-ref hint on error (cheap) — name valid refs in the error so the model can't confabulate; strengthens fix(webAgent): keep the aria snapshot fresh on client-rendered pages (SPA readiness + stale-ref recovery) #625's recovery.
3. Settle/hydration gate before snapshot — shrink the stale window (bounded networkidle / "tree stopped growing").
4. Stable refs across snapshots (deepest) — reuse IDs for conceptually-same elements so a ref survives a re-render.

Relationship to #446

#446 (search_page/find_elements) gives a stable-descriptor → current-ref addressing path (CSS/text → live nearestRef), which is more robust than counter refs and a cleaner recovery route than confabulation — but it's model-driven and its nearestRef still depends on data-pilo-ref freshness (re-minted only on snapshot), so it composes with #625 rather than replacing it. Its plumbing (closest('[data-pilo-ref]'), live querySelector) is exactly what option 1 would build on.

Recommendation

At ~3%, #625's recovery is right-sized. Measure stale-ref rates across sites first (the partial cloud eval on the #625 build will help) before investing in option 1 — build the self-heal only if rates elsewhere justify it.

🤖 Generated with Claude Code

[lmorchard]

Concrete cloud data point: #625's refresh fires but can't out-run a churning page

From partial eval run pilo-batch-github-eval-667rj (build 257ea70, which includes #625's stale-ref refresh), task webvoyagerx--Booking--0 failed stale_element_refs: "Task failed after 5 consecutive errors (8 total): Invalid element reference 'E184'."

Event-log analysis confirms #625 is working here — a fresh page-snapshot is appended on every iteration through the failure (snapshot messages grow 18→19→20→21→22→23), the opposite of the pre-#625 deadlock where the conversation froze. The problem is that the page is the Mexico hotel results page — ~1,984 refs with live prices/availability churning — so refs go stale within the action latency, faster than the model can act on the fresh snapshot it's handed. E184 is even present in the latest snapshot, yet resolves as invalid by click time. Refresh-per-error can't keep up; it burns the 5-consecutive-error budget and dies.

This is exactly the residual case this issue's option 1 (browser-layer self-heal) is for: re-resolving a stale ref by stored identity (__piloIdentityMap) below the model loop would turn these per-action churns into transparent retries with no model round-trip, where a snapshot refresh alone can't. (Also relevant: option 4, stable refs across snapshots.)

Contrast: on a normal satisfiable page (Zoo wireless-mouse checkout, 11/11 in #625's confirmation run) the refresh fully recovers. So the boundary is specifically heavy, high-churn pages — worth keeping this Booking-0 trace as the motivating example.