Skip to content

fix(android): preserve editable-field metadata in snapshots - #2290

Merged
thymikee merged 2 commits into
callstack:mainfrom
AdzeB:fix/android-field-observation-metadata
Sep 6, 2026
Merged

fix(android): preserve editable-field metadata in snapshots#2290
thymikee merged 2 commits into
callstack:mainfrom
AdzeB:fix/android-field-observation-metadata

Conversation

@AdzeB

@AdzeB AdzeB commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Android structured snapshots and get attrs now preserve native editable, password, hintShowing, and selection offsets, including explicit false/zero, on every response level including the token-cheap digest. Empty accessibility text remains value: ""; unavailable facts remain absent. Documentation explains availability and interpretation.

This extends #2066's internal hint handling. Selection offsets are accessibility offsets, independent of editable (read-only selectable text exposes them too); they are not character counts or proof of value equality. Exact opaque-value verification remains separate (#2289); no new secret-value reads or fill-verification behavior are introduced.

Closes #2288.

Changes

  • Helper: emits text when present (even empty), editable, password, hint-showing (API 26+) always, and each nonnegative selection-start / selection-end independently; -1 stays absent.
  • Parser → raw/regular snapshot nodes carry the five facts; explicit false/zero survive, missing stays undefined.
  • get attrs --level digest keeps the five facts (SELECTOR_DIGEST_NODE_FIELDS).
  • Docs: new "Android field metadata" section in snapshots.md.

Validation

Head: e1b7be604154b701cc7f8f5e2e6e678aad1527da (rebased onto main 4b7c561d1e).

  • pnpm check:affected --run: format, lint, typecheck, layering, fallow, build, 741 test files / 5,713 tests passed.
  • Regressions: parser-to-snapshot (false/zero/empty, non-editable selectable node, unavailable → absent) and the attrs digest (explicit false/zero/empty kept, unavailable omitted). The digest test is red without the field-list change; the parser tests are red without the parser change.
  • pnpm package:android-snapshot-helper:npm built the 0.20.11-dev helper (helper unit suite runs inside the build).
  • Live on a private Pixel 9 Pro XL API 37 AVD with the rebuilt helper and an isolated state dir, get attrs --level digest: the name field after fill "Grace" reports editable: true, password: false, hintShowing: false, selectionStart: 5, selectionEnd: 5; the form-title ViewGroup reports editable: false, password: false, hintShowing: false and no selection keys. The text snapshot is unchanged.

The earlier test-isolation commit was dropped: main already mocks notifyIosRunnerAppRelaunched and the lifecycle test passes without it; the intermittent pkill leak is tracked in #2314.

@thymikee

thymikee commented Sep 5, 2026

Copy link
Copy Markdown
Member

Two blocking production-route gaps remain at exact head a6fa69a8c0cc3dcd83568db20f57fd13936c7eba:

  1. get attrs --level digest drops editable, password, hintShowing, selectionStart, and selectionEnd because SELECTOR_DIGEST_NODE_FIELDS in src/daemon/response-views.ts omits them. Digest is the supported token-cheap attrs response, so add these semantic fields and a digest-route regression covering explicit false/zero/empty plus omission when unavailable.

  2. The helper emits selection offsets only inside if (node.isEditable()), but feat(android): expose editable-field metadata in structured snapshots #2288 requires every available nonnegative accessibility offset. Read-only selectable text can expose a valid selection independently of editability. Read and emit each nonnegative offset independently, and add a parser-to-snapshot regression for a non-editable selectable node; negative/unavailable values must remain absent.

The rest of the XML → parser → raw/regular snapshot route looks correct, and the live Android evidence is credible for the unchanged production commit. Final-head CI has not reported yet. Holding ready-for-human until these gaps are fixed.

AdzeB and others added 2 commits September 5, 2026 23:16
…n offsets past editability

- `get attrs --level digest` kept only the pre-callstack#2288 semantic fields, so
  `editable`/`password`/`hintShowing`/`selectionStart`/`selectionEnd`
  vanished on the token-cheap route. The digest now keeps them, with a
  regression covering explicit false/zero/empty and omission when unavailable.
- The helper emitted selection offsets only inside `isEditable()`, but
  read-only selectable text exposes a selection too. Each nonnegative offset
  is now emitted independently; -1 stays absent. Parser-to-snapshot regression
  for a non-editable selectable node.
- Docs: the field-metadata notes get their own section instead of leading the
  efficiency tips.
- Dropped the test-isolation commit: main already mocks
  notifyIosRunnerAppRelaunched, and the lifecycle test passes without it.
@thymikee
thymikee force-pushed the fix/android-field-observation-metadata branch from a6fa69a to e1b7be6 Compare September 5, 2026 21:29
@thymikee

thymikee commented Sep 5, 2026

Copy link
Copy Markdown
Member

Both gaps are fixed at head e1b7be604154b701cc7f8f5e2e6e678aad1527da (branch rebased onto main 4b7c561d1e, contributor's commit kept as-is, one fix-up commit on top):

  1. Digest route. SELECTOR_DIGEST_NODE_FIELDS in src/daemon/response-views.ts now includes editable, password, hintShowing, selectionStart, selectionEnd. New test in response-views.test.ts covers explicit false/zero/empty kept and omission when the node carries none of them; it fails without the field-list change.
  2. Selection offsets. AccessibilityTreeXml.java reads getTextSelectionStart() / getTextSelectionEnd() outside isEditable() and emits each nonnegative offset on its own (appendNonNegativeAttribute); -1 stays absent. New parser-to-snapshot test for a non-editable node with selection-start="2" selection-end="5", alongside the existing unavailable → absent case.

Also: the test(session): isolate runtime prewarm side effect commit is dropped. On current main the lifecycle test passes without it (main already mocks notifyIosRunnerAppRelaunched), and the intermittent pkill leak it was chasing is #2314. The docs paragraph moved out of "Efficient snapshot usage" into its own "Android field metadata" section.

Validation. pnpm check:affected --run green: format, lint, typecheck, layering, fallow, build, 741 test files / 5,713 tests. Helper rebuilt with pnpm package:android-snapshot-helper:npm (its unit suite runs in the build). Live on a private Pixel 9 Pro XL API 37 AVD, rebuilt 0.20.11-dev helper, isolated state dir, get attrs --level digest --json:

  • id=field-name after fill "Grace": editable: true, password: false, hintShowing: false, selectionStart: 5, selectionEnd: 5
  • id=form-title (ViewGroup): editable: false, password: false, hintShowing: false, no selection keys

Text snapshot output is unchanged. --json / --raw grow by the three explicit booleans per node, which is the #2288 contract (explicit false is the signal).

Re-validation gotcha: a -dev helper build gets versionCode: 1, so an AVD that already carries a release helper (versionCode 200xx) keeps the stale one as "current" and the new fields never appear. adb uninstall com.callstack.agentdevice.snapshothelper plus a daemon restart before reading live output.

@thymikee

thymikee commented Sep 5, 2026

Copy link
Copy Markdown
Member

Sentinel: clean coordinator + independent Terra/high review at e1b7be6. Both previous blockers are resolved: non-editable selection offsets survive helper capture, and get/find digest output preserves the field facts, including false/zero/empty values. Focused regressions target both fixes. Exact-head runnable CI is green, including Android smoke and coverage/integration; the PR records private-Pixel live digest evidence (not independently rerun in this review). Ready for human review/merge; no merge performed.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 5, 2026
@thymikee
thymikee merged commit f30328d into callstack:main Sep 6, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(android): expose editable-field metadata in structured snapshots

2 participants