Skip to content

Detect occluded widgets in hierarchy dumps #11

Description

@priyanshujain

uiautomator's XML dump is a flat pre-order list of every node in the view tree, including nodes painted over by something on top (modal dialogs, drawer overlays, bottom sheets, snackbars, system permission prompts, Compose Surfaces with elevation).

internal/hierarchy/hierarchy.go returns the first selector match without any z-order or coverage check. A spec like:

const onLoginScreen = extract((s) => !!s.ax.find("id:login-form"));

will match a covered login form sitting under a confirmation dialog, fire its gated action generator, and tap into the dialog. The property's residual then reflects an action that never logically happened.

Proposed fix (when we see it bite a real spec):

  • hierarchy.Element gains Occluded bool
  • hierarchy.Parse runs one O(n^2) pass to set the flag based on later-in-tree elements covering the center point
  • Find/FindAll skip occluded elements by default; opt-in flag for the rare case where you want them

Deferred because we have no observed failure yet. Filing so we don't re-derive this from scratch when someone reports "the tap went to the wrong place."

Reference: Kea2 (FSE Companion '26) calls this "widget occlusion detection" and lists it as one of two engineering wins they shipped to cut false-positive precondition matches at industrial scale (Tencent/WeChat, AnkiDroid, Amaze).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions