Feature: Window/Iframe support

[wurzelkuchen]

Quick Description

Current AI-driven automation might have issues with deep DOM hierarchies. We need to verify and potentially enhance Neodymium's AI integration to ensure it can traverse and interact with elements nested within multiple layers of iframes (stacked iframes). And also to handle new windows.

User Story

As a test automation engineer,
I want the AI integration to successfully identify and interact with elements without worrying about of nested iframes or different tabs/windows in the browser
so that I can automate complex enterprise applications without manual workarounds.

General Approach

• Create a local test with multiple nested iframes (3+ levels deep).
• Create a local test where a new window/tab is opened from the webiste (like for paypal payment) and needs to be used.
• Test the current AI prompt resolution against these nested elements.

Phase 1 Requirements

• Research how the current AI context is built: Does it include the full frame tree? Can it work with new Windows/Tabs
• Make sure that (nested) Iframes are icluded in the simplified DOM we send to the AI and can be used by the AI.
• Add a way to switch tabs/windows
• Verify that coordinate mapping (if used) accounts for frame offsets.

[m13v]

this is the same pattern as cross-process accessibility trees on desktop. framework presents a unified element graph but each child owns its own coord space and lifecycle. the fix isn't better DOM serialization, it's making frame context a first-class field on every action. AI passes (frame_id, ref), not just (ref), and 'switch frame' is an explicit op the planner must call. otherwise the LLM emits a click targeting the right element id at the wrong offset and the trace doesn't tell you why it missed. windows need the same treatment or the planner loses track of which one it just opened.

[wurzelkuchen]

Thanks, that’s a valuable contribution. I agree we should move forward with this approach.

Regarding the implementation: I’m still thinking about a way to have the 'switch' operation handled implicitly by the framework whenever a new frame_id is detected in an action. I haven't fully mapped out the edge cases for that yet (especially with nested frames and cross-origin boundaries), but I'd prefer to keep that logic out of the planner if possible.