Embed OU tree data in flow responses for unauthenticated OU selection

[DonOmalVindula]

Problem Summary

The OU tree picker in onboarding flows currently fetches tree data by calling OU REST APIs (GET /organization-units, GET /organization-units/{id}/children) directly from the frontend. This works in the admin console because the caller is authenticated, but it creates a hard dependency on API access. If we want to support OU selection in unauthenticated contexts - such as self-registration flows where a user picks their OU - the current approach breaks because the caller has no bearer token to call the OU APIs.

More broadly, the OU tree picker is the only flow input that requires out-of-band API calls to populate its data. Every other input type (SELECT, TEXT_INPUT, PASSWORD_INPUT, etc.) is self-contained in the flow response. This inconsistency limits where OU selection can be used and ties the frontend to a specific data-fetching strategy.

---

Goal

Make OU tree data available through the flow response itself, so the frontend can render the OU tree picker without calling OU APIs separately. The flow engine becomes the single data source for all inputs, including hierarchical tree selection.

---

Constraints

• Large OU trees: Some deployments may have thousands of OUs across deep hierarchies. Sending the full tree in a single response is not viable.
• No server-side state accumulation: The executor should not accumulate the expanded tree in RuntimeData across interactions - this would grow unboundedly as the user expands nodes.
• Existing flow engine model: The solution must work within the existing ExecUserInputRequired loop - the executor returns data, the client responds, the executor runs again. No new flow engine concepts.
• Backward compatibility: The existing OU_SELECT input type and API-based tree picker in the admin console must continue to work unchanged.

---

High-Level Approach

Introduce a stateless, lazy-loaded tree pattern where:

1. The executor returns one page of tree nodes per interaction (not the full tree).
2. The frontend assembles the tree client-side by merging successive responses.
3. Expanding a node or loading more children are flow interactions - the same ExecUserInputRequired → submit → re-execute loop that every other multi-step executor uses.
4. The executor holds no tree state - each interaction is an independent GetOrganizationUnitChildren call.

This mirrors how the existing OrganizationUnitTreePicker already works (it fetches one level at a time via API calls and assembles the tree in React state), but replaces the API calls with flow interactions.

---

Architecture Overview

Current: API-driven tree picker

The tree picker and the flow are separate systems. The picker uses authenticated API calls; the flow only receives the final selection.

Proposed: flow-embedded tree

The tree picker gets data through the flow. Every expand/paginate is a flow interaction - the executor fetches one page from the OU service and returns it. No separate API calls needed. Works authenticated or unauthenticated.

Interaction lifecycle within the flow

The flow stays on the same node throughout all expand/paginate interactions. Only the final ouId submission causes the flow to advance. This is the same pattern as SMSOTPAuthExecutor (send → verify) or BasicAuthExecutor (prompt → validate → retry).

Why stateless?

If the executor accumulated the expanded tree server-side (in RuntimeData), the state would grow with every expand - a user browsing a large tree could cause unbounded growth. Instead:

• The executor is a thin proxy: receive parent ID + offset → query one page → return nodes.
• The frontend is the tree assembler: maintains the expanded tree in React state, merges each response into the appropriate parent.
• Each interaction is independent - the executor doesn't need to know what was previously expanded.

---

Flow Response Format

Extended OU_SELECT with tree data

The existing OU_SELECT input type is extended. When the executor includes treeData in the response, the frontend renders the flow-based tree picker. When treeData is absent, the frontend falls back to the existing API-based picker.

Tree data in the response

A new treeData field is added to the FlowData response model (alongside existing inputs, actions, meta, and additionalData). It carries one page of tree nodes:

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "ouId", "type": "OU_SELECT", "required": true }
        ],
        "treeData": {
            "parentId": "",
            "nodes": [
                { "value": "root-id", "label": "Root", "hasChildren": true },
                { "value": "default-id", "label": "Default", "hasChildren": false }
            ],
            "totalResults": 2,
            "offset": 0,
            "limit": 20
        },
        "actions": [
            { "ref": "action_submit", "nextNode": "ou_selection" }
        ]
    }
}

Field	Description
parentId	Whose children these nodes are. Empty string = root level.
nodes	One page of child nodes.
nodes[].value	OU ID - used as the selection value.
nodes[].label	Display name.
nodes[].hasChildren	Whether this node can be expanded.
totalResults	Total children under parentId (for pagination).
offset / limit	Current page position.

Client-to-server interactions

User action	Flow input	Result
Page loads (initial)	No special inputs	Root-level nodes returned
Expand a node	{ "expandOuId": "node-id" }	Children of that node (page 0)
Load more under a node	{ "expandOuId": "node-id", "offset": "20" }	Next page of children
Select an OU	{ "ouId": "selected-id" }	ExecComplete, flow advances

All interactions route back to the same executor node (via the prompt's nextNode). The executor distinguishes them by which input keys are present.

---

hasChildren - Avoiding N+1 Queries

To populate hasChildren for each node, the executor needs to know whether each child OU has its own children. Naive approach: call GetOrganizationUnitChildren(childId, 1, 0) for each child - this is N+1 queries per page.

Better approaches:

Option A: Batch check - New service method HasChildrenBatch(ctx, ids []string) (map[string]bool, error) that runs a single query:

SELECT ou.OU_ID, ou.HANDLE, ou.NAME, ou.DESCRIPTION,
       EXISTS(SELECT 1 FROM ORGANIZATION_UNIT c WHERE c.PARENT_ID = ou.OU_ID AND c.DEPLOYMENT_ID = $2) AS has_children
FROM ORGANIZATION_UNIT ou
WHERE ou.PARENT_ID = $1 AND ou.DEPLOYMENT_ID = $2
ORDER BY ou.NAME LIMIT $3 OFFSET $4

Option B: Include child count in list query - Modify GetOrganizationUnitChildren to return a childCount field via a subquery or window function, so hasChildren = childCount > 0 with no extra round trip.

Option B is preferable as it requires no additional queries and the information is derived in the same list query.

---

Scope and Backward Compatibility

What	Impact
OU_SELECT input type	Extended - when treeData is present, uses flow-based picker; otherwise falls back to API-based picker
Existing prompt strategy	Extended to support flow-embedded tree (scoped to user type's default OU subtree)
Existing promptAll strategy	Extended to support flow-embedded tree (full tree from root)
Existing caller strategy	Unchanged
Frontend admin console	Unchanged - OrganizationUnitTreePicker keeps using API calls
Frontend flow renderer	New component that reads treeData from flow response

The executor strategy (prompt or promptAll) determines whether tree data is included. Existing flows that rely on the API-based picker continue to work - the frontend only switches to the flow-based picker when treeData is present in the response.

---

Performance Considerations

• Flow interaction overhead: Each expand/paginate goes through the full flow engine cycle (load flow context from DB → execute node → save context back). This is heavier than a direct REST API call to the OU endpoint. However, the trade-off is acceptable because:
  • Tree browsing is a low-frequency, user-driven interaction (not automated/bulk).
  • The actual DB query per interaction is a single paginated SELECT - the same query the REST API would run.
  • The overhead is the flow context serialization/deserialization, which is small relative to the user's perceived latency (click → render).
• Single query per interaction: With the hasChildren subquery (Option B), each expand/paginate is exactly one SQL query. No N+1 problem.
• No accumulated state: The executor is stateless - flow context size does not grow with tree exploration. Only ouId is written to RuntimeData on final selection.

---

Security Considerations

• No new APIs or permissions. Tree data flows through the existing flow execution endpoint, which already handles authorization for the flow type.
• Unauthenticated flows: The flow engine controls what data is returned. The executor only exposes OU names and IDs - the same information visible in the OU tree picker today. No sensitive data is leaked.
• OU visibility in self-registration: When used in registration flows, the executor would expose the OU tree to unauthenticated users. Deployments should only enable OU_SELECT with tree data in registration flows if they intentionally want users to pick their OU. The caller strategy remains available for restricted scenarios.
• Validation unchanged. The final ouId selection is still validated by the executor (IsOrganizationUnitExists or IsParent), and user/service.go:validateOrganizationUnitForUserType() enforces OU-schema compatibility at user creation.

---

Alternatives Considered

Alternative 1: Send the full OU tree in a single response

• Pros: Simple - one response, no multi-step interaction
• Cons: Doesn't scale - a tree with thousands of OUs produces a massive payload. Frontend must receive and parse everything upfront.
• Decision: Rejected for scalability. Lazy loading is the standard pattern for large trees.

Alternative 2: Accumulate expanded tree state in executor RuntimeData

• Pros: Each response returns the full expanded-so-far tree - frontend is stateless
• Cons: RuntimeData grows unboundedly as the user expands nodes. Serializing/deserializing a large tree on every interaction is expensive. Flow session storage grows.
• Decision: Rejected. Keep the executor stateless; the frontend already has tree assembly logic.

Alternative 3: Proxy OU APIs through a special flow endpoint (e.g., /flows/{flowId}/resources/ous)

• Pros: Clean separation - tree picker uses a dedicated endpoint with flow-scoped auth
• Cons: New API surface to design, secure, and maintain. Breaks the "flow response is self-contained" principle. Requires flow-scoped token management.
• Decision: Rejected. Using the existing flow interaction model is simpler and requires no new endpoints.

Alternative 4: Embed tree data in AdditionalData as a JSON string

• Pros: No model changes - AdditionalData is map[string]string
• Cons: Stringly-typed, no schema validation, awkward to parse, pagination metadata doesn't fit naturally.
• Decision: Rejected. A typed treeData field is cleaner and aligns with how inputs, actions, and meta are structured.

---

Design Decisions

1. Should OU_TREE_SELECT be a new input type, or should OU_SELECT gain tree data capabilities?

Extend the existing OU_SELECT input type. When treeData is present in the flow response, the frontend uses the flow-based tree picker; when absent, it falls back to the existing API-based picker. This avoids having two input types for the same concept and keeps backward compatibility - existing flows without treeData work exactly as before.

2. Should the prompt strategy also support flow-embedded tree, or only promptAll?

Both prompt and promptAll should support flow-embedded tree. The prompt strategy scopes to a subtree (rooted at the user type's default OU), and promptAll shows the full tree from root. Both return tree data in the flow response - the only difference is the root node the executor starts from. This ensures OU selection works in unauthenticated contexts regardless of the strategy used.

3. What should the default page size be for tree node fetches?

Page size is configurable as an executor property (pageSize), with a sensible default fallback (e.g., 20). This keeps simple flows simple while allowing tuning for deployments with large OU trees.

4. Should expand interactions use a new action type, or reuse the existing action ref?

Expand and select use the same action ref, distinguished by input keys. The tree picker submits all interactions through a single action. The executor checks which input key is present: expandOuId (with optional offset) triggers tree browsing, while ouId triggers final selection. This keeps the PROMPT node simple (one action) and the frontend doesn't need to map UI events to different action refs - it just submits different input payloads through the same submitFlowInput() call.

5. How should hasChildren be determined - batch check (Option A) or subquery (Option B)?

Option B (subquery) is preferred. Including a correlated EXISTS subquery in the list query derives hasChildren with no extra round trip. The added query complexity is minimal compared to the cost of a separate batch call (Option A), and it keeps the data-fetching path as a single query per interaction.

---

Tracking issue: #2161

[DonOmalVindula]

Related Discussion

This builds on the architecture established in #1816 (Interactive OU Selection via OUResolver Prompt Strategy). The core interaction model is the same - stateless executor, ExecUserInputRequired loop, ouId/expandOuId input keys, client-side tree assembly.

This discussion extends that design with:

1. A typed treeData response field (instead of flat SELECT options) for better tree rendering
2. Pagination support (offset/limit) for large numbers of children at one level
3. hasChildren flag on each node to avoid unnecessary expand round-trips
4. promptAll strategy support (full tree from root, not scoped to defaultOUID subtree)

[senthalan]

Can you give the request response format of the API call between the client and server when expanding the node and load
more under a node ?

[ThaminduDilshan]

Rather than introducing a new treeData object to the top level in data, what if we include this inside additionalData?

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "ouId", "type": "OU_SELECT", "required": true }
        ],
        "actions": [
            { "ref": "action_submit", "nextNode": "ou_selection" }
        ]
        "additionalData": {
            "treeData": {
                "parentId": "",
                "nodes": [
                    { "value": "root-id", "label": "Root", "hasChildren": true },
                    { "value": "default-id", "label": "Default", "hasChildren": false }
                ],
                "totalResults": 2,
                "offset": 0,
                "limit": 20
            },
        },
    }
}

[DonOmalVindula]

@senthalan Here are the request/response formats for each interaction type. Following @ThaminduDilshan's suggestion, treeData is placed inside additionalData instead of as a new top-level field.

1. Initial page load (root nodes returned)

Request:

POST /flows/{flowId}

{
    "actionRef": "action_submit",
    "inputs": {}
}

Response:

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "ouId", "type": "OU_SELECT", "required": true }
        ],
        "actions": [
            { "ref": "action_submit", "nextNode": "ou_selection" }
        ],
        "additionalData": {
            "treeData": "{\"parentId\":\"\",\"nodes\":[{\"value\":\"root-id\",\"label\":\"Root\",\"hasChildren\":true},{\"value\":\"default-id\",\"label\":\"Default\",\"hasChildren\":false}],\"totalResults\":2,\"offset\":0,\"limit\":20}"
        }
    }
}

Decoded treeData for readability:

{
    "parentId": "",
    "nodes": [
        { "value": "root-id", "label": "Root", "hasChildren": true },
        { "value": "default-id", "label": "Default", "hasChildren": false }
    ],
    "totalResults": 2,
    "offset": 0,
    "limit": 20
}

2. Expand a node

Request:

{
    "actionRef": "action_submit",
    "inputs": {
        "expandOuId": "root-id"
    }
}

Response:

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "ouId", "type": "OU_SELECT", "required": true }
        ],
        "actions": [
            { "ref": "action_submit", "nextNode": "ou_selection" }
        ],
        "additionalData": {
            "treeData": "{\"parentId\":\"root-id\",\"nodes\":[{\"value\":\"engineering-id\",\"label\":\"Engineering\",\"hasChildren\":true},{\"value\":\"marketing-id\",\"label\":\"Marketing\",\"hasChildren\":false},{\"value\":\"sales-id\",\"label\":\"Sales\",\"hasChildren\":true}],\"totalResults\":3,\"offset\":0,\"limit\":20}"
        }
    }
}

Decoded treeData:

{
    "parentId": "root-id",
    "nodes": [
        { "value": "engineering-id", "label": "Engineering", "hasChildren": true },
        { "value": "marketing-id", "label": "Marketing", "hasChildren": false },
        { "value": "sales-id", "label": "Sales", "hasChildren": true }
    ],
    "totalResults": 3,
    "offset": 0,
    "limit": 20
}

3. Load more children under a node (paginate)

Request:

{
    "actionRef": "action_submit",
    "inputs": {
        "expandOuId": "engineering-id",
        "offset": "20"
    }
}

Response:

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "ouId", "type": "OU_SELECT", "required": true }
        ],
        "actions": [
            { "ref": "action_submit", "nextNode": "ou_selection" }
        ],
        "additionalData": {
            "treeData": "{\"parentId\":\"engineering-id\",\"nodes\":[{\"value\":\"platform-id\",\"label\":\"Platform\",\"hasChildren\":false},{\"value\":\"devops-id\",\"label\":\"DevOps\",\"hasChildren\":false}],\"totalResults\":22,\"offset\":20,\"limit\":20}"
        }
    }
}

Decoded treeData:

{
    "parentId": "engineering-id",
    "nodes": [
        { "value": "platform-id", "label": "Platform", "hasChildren": false },
        { "value": "devops-id", "label": "DevOps", "hasChildren": false }
    ],
    "totalResults": 22,
    "offset": 20,
    "limit": 20
}

4. Final selection (flow advances)

Request:

{
    "actionRef": "action_submit",
    "inputs": {
        "ouId": "engineering-id"
    }
}

Response (flow moves to the next step):

{
    "flowId": "abc-123",
    "flowStatus": "INCOMPLETE",
    "type": "VIEW",
    "data": {
        "inputs": [
            { "identifier": "username", "type": "TEXT_INPUT", "required": true }
        ],
        "actions": [
            { "ref": "action_submit", "nextNode": "next_step" }
        ]
    }
}

Note on additionalData typing

Since additionalData is map[string]string on the backend, treeData would be stored as a JSON-encoded string. This follows the same pattern already used by consentPrompt in the consent executor. The frontend would need to JSON.parse() the value when reading it.

[senthalan]

Taking a step back and challenging this requirement. Will we have the usecase to render the OU tree within the unauthentication flow? There will be definitely usecase to have an OU selector from a list of OUs, but I don't think of a usecase where the user selects the OU from an OU tree in an unauthenticated flow (self registration)

@darshanasbg @ThaminduDilshan WDYT?