harden Dataverse derived metadata reconciliation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3e31c561-a943-4464-92f3-04ab071393c0

Author: Shubham Agarwal

plugins/mobile-apps/agents/data-model-architect.md

@@ -207,56 +207,45 @@ and document their role instead.
 ## Step 6a — Cross-entity Read Audit
 
 **Print before starting:**
-> "→ Auditing planned screens for cross-entity reads (calc-column candidates)…"
+> "→ Auditing planned screens for supported cross-entity read paths…"
 
 **Run condition:** execute this step when EITHER (a) `<working_dir>/_screens_section.md` exists at this point in the workflow OR (b) you were invoked with `mode: cross-entity-audit`. **Skip silently otherwise** (default-mode first-pass run, before screen-planner has produced its section) — the orchestrator will re-spawn you in `mode: cross-entity-audit` after Gate 4a/4b lands.
 
 When `mode: cross-entity-audit`, the orchestrator's prompt also includes the path to the existing `_dm_section.md` so you can append (do NOT regenerate it from scratch — Steps 1–6 are skipped in this mode).
 
-This step exists because of the runtime constraint documented at [`shared/references/data-performance.md` § Cross-entity Reads](${PLUGIN_ROOT}/shared/references/data-performance.md#cross-entity-reads) — the SDK has no `$expand`, so cross-entity fields on hot paths (lists, dashboards) MUST be denormalized via calculated columns at the data-model layer. This step proposes those calc columns based on the screen plan; `/setup-datamodel` (or `/add-dataverse`) Phase 6.1b creates them.
+This step exists because the generated SDK has no `$expand`. It classifies each
+cross-entity field into a supported formatted lookup or bounded chained fetch.
+Dataverse does not support defining calculated/formula expressions through
+code, so this audit never proposes generated formula metadata.
 
 **Algorithm:**
 
 1. **Read the screen plan.** Look for `<working_dir>/_screens_section.md` first (graph-only mode after Gate 4a). If absent, parse `<working_dir>/native-app-plan.md` and extract the `## Screens` section. Walk every per-screen spec and collect every `related_entity_fields` block.
 
 2. **Per entry, branch on `recommends`:**
 
-   - **`recommends: calc-column`** — first classify the projection instead of assuming every dotted path is a supported formula:
-     - **Supported calculated projection** — a scalar value reachable only through N:1 lookups and supported by the target Dataverse formula capability. Emit a calc column.
-     - **Lookup annotation** — the screen only needs the related record's display name already exposed by the lookup. Reuse the lookup annotation; create no column.
-     - **Materialized projection** — the value must be snapshotted for history/offline use or the formula capability is unsupported. Emit an ordinary column and state which create/update flow owns synchronization.
-     - **Chained fetch** — 1:many, M:N, aggregate, conditional, or otherwise unsupported navigation. Create no column; the screen-builder fetches it.
+   - **`recommends: formatted-lookup`** — verify the source is the primary
+     display name of a direct lookup. Create no column.
+   - **`recommends: chained-fetch`** — create no column. The screen-builder
+     performs one bounded related request outside row rendering.
+   - **`recommends: external-projection-required`** — record a blocker for a hot
+     list/dashboard field that cannot use the direct lookup annotation. Omit
+     the field until the user supplies a maker-created formula column or other
+     server-owned projection.
 
-     Confirm the `cardinality` is `1:1` (calc columns CANNOT traverse 1:many or M:N). If cardinality or formula capability is unsupported, choose `materialized projection` only when snapshot semantics are required; otherwise downgrade to `chained-fetch` and add a `DONE_WITH_CONCERNS` note. For a supported calculation, propose a calculated column on the **primary entity of that screen** (the entity its primary `Data` service queries):
-     - **Logical name:** `<prefix>_<resolved_field>_calc` (lowercased, e.g. `cr3e9_gatename_calc`)
-     - **Schema name:** PascalCase variant (e.g. `Cr3e9_GateName_calc`)
-     - **Display name:** human label from the planner's `field` value (e.g. "Gate name")
-     - **Type:** matches the resolved field's TypeScript type → Dataverse type (`string` → `Edm.String`, `datetime` → `Edm.DateTimeOffset`, `decimal` / `money` → `Edm.Decimal`, `integer` → `Edm.Int32`, `boolean` → `Edm.Boolean`)
-     - **Formula source:** the dotted path from the planner's `source` field, normalized — e.g. `cr3e9_flightid → cr3e9_gateid → cr3e9_gatename` becomes `cr3e9_flightid.cr3e9_gateid.cr3e9_gatename`. The formula is N:1 lookup chain only; no aggregations, no conditionals, no string concat in v0.
-
-   - **`recommends: chained-fetch`** — do NOT add any column. The screen-builder handles this at scaffold time per the decision table in `data-performance.md`. Just include the entry in the addendum's `Chained-fetch fields (informational)` row so the user sees what the screen-builder will scaffold.
-
-3. **De-duplicate.** A field driven by N screens (e.g. "Gate name" used on home, list, AND detail) collapses to ONE calc-column row in the addendum. Track all driving screens in the `Driven by` column.
-
-4. **Cap at 20 calc columns per parent entity.** If you exceed, truncate and add a `DONE_WITH_CONCERNS` note — large calc-column counts indicate a denormalization problem that should be solved at the data-model level (probably an extracted entity), not by piling on calc columns.
+3. **De-duplicate.** Collapse identical source/resolution pairs and track all
+   consuming screens.
 
 5. **Emit the addendum.** Write the `### Cross-entity Reads (auto-derived from screen plan)` subsection of `_dm_section.md`. Schema:
 
    ```markdown
    ### Cross-entity Reads (auto-derived from screen plan)
 
-   | Calc column | On table | Type | Resolves | Driven by |
-   |---|---|---|---|---|
-   | cr3e9_flightnumber_calc | cr3e9_inspection | string | cr3e9_flightid.cr3e9_flightnumber | inspections list |
-   | cr3e9_gatename_calc | cr3e9_inspection | string | cr3e9_flightid.cr3e9_gateid.cr3e9_gatename | home, inspections list |
-   | cr3e9_tailnumber_calc | cr3e9_inspection | string | cr3e9_flightid.cr3e9_aircraftid.cr3e9_tailnumber | inspections list |
-
-   **Chained-fetch fields (informational — screen-builder will scaffold these, no schema changes):**
-
-   | Field | On screen | Cardinality | Source |
+   | Field | Resolution | Source | Driven by |
    |---|---|---|---|
-   | Defect count | inspection detail | 1:many | cr3e9_inspectionzoneid → cr3e9_defect |
-   | Inspector email | inspection detail | 1:1 | _ownerid_value → systemuser.internalemailaddress |
+   | Flight | formatted-lookup | cr3e9_flightid primary display | inspections list |
+   | Inspector email | chained-fetch | _ownerid_value → systemuser.internalemailaddress | inspection detail |
+   | Gate code | external-projection-required | cr3e9_flightid → cr3e9_gateid → cr3e9_code | home |
    ```
 
    In `mode: default` (Step 6a runs because `_screens_section.md` was found), append this subsection to the Step 7 output. In `mode: cross-entity-audit`, append it directly to the existing `_dm_section.md` (read it, append the subsection AFTER `### Notes` if present, otherwise at the end, then write back) and skip Step 7 entirely — return immediately.

plugins/mobile-apps/agents/native-app-planner.md

@@ -498,13 +498,16 @@ Reject loop = re-spawn `screen-planner` with the user's feedback (layout, screen
 ### Step 5c — Cross-entity Read Audit (Round 2 data-model pass)
 
 **Print before spawning:**
-> "→ Auditing the locked screen plan for cross-entity reads (calc-column candidates from related_entity_fields blocks)…"
+> "→ Auditing the locked screen plan for supported cross-entity read paths…"
 
-**Run condition:** execute this step ONLY after Gate 4b has been approved AND the screen-planner's per-screen specs include at least one `related_entity_fields` block. Skip silently otherwise (no cross-entity reads = no calc-column proposals needed).
+**Run condition:** execute this step ONLY after Gate 4b has been approved AND the screen-planner's per-screen specs include at least one `related_entity_fields` block. Skip silently otherwise.
 
 **Detection (cheap):** before spawning, `Grep` the locked plan for `related_entity_fields:` in `<working_dir>/native-app-plan.md`. Zero matches → skip Step 5c entirely, mark `[x]` and proceed to Step 6. One or more matches → spawn the audit pass below.
 
-This step exists because of the runtime constraint documented at [`shared/references/data-performance.md` § Cross-entity Reads](${PLUGIN_ROOT}/shared/references/data-performance.md#cross-entity-reads) — the SDK has no `$expand`, so cross-entity fields on hot paths (lists, dashboards) MUST be denormalized via calculated columns at the data-model layer. The screen-planner emits `related_entity_fields` per screen; this step turns those into calc-column proposals.
+This step exists because the SDK has no `$expand`. It verifies that every
+cross-entity field uses a formatted lookup or bounded chained fetch, and flags
+hot-path fields that require an externally supplied projection. It never
+synthesizes calculated/formula metadata.
 
 #### 5c.1 — Spawn `data-model-architect` in `cross-entity-audit` mode
 
@@ -530,22 +533,23 @@ Wait for return; apply the Step 3.0 status switch:
 - `DONE_WITH_CONCERNS: <list>` → embed addendum, propagate concerns into your own final `DONE_WITH_CONCERNS:`.
 - `NEEDS_CONTEXT:` / `BLOCKED:` — propagate up per the standard switch.
 
-#### 5c.2 — Gate 1 addendum (calc-column approval)
+#### 5c.2 — Gate 1 addendum (cross-entity read paths)
 
 If 5c.1 wrote a `### Cross-entity Reads` addendum, present it to the user as a Gate 1 addendum (NOT a fresh Gate 1 — the original schema is already approved and unchanged):
 
 ```
 ## Gate 1 — Addendum: Cross-entity Reads
 
-The screen plan you approved at Gate 4b reads N fields from related entities (gate names on inspections, customer phones on orders, etc.). Because the Power Apps SDK has no $expand, those fields need calculated columns on the parent tables to display efficiently — otherwise list screens would either render "—" or trigger N+1 fetches per row.
+The screen plan reads N fields from related entities. The generated SDK has no
+$expand, so each field must use a formatted lookup, a bounded chained fetch, or
+an external server-owned projection.
 
-Proposed calculated columns (auto-derived from your screen plan, no schema reshape):
+Proposed read paths:
 
 [paste the ### Cross-entity Reads table from _dm_section.md]
 
-[paste the Chained-fetch fields (informational) table if present — these need NO schema change, the screen-builder handles them at scaffold time]
-
-Approve to add these calc columns to the data model? (Reject → revise the audit. Approve → /setup-datamodel will create them in Phase 6.1b.)
+Approve these read paths? Any `external-projection-required` row remains a
+blocker until the user supplies that projection outside this workflow.
 ```
 
 Reject loop = re-spawn data-model-architect in `mode: cross-entity-audit` with the user's feedback (e.g. "drop cr3e9_tailnumber_calc, the list doesn't actually show it"). Approve = mark `[x]` Gate 1 addendum approved, proceed to Step 6.

plugins/mobile-apps/agents/screen-builder.md

@@ -116,26 +116,26 @@ You will be invoked by `/create-mobile-app` Step 11 or `/edit-app` screen-rebuil
   ```
 
   For lookup labels, select the real `_<lookup>_value` field in your `$select` and read with `lookupName(record, '<lookupLogicalName>')`. For choice / status / boolean / datetime / money labels, read with `formattedValue(record, '<columnLogicalName>')` or fall back to the generated option const. On bounded lists that use `useSearchFilter(...)`, fields MUST be real string properties from generated types; never add an inferred display-name field just to make search prettier. Cursor lists do not use `useSearchFilter`; they push search into the service `filter` option.
-- **HARD RULE — Cross-entity Field Resolution.** Before writing the screen's `select: [...]` or load step, walk every UI field your spec displays. For each field that sources data from an entity OTHER than the screen's primary fetch target, follow this algorithm exactly. The full reference (with cost-profile rationale, calc-column naming, and pattern examples) is at [`shared/references/data-performance.md` § Cross-entity Reads](${PLUGIN_ROOT}/shared/references/data-performance.md#cross-entity-reads). The screen-builder MUST apply the rule mechanically — do NOT invent your own resolution.
+- **HARD RULE — Cross-entity Field Resolution.** Before writing the screen's `select: [...]` or load step, walk every UI field your spec displays. For each field that sources data from an entity OTHER than the screen's primary fetch target, follow this algorithm exactly. The full supported-path reference is at [`shared/references/data-performance.md` § Cross-entity Reads](${PLUGIN_ROOT}/shared/references/data-performance.md#cross-entity-reads). The screen-builder MUST apply the rule mechanically — do NOT invent your own resolution.
 
-  1. **Calc column check first.** Open `src/generated/models/<PrimaryEntity>Model.ts` and search for a column matching `<prefix>_<field>_calc` (or any `_calc`-suffixed column resolving the field you need). If present, add it to your `select: [...]` and render directly. Done — no chained fetch needed.
-  2. **No calc column? Branch on screen archetype × cardinality** (archetype is in your spec under `**Archetype:**`):
+  1. **Follow the planned recommendation** from `related_entity_fields`.
 
-     | Archetype | Cardinality | Action |
+     | Recommendation | Action |
      |---|---|---|
-     | List (`top ≥ 5`), Tab-root, Dashboard | 1:1 (N:1 lookup chain) | **STOP — do NOT chain a fetch in the list `map()` / `renderItem`.** Emit a `// TODO(cross-entity-read):` header comment naming the field, the related entity, and the recommended `<prefix>_<field>_calc` column to add. Render the cell as `'—'` for now. Then return `BLOCKED [<screen_name>]: list field <field> requires cross-entity read; needs calc column <prefix>_<field>_calc on <primary_entity> — re-run /setup-datamodel (or /add-dataverse for existing apps) to add the calc column.` Do NOT scaffold an N+1 fetch storm. |
-     | Detail (single record) | 1:1 (N:1 lookup chain) | Scaffold a chained `<RelatedService>.get(record._<lookup>_value, { select: [...] })` in the screen's load step. One record on screen = one extra round trip is fine. Display via `lookupName(...)` or direct field read. |
-     | Any | 1:many or M:N | Scaffold a chained `<ChildService>.getAll({ filter: \`_<parentid>_value eq '${id}'\`, select: [...] })`. Calc columns CANNOT traverse 1:many or M:N — chained fetch is the ONLY pattern. |
+     | `formatted-lookup` | Select the real `_<lookup>_value` field and render `lookupName(record, '<lookupLogicalName>')`. |
+     | `chained-fetch` | Perform one bounded related `get` / `getAll` in the screen load step, never inside `map()` or `renderItem`. |
+     | `external-projection-required` | Render no fake fallback data. Return `BLOCKED` and name the field/source. The user must supply a supported server-owned projection outside this workflow. |
 
-  3. **Verify before exit.** Every UI field in your spec must have either (a) a `select` entry on the primary fetch (covered by direct column or calc column) OR (b) a chained fetch path. If a field has neither, return `BLOCKED [<screen_name>]: field <field> on <screen> has no fetch path — add to spec or add calc column`.
+  2. **Verify before exit.** Every UI field in your spec must have either a
+     primary select, formatted lookup annotation, or bounded chained fetch.
+     Otherwise return `BLOCKED [<screen_name>]: field <field> requires an external projection`.
 
-  4. **TODO comment shape** (when emitting at step 2 / list branch):
+  3. **TODO comment shape** for an external projection:
 
      ```ts
      // TODO(cross-entity-read): screen displays <field> from related <entity>.
-     // Re-run /setup-datamodel (or /add-dataverse for existing apps) and add calc
-     // column <prefix>_<field>_calc to <primary_entity> for one-round-trip reads.
-     // List screens MUST NOT chain fetches in renderItem — N+1 storm.
+     // Supply a maker-created formula column or another server-owned projection,
+     // then rerun Dataverse reconciliation. Never chain reads in renderItem.
      ```
 
 - **HARD RULE — server-managed columns are NEVER in a create or update payload.** The Dataverse server owns these fields; including them in a `*Service.create({...})` or `*Service.update({...})` returns HTTP 400 on every save. Generated `create()` types may include server-managed fields (`ownerid`, `statecode`, primary IDs, etc.) because they mirror the full model; do **not** satisfy those types by emitting junk values. For any screen with create/update behavior, use a narrow write helper/type whose input contains only editable fields. If the skeleton imports an app-level helper, call it; otherwise define the helper inside your assigned screen file. Do **not** create or modify shared `src/utils/`, `src/hooks/`, or service files from a screen-builder. Forbidden keys in any create/update payload: