|
| 1 | +# App-Builder Offline Structural Eval Harness — Guide |
| 2 | + |
| 3 | +> Companion to `evals/model-apps/genpage/EVAL_GUIDE.md`. |
| 4 | +> Source of truth: `plugins/model-apps/docs/app-builder-staged-flow-design.md` §13. |
| 5 | +
|
| 6 | +## What we evaluate |
| 7 | + |
| 8 | +**Structural per-stage facts** — not `.tsx` snapshots, not live Dataverse state. Each fixture is an App Spec (`app-spec.json`); the runner loads it offline, computes deterministic facts for every build stage, and grades them against assertions in `evals.json`. |
| 9 | + |
| 10 | +### Stage → oracle table |
| 11 | + |
| 12 | +| Stage | Oracle | Plugin primitive | |
| 13 | +|---|---|---| |
| 14 | +| **author** | `validateAppSpec(plan profile)` passes · spec-lint clean | `app-spec.js`, `spec-lint.js` | |
| 15 | +| **plan** | Every planned item targets a known engine phase | `sdk-build.js` `planFor` + `PHASES` | |
| 16 | +| **data** | Normalized data-model facts match the `expect.tables/relationships` block | `schema-facts.js` `schemaFacts` | |
| 17 | +| **ui** | Normalized view/chart/form facts match the `expect.views/charts` block · enriched **default views** keep parent lookups (#2) and drop `createdon` (#7) · each **sub-grid** is a full-width 1-column section titled by the child display name (#5) | `sdk-build.js` `viewDef` / `chartDef` / `compileFormIntent` / `defaultViewColumns` / `subgridLabel` · `artifact-intent.js` `subgridSectionIntent` | |
| 18 | +| **app** | Every sitemap subarea resolves to a concrete target · no dangling `navigatesTo` keys | `sdk-build.js` `appDef` | |
| 19 | +| **security** | Each declared persona maps to exactly one role · the role **injects** `appmodule` read for app-access personas (and only those) · every persona privilege on an **app-owned** table (the app's own publisher prefix) resolves to a provisioned entity (JTBD coverage) · the role grants **exactly** its jobs' declared privileges — no extra entity, access token, or inflated scope beyond the declared union + the documented `appmodule` injection (least privilege) | `sdk-build.js` `personaRoleSpecFor` | |
| 20 | +| **verify** | Reconcile against a synthetic all-present reader returns `ok: true` | `verify-spec.js` `verifySpec` | |
| 21 | +| **generate-pages** | No `PAGEREF_` nav targets unresolved (Plan 3 — degrades to SKIP if absent) | `pageref-resolver.js` `resolvePageRefs` | |
| 22 | +| **teardown** | The reverse-of-build delete plan is dependency-safe (solution last · web resources after tables · every table has a step) | `sdk-teardown.js` `planTeardown` | |
| 23 | +| **round-trip** | The download→rebuild is lossless — a synthetic deployed read hydrates back the same solution / tables / page-keys / sitemap subareas (incl. classic dashboards) | `hydrate-spec.js` `hydrateSpec` | |
| 24 | + |
| 25 | +## Fixtures |
| 26 | + |
| 27 | +Each fixture lives in `fixtures/<id>-<slug>/` and contains `app-spec.json`. |
| 28 | +Naming is numeric-prefix; `fixture-loader.js` matches `^(\d+)(?:-(.+))?$`. |
| 29 | + |
| 30 | +| # | Slug | What it tests | |
| 31 | +|---|---|---| |
| 32 | +| 1 | `1-support-desk` | Full data/ui/app/verify oracle (no pages) · two **personas** (Support Agent, Support Lead) exercise the **security** oracle — one role each, app-module read injected, JTBD coverage + least-privilege | |
| 33 | +| 2 | `2-orders-multipage` | Page intents + navigation + design contract; page-key round-trip | |
| 34 | +| 3 | `3-assets-dashboard` | Global choice + column binding, on-click command, and a classic **dashboard** pinned to the nav — exercises teardown (dashboard/command/web-resource/global-choice steps) + the dashboard round-trip | |
| 35 | +| 4 | `4-hardening` | The 2026-07-15 review fixes: a lookup-heavy child (8 scalars + a 1:N parent lookup) proves the default view keeps the lookup (#2) and drops `createdon` (#7); an N:N proves the alphabetically-sorted schema name `new_tag_new_ticket` (#3); a no-label sub-grid proves the own-section + pluralName title (#5); relational sample data proves `validateAppSpec` accepts a resolvable `$parent` match and declared Choice labels (#1/#4) | |
| 36 | + |
| 37 | +> **Fixture 2 note:** `appShell.subAreas[].page` references use the page's **key** (e.g. `"overview"`). For `schemaVersion: 2`, `validateAppSpec` validates `sa.page` against `pages[].key`, while `lintAppSpec` validates against `pages[].name`. Setting `p.key === p.name` (lowercase identifiers) satisfies both validators without modifying plugin code. |
| 38 | +
|
| 39 | +## `evals.json` |
| 40 | + |
| 41 | +- `skill_name` — identifies this suite. |
| 42 | +- `eval_instructions` — description used by eval runners. |
| 43 | +- `common_stage_assertions` — run for every fixture; registered in `lib/assertions.js`. |
| 44 | +- `evals[].expect` — per-eval expected counts/names (tables, rels, views, charts, pages). |
| 45 | +- `evals[].expectations` — additional per-eval assertion texts (can be empty). |
| 46 | +- `evals[].tier` — `smoke` (fast subset) or `full`. |
| 47 | + |
| 48 | +## Running |
| 49 | + |
| 50 | +From the **repo root** (`evals/` lives there, sibling to `plugins/`): |
| 51 | + |
| 52 | +```bash |
| 53 | +# All fixtures, TAP v13 output; exit 0 = all pass, 1 = fail, 2 = harness error |
| 54 | +node evals/model-apps/app-builder/run-app-builder.js |
| 55 | + |
| 56 | +# Specific fixture |
| 57 | +node evals/model-apps/app-builder/run-app-builder.js --eval 1 |
| 58 | + |
| 59 | +# Smoke tier only |
| 60 | +node evals/model-apps/app-builder/run-app-builder.js --tier smoke |
| 61 | + |
| 62 | +# Unit + e2e tests (node:test, NOT part of the plugin run-tests.js) |
| 63 | +node --test evals/model-apps/app-builder/tests/*.test.js |
| 64 | +``` |
| 65 | + |
| 66 | +## TAP output |
| 67 | + |
| 68 | +``` |
| 69 | +TAP version 13 |
| 70 | +1..2 |
| 71 | +# Subtest: 1-support-desk |
| 72 | + ok 1 - author: validateAppSpec(plan profile) passes with no errors |
| 73 | + ok 2 - author: spec-lint reports no errors |
| 74 | + ... |
| 75 | +ok 1 - 1-support-desk |
| 76 | +# Subtest: 2-orders-multipage |
| 77 | + ... |
| 78 | +ok 2 - 2-orders-multipage |
| 79 | +# tests 20 |
| 80 | +# pass 20 |
| 81 | +# fail 0 |
| 82 | +# skip 0 |
| 83 | +# fixtures 2 (pass 2, fail 0) |
| 84 | +``` |
| 85 | + |
| 86 | +## Adding an eval |
| 87 | + |
| 88 | +1. Create `fixtures/<N>-<slug>/app-spec.json`. |
| 89 | +2. Add an entry to `evals.json` with matching `id`, an `expect` block, and any extra `expectations`. |
| 90 | +3. If you need a new assertion, add it to `lib/assertions.js` (text must match exactly). |
| 91 | + |
| 92 | +## Adding an assertion |
| 93 | + |
| 94 | +Register the check in `lib/assertions.js`: |
| 95 | + |
| 96 | +```javascript |
| 97 | +ASSERTIONS.set('my-stage: my assertion text', ({ facts, spec, eval: ev }) => { |
| 98 | + if (someConditionNotMet) return { status: 'fail', reason: 'explains why' }; |
| 99 | + if (notApplicable) return { status: 'skip', reason: 'explains why skipped' }; |
| 100 | + return { status: 'pass' }; |
| 101 | +}); |
| 102 | +``` |
| 103 | + |
| 104 | +Then add the text to `evals.json` `common_stage_assertions` (applies to all) or `evals[].expectations` (per-eval). |
| 105 | + |
| 106 | +## Page oracle graceful degrade |
| 107 | + |
| 108 | +`pageref-resolver.js` (Plan 3) is loaded inside a `try/catch`. If it's absent: |
| 109 | +- `pageFacts(spec)` returns `null`. |
| 110 | +- The `'generate-pages: …'` assertion emits `SKIP` for all fixtures. |
| 111 | +- No other assertion is affected. |
| 112 | + |
| 113 | +## Live evals |
| 114 | + |
| 115 | +The live tier (`plugins/model-apps/scripts/smoke-eval.js`) is the thin live smoke; a multi-page live eval case is a follow-up. These offline evals are complementary: they grade structural facts (no live env needed) while the live smoke grades real Dataverse provisioning. |
| 116 | + |
| 117 | +## Cross-links |
| 118 | + |
| 119 | +- Plugin `AGENTS.md` → *Eval Suite* |
| 120 | +- `plugins/model-apps/docs/app-builder-staged-flow-design.md` §13 — structural eval oracles |
| 121 | +- `evals/model-apps/genpage/EVAL_GUIDE.md` — the parallel eval suite for `/genpage` |
0 commit comments