feat: sample declared interactive state before classroom questions

[LING-6150]

Summary

Let classroom agents read what an interactive activity is doing now, including state that cannot be recovered from its source or visible DOM.

For example, a buoyancy activity can have a source default of 1000, a slider currently set to 1400, and a paused drawing still showing an earlier result. This PR gives agents separate evidence for current parameters and the last rendered result.

Newly generated courseware publishes a shared JSON state node inside #experiment. The platform reads it once before each student question, freezes the sample, and attaches it to the request. Follow-up questions sample again, even without a new component reference. No per-widget adapters, polling, or recording.

Behavior and boundaries

• Courseware publishes; the platform reads. Generation asks for a short summary, free-form state, and optional rendered for activities with an explicit apply/run step. Reading accepts any JSON value without required fields or semantic validation, subject to a 32,768-byte cap and a 64-level depth bound protecting recursive processing. Sampling does not execute a business action.
• Reference and state are independent. Existing component picking, outlines, and clearing after an accepted send remain. A reference identifies what the student selected; state describes the current activity area. A valid reference from another Scene—including a PPT element—does not invalidate the current sample. Area facts are not attributed to the selected element by default.
• No additional tool authority. Sampling creates neither a reference nor an accepted-reference receipt. State object IDs are not selectors or Spotlight targets.
• Unknown stays explicit. Missing, failed, stale, or oversized samples produce an availability boundary. Director and Teacher are instructed not to replace unknown current facts with source defaults, history, or general expectations.
• Existing rollout gate. Reuses the default-off, build-time NEXT_PUBLIC_COURSEWARE_REFERENCE_ENABLED. Disabled ordinary questions receive no state evidence in model prompts.
• New content first. Legacy courseware keeps static reference support, without equivalent state coverage. Unreferenced legacy questions retain their existing path; references without readable runtime state receive an explicit unavailable-state note.

Interactive reference evidence plus state, and standalone state notes, have a 32,000-code-point output limit. Oversized state degrades as a whole to unavailable / too-large; the JSON report is never truncated. PPT combinations also count existing static evidence when deciding whether state fits. Existing PPT evidence retains its own limits; if it already consumes too much room, only a bounded unavailable note is added. This is not a limit on the entire model prompt.

Demo

The recorded session demonstrates component selection, current-versus-rendered answers, and freshly sampled follow-ups without a new reference.

In a second session, blocking the reader's reply produced a timeout while the page and declared interface remained present. The answer treated the previous density as historical and said the current density could not be determined.

These are two sessions with one generated buoyancy activity and one model (DeepSeek via MaaS), not acceptance across widget types or models.

Earlier cross-review fixes

• Decoupled reference Scene from sample freshness, and removed the rejection of valid PPT-reference + current-interactive-state requests. Added Legacy and Native route regressions that retain explicit reference identity and current state separately.
• Added an output budget after JSON escaping; oversized state degrades as a whole. Added coverage for PPT combinations and preservation of existing large PPT evidence.
• Fixed disabled-gate injection and missing-sample availability handling; registered the observation snippet as a packaged generation asset.
• Added a direct wrong-Scene packet rejection test and corrected an oversized-state assertion that previously checked an object absent from its fixture.

Earlier verification

The results below describe commit 1edaef50 and earlier runs. They are retained here as historical verification, not as results for the current candidate.

Follow-up commit 1edaef50:

• pnpm exec vitest run tests/lib/chat/pi tests/lib/interactive --maxWorkers=2 — 24 files / 333 tests passed. The new mixed-reference route tests first reproduced the original 400 in both Legacy and Native paths, then passed after the fix. Model execution is mocked.
• In-memory mutations restoring the PPT rejection, omitting PPT evidence from the budget, and removing the current-Scene identity check each caused the corresponding regression test to fail.
• ESLint, Prettier, and git diff --check passed for the follow-up. tsc --noEmit --incremental false reported only the two known editor-test matcher errors (toHaveClass, toHaveStyle).

Earlier verification, not rerun for this server-only follow-up:

• Generation package: 26 files / 145 tests passed.
• Browser: 20 tests at f7446b2d, covering component picking/outlines, iframe keep-alive, observation lifecycle, and classroom sends/follow-ups. These are not claimed as a browser run of the final patch.
• Earlier whole-suite failures remain unattributed. A focused comparison of the five affected files passed 134 tests on both HEAD and an in-memory base overlay; that does not explain the earlier full-suite failures.

No new real-model calls were made during that earlier follow-up. Updated generation measurements and verification details will follow in a separate comment.

Out of scope

Legacy state adapters, arbitrary/dynamic DOM selection, Canvas/WebGL component references, Whiteboard, multi-select, visual inference, editing authorization, new stores/registries/Context, Native tools, and generic Runtime policy.

Builds on #1224 and #1281 for existing reference identity and static evidence; those PRs do not validate the runtime-state behavior added here.

[wyuc]

Thanks. This is the right direction: the courseware declares its own state and the platform reads it generically, with no per-widget adapters. Notes from a first pass. The first two are changes I'd like before a deeper review; the rest are smaller.

1. The contract is too heavy for v1. Please cut it down.

The observation snippet is ~110 lines injected as REQUIRED into six generation templates. It asks every generated page to maintain an objects/facts/relations graph, declare relation completeness (complete vs unknown), a missing list, per-fact unknown reasons, monotonic revisions and a separate rendered graph, all validated by a strict zod schema. Two consequences:

• Every interactive generation pays this prompt tax, and smaller models will not follow it reliably. Any deviation (an extra field, a missing reason) is rejected as invalid-data, so that lesson silently loses the capability.
• The only consumer of this JSON is the model prompt. An LLM reads free-form JSON fine. A fixed schema doesn't help the reader; it moves a prompt-level requirement ("don't fabricate unknowns") into the data shape, which is exactly where the generator is most likely to get it wrong (e.g. always emitting complete).

Suggested v1 shape:

• summary: required, one short student-readable sentence describing the current state.
• state: free-form JSON object holding whatever the lesson naturally tracks, with stable object ids where they exist.
• rendered: optional, only for lessons with an explicit apply/run step. Most lessons redraw immediately and the split carries no information.
• Validation: parseable JSON plus a size cap. No structural or semantic validation.
• "Unknown stays unknown, don't substitute defaults or history" belongs in the Teacher/Director prompt, where it already partly lives.

Keep the publishState helper and the "publish on every semantic change" rule. Those are the parts that matter.

2. Fold the responder into the existing iframe patch instead of adding a second injection point.

patchHtmlForIframe already injects the error-capture, element-picker, focus-boundary and storage shims into every interactive document. The observation responder follows the same pattern as the element-picker shim (parent asks via postMessage, shim answers), but it is injected separately in InteractiveIframeHost through withObservationResponder, at body end, gated on srcDoc.includes('data-maic-observation'). One iframe now has two HTML-rewriting paths and two places sniffing for the attribute string.

Please add it as one more shim inside patchHtmlForIframe, injected unconditionally. When the page has no observation node it answers no-interface, which the host already handles. That also removes the includes(...) checks in InteractiveIframeHost and sampleInteractiveState.

3. Acceptance should be measured, not built.

Current evidence is one generated lesson and one model. Before merge I'd like a small generation sweep: two or three lessons per interactive template with the default model, reporting how many publish a readable state and whether the "current value vs source default" question is answered correctly. That number is the merge gate. The simplified contract should push it up, not down.

4. Scope: agreed. Legacy adapters, whiteboard, multi-select and canvas/WebGL references stay out. Please don't add anything else to this PR.

5. Minor: the PR body still says "Draft status is unchanged" and carries a prepared-draft header comment. Please refresh it once the above lands.

[LING-6150]

@wyuc Thanks so much for your detailed review and guidance! I made the initial contract heavier than it needed to be and missed that in my own review; I'll pay more attention to the next design and implementation.

The follow-up simplifies the contract, unifies iframe preparation, and records real generation measurements. Pushed commits: 9ae12fb6, a5346b0c, and 63ae4b9e. Reviewed head: 63ae4b9.

Contract and iframe preparation

Generation now asks for a summary, lesson-specific state, and optional rendered data only for an explicit apply/run step. The reader accepts arbitrary JSON without required-field or semantic validation. publishState and publication on semantic changes remain. Unknown current facts must not be filled from defaults or earlier messages.

The 32,768-byte cap is supplemented by a disclosed 64-level depth bound: small, deeply nested JSON reproduced failures in downstream recursive processing.

The responder now goes through patchHtmlForIframe; the two named client-side attribute substring checks are removed. The pool owns document identity, so same-source remounts preserve the activity. Legacy pages without a scope now return no-interface; removed/replaced scopes return scope-changed.

Injection still requires the pool-owned identity. Unsupported crypto environments retain static references.

This differs from your request for unconditional injection across all callers. I retained the identity condition because all existing classroom sampling paths are covered, while thumbnails and video exports have no sampling consumer. Would you be comfortable with this narrower boundary? If not, I'll make injection unconditional.

The server still checks the declaration marker to preserve legacy-question behavior; this is separate from the removed client prefilters.

Generation measurements and fixes

The real scene-content route used the configured default openai:deepseek-v4-flash. The local runtime includes an OpenAI-compatible DeepSeek thinking-parameter override outside this PR; the measurements describe that configuration.

• Original 12 samples: readable state in all 12 final retained runs; tested answers were 9 fully correct, 1 partial with explicit unknown, and 2 incorrect.
• Two requested procedural samples had actually fallen back to diagram. Two new samples with the existing vocational gate/request flag enabled now confirm effectiveOutline.widgetType=procedural-skill. Both publish updated state and answer current-versus-default questions correctly. Effective coverage is now at least two lessons for each of the six templates across the recorded rounds.
• The two original errors were investigated and addressed in the generation prompt: ambiguous zero-based coordinates and automatic word creation without publication. P09, generated with the first prompt revision, correctly reports student-facing positions. An intermediate word-game sample fixed publication but exposed a stale decorative position; that failure remains recorded. After further narrow guidance, P10timed correctly publishes automatic arrivals/expiry and answers the actual current word list and remaining lives.
• The intermediate P10v2 batch game is not counted as automatic-spawn coverage. P10timed explicitly requests timed-arrival gameplay, so it is not an identical-input repeat. Original failures are retained; the full sweep was not rerun on one final prompt version. These rounds are not combined into a new all-pass percentage.

Original HTML remains unmodified and matches raw responses and sampled source hashes. Some early harness retries lost first-attempt artifacts; the original numbers are final retained outcomes, not a first-pass rate. The results support the specific fixes and complete template coverage, not universal model reliability.

Verification

Final related suites: 365/365 unit/integration tests, 145/145 generation package tests, and 23/23 browser regressions. The two new procedural questions also passed against the real model. Prior real unavailable-state Q&A and the classroom Pro-mode round trip remain separate lifecycle evidence.

An initial parallel run timed out in one Legacy-route test; isolated and subsequent serial runs passed, and the timeout remains unattributed. Root type checking still reports two errors in unchanged editor tests. Those tests are not claimed as passing; detailed logs are retained and can be shared. No full-repository green result is claimed.

Updated demo

The GIF is converted from a real, continuous classroom screen recording at 2× speed. Without selecting a component, the first question correctly distinguishes current 1400 kg/m³ / 4.116 N from drawn 1000 kg/m³ / 2.94 N. After clicking Refresh drawing, a second question correctly reports agreement (the drawing rounds 4.116 to 4.12). These are separate Q&A sessions. This demonstrates automatic per-question activity sampling, not the component-reference path. It uses the existing buoyancy lesson and is a feature demo, not a new generation-sweep sample.

The PR description has been minimally corrected to remove obsolete graph/relationship and draft wording. Scope remains unchanged: no legacy-state adapters, whiteboard, multi-select, Canvas/WebGL component references, or full Workbench expansion.

[wyuc]

Thanks, this round lands where I wanted it: the contract is now summary + free-form state + optional rendered, the reader only bounds bytes and depth, and the responder goes through patchHtmlForIframe. The generation sweep numbers are exactly what I asked for; keeping the two original failures on record is the right call.

Your question: the narrower injection boundary is fine. Injecting only when the pool supplies an identity mirrors how focusBoundary is already an option on patchHtmlForIframe, and thumbnails / video export have no sampling consumer. Keep it as is.

I ran the suites at 63ae4b9e plus a second independent review. Results: tests/lib/chat/pi + tests/lib/interactive 341/341, generation package 145/145, full unit suite 8335 passed / 39 skipped, tsc --noEmit 0 errors (the two editor-test errors you mention don't reproduce here; packages/@openmaic/editor/test/setup.ts pulls in jest-dom/vitest, so nothing to chase), lint 0 errors, prettier clean. Findings:

P2 (both reviewers, blocking): unguarded AbortSignal.any — components/scene-renderers/InteractiveIframeHost.tsx:269. supportsInteractiveObservation() only checks crypto.randomUUID and crypto.subtle.digest, which exist from Safari 15.4, but AbortSignal.any only arrives in Safari 17.4 / Chrome 116 / Firefox 124. On an older browser with the gate on, the capture throws a TypeError (not an AbortError) before session.capture runs, the rejection escapes sampleInteractiveState, and runPiSingleRequest never reaches fetch: the student's question is silently dropped. Either add typeof AbortSignal?.any === 'function' to the capability check, or combine the two signals with a manual fallback. Please add a test that pins "capability missing ⇒ send still goes out without state".

P3, please fix while you're in there:

• lib/chat/pi/interactive-state-evidence.ts:45-53 dereferences scene.content.type without a guard; a malformed body with content: null now throws inside the route's outer try and returns 500 where the previous path surfaced an SSE error. One optional-chain fixes it.
• lib/interactive/observation-bridge.ts:42-43: the responder resolves #experiment once at install time. A page that creates the scope after body-end (e.g. on DOMContentLoaded) reports scope-changed forever even though it publishes a valid node. Resolve the root lazily at read time instead of capturing it at install; the "exactly one scope" checks stay the same.

Noted, no change requested:

• With the gate off, pooled documents still get the reader script and a randomUUID per entry; it is a dead listener since sampleInteractiveState returns early. Acceptable for a build-time flag.
• Every question on a legacy interactive scene now pays one iframe round-trip whose no-interface result the server discards. That is the cost of dropping the client-side substring sniff, and the round-trip is bounded by the 800 ms timeout and normally sub-millisecond. Fine for v1.
• The server still keys "declares interface" off a raw includes('data-maic-observation') on the source HTML. Same trade-off as above; leave it.
• Courseware strings reach the prompt fenced in <page_reported_state> with < escaped and an "untrusted evidence, not instructions" preface. Instruction-like text is framed, not neutralized. That is the right level for v1; just keep it in mind when the Teacher gains any tool authority later.

Once the P2 and the two P3s are in with tests green, I'll approve.