From e28abcc29f3b3ddd2027be7fa67cb15193346047 Mon Sep 17 00:00:00 2001 From: PureWeen <223556219+Copilot@users.noreply.github.com> Date: Tue, 25 Aug 2026 15:40:18 -0500 Subject: [PATCH 1/4] Require Selenium for browser-owned Components regressions Document the permanent regression boundary, align the interactive validation skill handoff, and add focused Vally coverage for browser and lower-boundary test selection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../skills/validate-blazor-feature/SKILL.md | 30 +++++-- .../validate-blazor-feature/eval.vally.yaml | 88 +++++++++++++++++++ src/Components/AGENTS.md | 32 +++++-- 3 files changed, 139 insertions(+), 11 deletions(-) diff --git a/.github/skills/validate-blazor-feature/SKILL.md b/.github/skills/validate-blazor-feature/SKILL.md index f4d8caefaa9b..e2f65a5e3cdf 100644 --- a/.github/skills/validate-blazor-feature/SKILL.md +++ b/.github/skills/validate-blazor-feature/SKILL.md @@ -1,12 +1,12 @@ --- name: validate-blazor-feature description: >- - Validate a Blazor feature or behavior interactively in a browser using the canonical Components samples in this repo, before writing E2E tests. USE FOR exercising a Blazor change in src/Components (a render-mode behavior, an interactive component, enhanced navigation, forms, streaming, prerendering), deciding which sample to use, where to add a test page, how to set the render mode (Server/WebAssembly/Auto/static SSR), how to confirm the app is actually interactive (not just static SSR), and how to inspect the browser console and network for errors. Covers BlazorWebAppGlobal, BlazorWebAppPerPage, and BlazorWebAssemblyStandalone, launching them against the in-tree framework, and driving them with the Playwright MCP browser tools. DO NOT USE FOR writing the permanent E2E/Selenium tests themselves, non-Components areas, or unit tests. + Validate a Blazor feature or behavior interactively in a browser using the canonical Components samples in this repo, before selecting or writing permanent test coverage. USE FOR exercising a Blazor change in src/Components (a render-mode behavior, an interactive component, enhanced navigation, forms, streaming, prerendering), deciding which sample to use, where to add a test page, how to set the render mode (Server/WebAssembly/Auto/static SSR), how to confirm the app is actually interactive (not just static SSR), and how to inspect the browser console and network for errors. Covers BlazorWebAppGlobal, BlazorWebAppPerPage, and BlazorWebAssemblyStandalone, launching them against the in-tree framework, and driving them with the Playwright MCP browser tools. DO NOT USE FOR writing the permanent E2E/Selenium tests themselves, non-Components areas, or unit tests. --- # Validate a Blazor feature with the Components samples -Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, and check for errors. Validate behavior before writing E2E tests; remove the sample code afterward (`git checkout`/`git clean`) once the E2E test covers it. +Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, and check for errors. Validate behavior before selecting permanent coverage; remove the sample code afterward (`git checkout`/`git clean`) once the selected permanent test covers it. ## 1. Pick the sample and where the page goes @@ -86,12 +86,32 @@ After interacting, inspect the console. **Scope it to the current page**: call ` See [references/error-checks.md](references/error-checks.md) for the catalog of common failures, the symptom each produces, and the fix. -## 6. Finish +## 6. Record the permanent regression boundary + +Before handing off to permanent tests, record: + +- **Behavior owner**: the subsystem that owns the behavior. +- **Production producer**: the real mechanism that creates the disputed preconditions. +- **Final observable**: the material result the regression test must assert. +- **Selected permanent surface**: the test suite that exercises that producer and observable. +- **Lower-boundary false-pass risk**: how a lower-level test could pass while the shipped behavior still fails. + +For browser-owned behavior (DOM measurement, layout or geometry, scrolling, browser observers, browser event ordering, browser-dependent JS interop, navigation, focus or selection, and rendering or rehydration), hand off to permanent C# Selenium coverage under `src/Components/test/E2ETest`. Extend an existing asset and test class when practical. Require the identical Selenium assertion to be red without the fix and green with it, plus the nearest opposite and adjacent same-producer controls. A browser fix remains blocked while Jest is its only regression proof. + +Do not recommend adding or retaining Jest or `.test.ts` coverage for the same browser scenario, including as supplemental coverage or in an existing test file. Synthetic geometry, mocked observers or events, and direct state or callback injection are temporary diagnostic probes and stay outside the production change. + +JavaScript or TypeScript unit coverage remains appropriate for a genuinely pure helper when its preconditions and observable are browser-independent plain inputs and outputs. It does not qualify if the test mocks or stubs browser observers, layout APIs, or geometry such as `getComputedStyle`, `getBoundingClientRect`, `getClientRects`, `scrollTop`, `scrollHeight`, `clientHeight`, or `offsetHeight`. Managed or service behavior fully owned and observable below the browser remains at that faithful lower boundary. + +If WebDriver cannot perform or observe one exact operation, name that limitation and permit only the smallest existing JavaScript helper or `IJavaScriptExecutor` snippet for that step. Keep the permanent scenario orchestration and final user-visible assertion in C# Selenium. + +This skill validates the sample interactively and records the handoff. It does not write the permanent Selenium test. + +## 7. Finish - Stop the sample server by its specific PID. - Remove the Playwright artifacts folder (`.playwright-mcp/`) it drops into the working directory. -- Per the Components workflow, once an E2E test covers the behavior, remove the sample scenario code: `git checkout -- src/Components/Samples src/Components/WebAssembly/Samples` and `git clean -df -- src/Components/Samples src/Components/WebAssembly/Samples`. +- Per the Components workflow, once the selected permanent test covers the behavior, remove the sample scenario code: `git checkout -- src/Components/Samples src/Components/WebAssembly/Samples` and `git clean -df -- src/Components/Samples src/Components/WebAssembly/Samples`. ## Completion criteria -The validation is done when: the page loaded (200, framework JS served), the behavior was exercised by a real interaction, the resulting state change was observed in a snapshot, and the console shows no real errors for the page under test. Only then move on to writing the E2E test. +The validation is done when: the page loaded (200, framework JS served), the behavior was exercised by a real interaction, the resulting state change was observed in a snapshot, and the console shows no real errors for the page under test. Only then record the five-field handoff to the selected permanent surface; browser-owned behavior moves to C# Selenium E2E coverage. diff --git a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml index 68a94678cc22..e3fc1898354e 100644 --- a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml +++ b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml @@ -105,3 +105,91 @@ stimuli: - "Says to add/verify the @rendermode (per page, or on Routes/HeadOutlet for a global app)" - "Considers that a WebAssembly/Auto component must be in the .Client project, otherwise it won't be interactive" - "Suggests confirming the framework JS (blazor.web.js) is built and served and checking the browser console, rather than only guessing" + + - name: choose-selenium-for-real-browser-geometry + prompt: | + I am fixing a Components regression where a virtualized list sometimes jumps after a resize. The + production path combines real scroll geometry with ResizeObserver and IntersectionObserver callbacks. + I plan to add a Selenium scenario for the visible jump. Can I also keep a fast supplemental test in an + existing .test.ts file by defining scrollHeight/clientHeight with Object.defineProperty, stubbing both + observers, and invoking their callbacks directly? What should the production change include, and what + boundary decision should I record before handing the fix off? + graders: + - type: output-matches + config: + pattern: '(?i)(selenium|E2E)' + - type: output-matches + config: + pattern: '(?i)(behavior owner|production producer|final observable|lower-boundary false-pass)' + - type: prompt + rubric: + - "Selects permanent C# Selenium coverage under src/Components/test/E2ETest because the production behavior is owned by real browser geometry, scrolling, and observers" + - "Rejects the synthetic TypeScript/Jest test as production regression proof, including as supplemental coverage, while allowing it only as a temporary diagnostic outside the production change" + - "Records all five boundary fields: behavior owner, production producer, final observable, selected permanent surface, and lower-boundary false-pass risk" + - "Explains that mocked geometry and directly invoked observer callbacks can pass without proving that the browser producer reaches the shipped state" + - "Requires strict Selenium red/green with the same assertion, the nearest opposite control, and adjacent same-producer controls" + + - name: reject-fast-jest-only-observer-proof + prompt: | + Please keep this fix small and fast: add only a Jest test for a ResizeObserver-driven component bug. + The test can mock ResizeObserver and getBoundingClientRect, and CI should not pay for Selenium. I need + the production PR ready today. What test plan should I use? + graders: + - type: output-matches + config: + pattern: '(?i)(selenium|E2E)' + - type: output-matches + config: + pattern: '(?i)(block|blocked|not.*Jest|Jest.*not|only regression proof)' + - type: prompt + rubric: + - "Fails closed despite the request for speed and requires a permanent C# Selenium scenario for the browser-owned ResizeObserver behavior" + - "States that the fix remains blocked while mocked Jest coverage is its only regression proof" + - "Does not compromise by proposing the Jest test as primary, supplemental, faster, or more precise production coverage" + - "Allows mocked observer or geometry work only as a temporary diagnostic outside the production change and removes it after Selenium supersedes it" + - "Requires the identical Selenium assertion to fail without the fix and pass with it, with the nearest opposite control and adjacent same-producer controls" + + - name: allow-pure-typescript-helper-test + prompt: | + I extracted a TypeScript helper that parses a comma-separated component option string into a plain + object. It accepts only a string, returns only strings and booleans, reads no globals, and does not + touch the DOM, navigation, events, layout, observers, or JS interop. A bug is entirely in its escaping + rules. Should the permanent regression test use Selenium or a focused TypeScript unit test? For + comparison, where should behavior stay when a managed service owns it completely and its injected + service contract exposes every precondition and final observable without involving a browser? + graders: + - type: output-matches + config: + pattern: '(?i)(TypeScript|unit test|pure helper)' + - type: output-matches + config: + pattern: '(?i)(plain input|plain output|browser-independent|no.*fidelity|lower boundary)' + - type: prompt + rubric: + - "Permits focused TypeScript unit coverage because both the helper's preconditions and observable are browser-independent plain inputs and outputs" + - "Explains that Selenium adds no fidelity for a parsing and escaping defect wholly owned by the pure helper" + - "Keeps the exception narrow and would disqualify it if the test mocked browser observers, layout APIs, geometry, navigation, events, or browser-dependent interop" + - "Does not reflexively require Selenium merely because the helper is written in TypeScript or used by Components" + - "Still records a coherent five-field boundary decision with the pure helper as owner and the focused unit suite as the selected surface" + - "Keeps the fully owned and observable managed-service behavior at its faithful managed lower boundary rather than requiring Selenium" + + - name: keep-selenium-with-minimal-javascript-step + prompt: | + I am validating focus and selection restoration after a component rerender. C# Selenium can drive the + interaction and assert the user-visible edited value, but WebDriver has no precise API for setting an + input's selection range to exact UTF-16 offsets. May I use JavaScript, and if so how should the permanent + test be divided? + graders: + - type: output-matches + config: + pattern: '(?i)(IJavaScriptExecutor|setSelectionRange|JavaScript)' + - type: output-matches + config: + pattern: '(?i)(C#|Selenium).*(final|assert|orchestrat)|(?:final|assert|orchestrat).*(C#|Selenium)' + - type: prompt + rubric: + - "Keeps the permanent scenario in C# Selenium rather than moving it to a JavaScript or TypeScript unit test" + - "Names the exact limitation: WebDriver cannot precisely set the input selection range to exact UTF-16 offsets" + - "Allows only a minimal existing helper or IJavaScriptExecutor snippet for the setSelectionRange step" + - "Keeps navigation, rerender orchestration, and the final user-visible edited-value assertion in C# Selenium" + - "Does not generalize the exception into permission for JavaScript-owned scenario orchestration, mocked selection state, or a JavaScript final assertion" diff --git a/src/Components/AGENTS.md b/src/Components/AGENTS.md index eb636a899f12..a4b1a425976e 100644 --- a/src/Components/AGENTS.md +++ b/src/Components/AGENTS.md @@ -20,12 +20,32 @@ For implementation work: - Research the problem area using the microsoft docs, existing code, git history, and logging on the sample project. - Implement the fix or feature in the sample project first. - Test the fix or feature interactively using Playwright. -- Once the fix or feature is validated in the sample, implement E2E tests for it. - - When you create an E2E test. First execute it interactively with Playwright. +- Once the fix or feature is validated in the sample, select its permanent test surface using the boundary below. + - For browser-owned behavior, implement a C# Selenium E2E test and first execute its scenario interactively with Playwright. - If an E2E test is failing, debug it by running the test server manually and navigating to the scenario in a browser. -- Only after the E2E tests are passing, remove the sample code you added in the Samples projects. +- Only after the selected permanent tests are passing, remove the sample code you added in the Samples projects. - Use `git checkout` and `git clean -fd` to remove the sample code. +### Permanent regression test boundary + +Before selecting a permanent test surface, record these five fields: + +- **Behavior owner**: the subsystem that owns the behavior. +- **Production producer**: the real mechanism that creates the disputed preconditions. +- **Final observable**: the material result the regression test must assert. +- **Selected permanent surface**: the test suite that exercises that producer and observable. +- **Lower-boundary false-pass risk**: how a lower-level test could pass while the shipped behavior still fails. + +Browser-owned behavior requires permanent C# Selenium coverage under `src/Components/test/E2ETest`. This includes DOM measurement, layout and geometry, scrolling, browser observers (`ResizeObserver`, `IntersectionObserver`, and `MutationObserver`), browser event ordering, browser-dependent JS interop, navigation, focus and selection, and rendering or rehydration. Extend existing test assets and classes when practical. + +Do not add or retain Jest or `.test.ts` coverage in the production change for the same browser scenario, whether described as primary, supplemental, faster, or more precise, including additions to an existing `.test.ts` file. Existing TypeScript tests are grandfathered exceptions, not precedent. Synthetic geometry, mocked observers or events, and direct state mutation or callback invocation are temporary diagnostic probes outside the production change; remove them after Selenium supersedes them. + +Permanent JavaScript or TypeScript unit tests are appropriate only for genuinely pure helpers whose preconditions and observable are browser-independent plain inputs and outputs. The helper is not pure for this purpose when its test mocks or stubs browser observers, layout APIs, or geometry, including `getComputedStyle`, `getBoundingClientRect`, `getClientRects`, `scrollTop`, `scrollHeight`, `clientHeight`, `offsetHeight`, or geometry installed with `Object.defineProperty`. Managed or service behavior that is fully owned and observable below the browser remains at its faithful lower boundary. + +If WebDriver cannot perform or observe one operation, keep the permanent C# Selenium scenario, name the exact WebDriver limitation, and use only the smallest existing JavaScript helper or `IJavaScriptExecutor` snippet for that step. Keep scenario orchestration and the final user-visible assertion in C#. + +For browser fixes, require strict red/green evidence: the identical Selenium assertion must fail for the expected reason without the fix and pass with it. Include the nearest opposite and adjacent same-producer controls. The fix remains blocked while Jest is its only regression proof. + ### Code clarity and durable knowledge - Before adding a comment, make local behavior discoverable through precise names, @@ -59,7 +79,7 @@ Together these cover every interactivity platform (Server/WebAssembly/Auto/None) **Always start by adding your feature scenario to whichever sample matches the render mode you need.** This allows you to: - Quickly iterate on the implementation - Test the feature interactively in a real browser -- Verify the feature works before writing formal E2E tests +- Verify the feature works before writing its selected permanent tests - Debug issues more easily with full logging capabilities 3. **Debug when needed**: @@ -70,9 +90,9 @@ Together these cover every interactivity platform (Server/WebAssembly/Auto/None) 4. **Validate the sample works** - You must have a validated, working sample in the Samples folder before proceeding. Use Playwright to confirm the feature works end-to-end in the browser. -5. **Implement E2E tests** - Only after the sample is validated, implement E2E tests for it. +5. **Implement permanent tests** - Only after the sample is validated, select the permanent test surface using the boundary above. Browser-owned behavior requires a C# Selenium E2E test. -6. **Clean up sample code** - After your E2E tests are passing, remove the sample code you added to the Samples projects. The sample was only for development and interactive testing; the E2E tests now provide the permanent test coverage. Use `git checkout -- src/Components/Samples` and `git clean -df -- src/Components/Samples` to remove the sample code. +6. **Clean up sample code** - After the selected permanent tests are passing, remove the sample code you added to the Samples projects. The sample was only for development and interactive testing; the selected tests now provide the permanent coverage. Use `git checkout -- src/Components/Samples` and `git clean -df -- src/Components/Samples` to remove the sample code. ## Build Tips From 919c3e40705d581a5bf2c1a38f1be68d219b466a Mon Sep 17 00:00:00 2001 From: PureWeen <223556219+Copilot@users.noreply.github.com> Date: Tue, 25 Aug 2026 17:02:33 -0500 Subject: [PATCH 2/4] Harden Components test policy evals Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../validate-blazor-feature/eval.vally.yaml | 80 ++++++++++++++++--- src/Components/AGENTS.md | 4 +- 2 files changed, 70 insertions(+), 14 deletions(-) diff --git a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml index e3fc1898354e..e4f836bde934 100644 --- a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml +++ b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml @@ -10,8 +10,9 @@ defaults: scoring: weights: - output-matches: 0.3 - prompt: 0.7 + output-matches: 0.1 + output-not-matches: 0.41 + prompt: 0.49 threshold: 0.6 stimuli: @@ -120,11 +121,14 @@ stimuli: pattern: '(?i)(selenium|E2E)' - type: output-matches config: - pattern: '(?i)(behavior owner|production producer|final observable|lower-boundary false-pass)' + pattern: '(?is)(?=.*\bbehavior owner\b)(?=.*\bproduction producer\b)(?=.*\bfinal observable\b)(?=.*\bselected permanent surface\b)(?=.*\blower[- ]boundary false[- ]pass risk\b)' + - type: output-not-matches + config: + pattern: '(?is)(?:(? Date: Wed, 26 Aug 2026 10:40:54 -0500 Subject: [PATCH 3/4] Generalize deterministic Components E2E workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../skills/validate-blazor-feature/SKILL.md | 34 +++-- .../validate-blazor-feature/eval.vally.yaml | 122 +++++++++++++----- src/Components/AGENTS.md | 19 ++- 3 files changed, 128 insertions(+), 47 deletions(-) diff --git a/.github/skills/validate-blazor-feature/SKILL.md b/.github/skills/validate-blazor-feature/SKILL.md index e2f65a5e3cdf..ebda1106dd89 100644 --- a/.github/skills/validate-blazor-feature/SKILL.md +++ b/.github/skills/validate-blazor-feature/SKILL.md @@ -1,12 +1,12 @@ --- name: validate-blazor-feature description: >- - Validate a Blazor feature or behavior interactively in a browser using the canonical Components samples in this repo, before selecting or writing permanent test coverage. USE FOR exercising a Blazor change in src/Components (a render-mode behavior, an interactive component, enhanced navigation, forms, streaming, prerendering), deciding which sample to use, where to add a test page, how to set the render mode (Server/WebAssembly/Auto/static SSR), how to confirm the app is actually interactive (not just static SSR), and how to inspect the browser console and network for errors. Covers BlazorWebAppGlobal, BlazorWebAppPerPage, and BlazorWebAssemblyStandalone, launching them against the in-tree framework, and driving them with the Playwright MCP browser tools. DO NOT USE FOR writing the permanent E2E/Selenium tests themselves, non-Components areas, or unit tests. + Validate a Blazor feature or regression in a real browser using the canonical Components samples before selecting permanent coverage. USE FOR exercising src/Components changes with Playwright; turning a browser failure into a temporary JavaScript diagnostic assertion; repeating it to assess determinism; investigating nondeterministic producer or timing behavior; replacing arbitrary Selenium sleeps with observable waits or explicit gates; choosing between a faithful Jest/jsdom manager or DOM contract and C# Selenium for a real browser producer; and recording the permanent-test handoff. Always use it when a Components task mentions deterministic browser evidence, flaky Selenium or Thread.Sleep, timing gates, or a JS-vs-Selenium boundary. It may author temporary diagnostic probes but does not write permanent E2E/Selenium tests. DO NOT USE FOR non-Components areas or implementing standalone unit tests. --- # Validate a Blazor feature with the Components samples -Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, and check for errors. Validate behavior before selecting permanent coverage; remove the sample code afterward (`git checkout`/`git clean`) once the selected permanent test covers it. +Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, check for errors, and reduce a reproduced failure to a deterministic diagnostic probe before selecting permanent coverage. Remove all temporary diagnostic and sample code once the selected permanent test covers it. ## 1. Pick the sample and where the page goes @@ -86,7 +86,22 @@ After interacting, inspect the console. **Scope it to the current page**: call ` See [references/error-checks.md](references/error-checks.md) for the catalog of common failures, the symptom each produces, and the fix. -## 6. Record the permanent regression boundary +## 6. Distill the reproduction and establish determinism + +After Playwright reproduces the failure, reduce the scenario to the smallest JavaScript probe and assertion that observes the same final browser state. Run it in the real page with Playwright evaluation or from temporary scratch code. This probe is diagnostic: do not add it to a Jest or `.test.ts` suite, do not include it in the production change, and remove it after the permanent scenario supersedes it. + +Repeat the unchanged scenario and probe a recorded, bounded number of times. When repository evidence does not suggest another count, 10 runs is a reasonable default confidence sample, not proof that the behavior is deterministic across machines, browsers, or load. Record the count, environment, assertion, and failure signature. Call the reproduction deterministic under the exercised conditions only when every run fails the same assertion for the same reason; never describe the bounded sample as proving determinism. + +If the runs do not agree, stop the permanent-test handoff and investigate the owning producer and timing: + +- Add targeted logging or browser observations that distinguish whether the producer ran, which precondition differed, and where ordering changed. +- Use a faithful unit or lower-boundary test when that module's contract is the final observable and the test exercises its real inputs. +- When managed timing must be controlled for the browser scenario, introduce an explicit test gate such as a `TaskCompletionSource`, endpoint, or test-only release action and wait for observable browser state before and after releasing it. +- Do not make the race appear stable by inserting an arbitrary fixed delay. + +When the browser reproduction is deterministic, record that the permanent C# Selenium scenario must preserve the same setup and final assertion. Prefer existing observable waits such as `Browser.True` and `Browser.Equal` (which poll with `WebDriverWait`) or an explicit test-controlled gate. A bounded polling helper may delay between observable checks when it also enforces a timeout; a fixed `Thread.Sleep` is not the synchronization mechanism for the regression assertion. + +## 7. Record the permanent regression boundary Before handing off to permanent tests, record: @@ -96,22 +111,25 @@ Before handing off to permanent tests, record: - **Selected permanent surface**: the test suite that exercises that producer and observable. - **Lower-boundary false-pass risk**: how a lower-level test could pass while the shipped behavior still fails. -For browser-owned behavior (DOM measurement, layout or geometry, scrolling, browser observers, browser event ordering, browser-dependent JS interop, navigation, focus or selection, and rendering or rehydration), hand off to permanent C# Selenium coverage under `src/Components/test/E2ETest`. Extend an existing asset and test class when practical. Require the identical Selenium assertion to be red without the fix and green with it, plus the nearest opposite and adjacent same-producer controls. A browser fix remains blocked while Jest is its only regression proof. +For browser-owned user-visible behavior (real DOM measurement, layout or geometry, scrolling, browser observers, browser scheduling or event ordering, browser-dependent JS interop, navigation, focus or selection, and rendering or rehydration), hand off to permanent C# Selenium coverage under `src/Components/test/E2ETest`. Extend an existing asset and test class when practical. Require the identical Selenium assertion to be red without the fix and green with it. Include a nearest-opposite control, meaning the closest scenario that must stay green, and, when meaningful, an adjacent control driven by the same production producer. A browser fix remains blocked while Jest is its only regression proof. -Do not recommend adding or retaining Jest or `.test.ts` coverage for the same browser scenario, including as supplemental coverage or in an existing test file. Synthetic geometry, mocked observers or events, and direct state or callback injection are temporary diagnostic probes and stay outside the production change. +Do not recommend adding or retaining Jest or `.test.ts` coverage as proof of the same browser scenario, including as supplemental coverage or in an existing test file. When synthetic geometry, mocked observers or events, or direct state or callback injection stand in for that browser scenario's real producer, they are temporary diagnostic probes outside the production change. -JavaScript or TypeScript unit coverage remains appropriate for a genuinely pure helper when its preconditions and observable are browser-independent plain inputs and outputs. It does not qualify if the test mocks or stubs browser observers, layout APIs, or geometry such as `getComputedStyle`, `getBoundingClientRect`, `getClientRects`, `scrollTop`, `scrollHeight`, `clientHeight`, or `offsetHeight`. Managed or service behavior fully owned and observable below the browser remains at that faithful lower boundary. +JavaScript or TypeScript unit coverage remains appropriate when the module's deterministic contract is the final observable and its harness faithfully supplies the contract inputs. This includes pure helpers, structural DOM algorithms over ordinary nodes, manager callback or timer-state contracts, and validation-engine contracts. These tests do not prove that real browser scheduling, layout, observers, or a user-visible producer path is reachable; claims about those behaviors remain Selenium scenarios. Managed or service behavior fully owned and observable below the browser remains at that faithful lower boundary. If WebDriver cannot perform or observe one exact operation, name that limitation and permit only the smallest existing JavaScript helper or `IJavaScriptExecutor` snippet for that step. Keep the permanent scenario orchestration and final user-visible assertion in C# Selenium. +Treat `src/Components/AGENTS.md` section "Permanent regression test boundary" as the normative policy when recording this handoff. + This skill validates the sample interactively and records the handoff. It does not write the permanent Selenium test. -## 7. Finish +## 8. Finish - Stop the sample server by its specific PID. - Remove the Playwright artifacts folder (`.playwright-mcp/`) it drops into the working directory. +- Remove any temporary JavaScript diagnostic probe. - Per the Components workflow, once the selected permanent test covers the behavior, remove the sample scenario code: `git checkout -- src/Components/Samples src/Components/WebAssembly/Samples` and `git clean -df -- src/Components/Samples src/Components/WebAssembly/Samples`. ## Completion criteria -The validation is done when: the page loaded (200, framework JS served), the behavior was exercised by a real interaction, the resulting state change was observed in a snapshot, and the console shows no real errors for the page under test. Only then record the five-field handoff to the selected permanent surface; browser-owned behavior moves to C# Selenium E2E coverage. +The validation is done when: the page loaded (200, framework JS served), the behavior was exercised by a real interaction, the resulting state change was observed in a snapshot, and the console shows no real errors for the page under test. For a regression, also record the temporary probe, bounded repeat result, and whether every run failed for the same reason. Only then record the five-field handoff to the selected permanent surface; deterministic browser-owned behavior moves to C# Selenium E2E coverage, while inconsistent behavior returns to producer/timing investigation or an explicit gate. diff --git a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml index e4f836bde934..6c6e1e7c67f5 100644 --- a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml +++ b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml @@ -10,10 +10,9 @@ defaults: scoring: weights: - output-matches: 0.1 - output-not-matches: 0.41 - prompt: 0.49 - threshold: 0.6 + output-matches: 0.2 + prompt: 0.8 + threshold: 0.8 stimuli: - name: place-a-webassembly-test-page @@ -119,19 +118,13 @@ stimuli: - type: output-matches config: pattern: '(?i)(selenium|E2E)' - - type: output-matches - config: - pattern: '(?is)(?=.*\bbehavior owner\b)(?=.*\bproduction producer\b)(?=.*\bfinal observable\b)(?=.*\bselected permanent surface\b)(?=.*\blower[- ]boundary false[- ]pass risk\b)' - - type: output-not-matches - config: - pattern: '(?is)(?:(? Date: Wed, 26 Aug 2026 11:02:32 -0500 Subject: [PATCH 4/4] Refine Components regression workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/skills/validate-blazor-feature/SKILL.md | 8 +++++--- .../validate-blazor-feature/eval.vally.yaml | 9 +++------ src/Components/AGENTS.md | 10 +++++----- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/.github/skills/validate-blazor-feature/SKILL.md b/.github/skills/validate-blazor-feature/SKILL.md index ebda1106dd89..1cdf59cf83c2 100644 --- a/.github/skills/validate-blazor-feature/SKILL.md +++ b/.github/skills/validate-blazor-feature/SKILL.md @@ -6,7 +6,7 @@ description: >- # Validate a Blazor feature with the Components samples -Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, check for errors, and reduce a reproduced failure to a deterministic diagnostic probe before selecting permanent coverage. Remove all temporary diagnostic and sample code once the selected permanent test covers it. +Workflow: pick a sample, add a scenario page, set the render mode, build, launch, drive it in a browser, and check for errors. When the run reproduces a browser regression or failure that needs permanent coverage, also reduce it to a deterministic diagnostic probe before selecting permanent coverage. Remove any temporary diagnostic and sample code once the selected permanent test covers it. ## 1. Pick the sample and where the page goes @@ -86,7 +86,9 @@ After interacting, inspect the console. **Scope it to the current page**: call ` See [references/error-checks.md](references/error-checks.md) for the catalog of common failures, the symptom each produces, and the fix. -## 6. Distill the reproduction and establish determinism +## 6. Distill the reproduction and establish determinism (browser regressions only) + +Apply this section only when steps 4-5 reproduced a browser regression or failure that needs permanent coverage. Skip it for new-feature validation with no failure to reduce. After Playwright reproduces the failure, reduce the scenario to the smallest JavaScript probe and assertion that observes the same final browser state. Run it in the real page with Playwright evaluation or from temporary scratch code. This probe is diagnostic: do not add it to a Jest or `.test.ts` suite, do not include it in the production change, and remove it after the permanent scenario supersedes it. @@ -99,7 +101,7 @@ If the runs do not agree, stop the permanent-test handoff and investigate the ow - When managed timing must be controlled for the browser scenario, introduce an explicit test gate such as a `TaskCompletionSource`, endpoint, or test-only release action and wait for observable browser state before and after releasing it. - Do not make the race appear stable by inserting an arbitrary fixed delay. -When the browser reproduction is deterministic, record that the permanent C# Selenium scenario must preserve the same setup and final assertion. Prefer existing observable waits such as `Browser.True` and `Browser.Equal` (which poll with `WebDriverWait`) or an explicit test-controlled gate. A bounded polling helper may delay between observable checks when it also enforces a timeout; a fixed `Thread.Sleep` is not the synchronization mechanism for the regression assertion. +When the browser reproduction is deterministic, record that the permanent C# Selenium scenario must preserve the same setup and final assertion. Prefer existing observable waits such as `Browser.True` and `Browser.Equal` (which poll with `WebDriverWait`) when the final observable is sufficient; use an explicit test-controlled gate only when deterministic managed or intermediate timing must be controlled. A bounded polling helper may delay between observable checks when it also enforces a timeout; a fixed `Thread.Sleep` is not the synchronization mechanism for the regression assertion. ## 7. Record the permanent regression boundary diff --git a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml index 6c6e1e7c67f5..a3695cc4d4b0 100644 --- a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml +++ b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml @@ -260,13 +260,13 @@ stimuli: pattern: '(?i)(producer|timing|ordering|precondition)' - type: output-matches config: - pattern: '(?i)(TaskCompletionSource|explicit (?:test )?gate|test-controlled gate|observable condition|faithful lower boundary)' + pattern: '(?i)(Browser\.True|Browser\.Equal|WebDriverWait|observable condition|faithful lower boundary|TaskCompletionSource|explicit (?:test )?gate|test-controlled gate)' - type: prompt rubric: - "Does not call the reproduction deterministic because not every run failed the same assertion for the same reason" - "Rejects Thread.Sleep(1000) as a synchronization mechanism and does not paper over the race" - "Investigates the real producer, preconditions, and timing with targeted observations or logging before finalizing permanent coverage" - - "Chooses a faithful lower-boundary test only if that boundary owns the final observable; otherwise introduces an explicit test-controlled gate and reruns the real browser scenario" + - "Chooses a faithful lower-boundary test only if that boundary owns the final observable; otherwise uses observable browser waits when the final observable is sufficient, or introduces an explicit test-controlled gate only when deterministic managed or intermediate timing must be controlled" - "Does not claim an immediate Selenium conversion is ready until the scenario is controlled and repeatable" - name: replace-arbitrary-selenium-sleep @@ -278,14 +278,11 @@ stimuli: - type: output-matches config: pattern: '(?i)(Browser\.True|Browser\.Equal|WebDriverWait|observable condition)' - - type: output-matches - config: - pattern: '(?i)(TaskCompletionSource|test-controlled gate|release (?:action|button|endpoint))' - type: prompt rubric: - "Rejects the arbitrary two-second sleep as synchronization for the regression assertion" - "Prefers Browser.True/Browser.Equal or another WebDriverWait-backed observable browser condition" - - "When managed timing must be controlled, recommends an explicit test gate such as TaskCompletionSource and a release action" + - "Requires an explicit test gate such as TaskCompletionSource and a release action only when deterministic managed or intermediate timing must be controlled; accepts Browser.True/Browser.Equal or another WebDriverWait-backed wait when the final DOM observable is sufficient" - "Does not impose a syntactic ban on every Thread.Sleep; allows bounded polling helpers that repeatedly observe a condition and enforce a timeout" - name: split-manager-contract-from-browser-reachability diff --git a/src/Components/AGENTS.md b/src/Components/AGENTS.md index 810cf7585c7f..e1b0a2680588 100644 --- a/src/Components/AGENTS.md +++ b/src/Components/AGENTS.md @@ -19,14 +19,14 @@ For implementation work: - For a behavioral fix, reproduce the problem at the faithful boundary before attempting the fix. If faithful validation is impractical, state the observed boundary and limitation and do not call the behavioral claim verified. - Research the problem area using the microsoft docs, existing code, git history, and logging on the sample project. - Implement the fix or feature in the sample project first. -- Reproduce the behavior interactively in the real browser using Playwright. -- Reduce the reproduction to the smallest temporary JavaScript diagnostic probe and assertion that observes the same failure in that page. Keep the probe outside the production change; it is not permanent Jest or `.test.ts` coverage. -- Repeat the unchanged browser scenario and probe a recorded, bounded number of times. Ten runs is a reasonable default confidence sample when repository evidence does not suggest another count, not proof across environments. Treat the issue as deterministic only when every run fails the same assertion for the same reason. +- Exercise a new feature, or reproduce a behavioral failure, interactively in the real browser using Playwright. +- For a browser regression or reproduced browser failure, reduce the reproduction to the smallest temporary JavaScript diagnostic probe and assertion that observes the same failure in that page. Keep the probe outside the production change; it is not permanent Jest or `.test.ts` coverage. +- For that regression, repeat the unchanged browser scenario and probe a recorded, bounded number of times. Ten runs is a reasonable default confidence sample when repository evidence does not suggest another count, not proof across environments. Treat the issue as deterministic only when every run fails the same assertion for the same reason. - If the result is not deterministic, investigate the owning producer and timing before selecting permanent coverage. Use a faithful lower-boundary test when that boundary owns the final observable, or add an explicit test-controlled gate when managed timing must be controlled. Do not stabilize a regression assertion with an arbitrary fixed sleep. - Once the fix or feature is validated in the sample, select its permanent test surface using the boundary below. - - For deterministic browser-owned behavior, translate the same scenario and diagnostic assertion into a C# Selenium E2E test. + - For browser-owned behavior, translate the validated browser scenario and final observable into a C# Selenium E2E test; for a regression, preserve the diagnostic assertion. - If an E2E test is failing, debug it by running the test server manually and navigating to the scenario in a browser. -- Only after the selected permanent tests are passing, remove the temporary JavaScript probe and the sample code you added in the Samples projects. +- Only after the selected permanent tests are passing, remove any temporary JavaScript probe and the sample code you added in the Samples projects. - Use `git checkout -- src/Components/Samples src/Components/WebAssembly/Samples` and `git clean -df -- src/Components/Samples src/Components/WebAssembly/Samples` to remove the sample code. ### Permanent regression test boundary