Merge branch 'main' into toast/animate-ds-legacy

Author: amandaye0h

.ai-pr-analyzer/modes/flaky-unit-test-analysis/README.md

@@ -0,0 +1,52 @@
+# `flaky-unit-test-analysis` mode
+
+Custom [`MetaMask/ai-analyzer`](https://github.com/MetaMask/ai-analyzer) mode that reviews **only the modified Jest unit test files** in a PR for known flaky-test patterns (J1–J10 from the [`flaky-test-detection`](https://github.com/MetaMask/skills/blob/main/domains/coding/skills/flaky-test-detection/skill.md) skill) and emits structured findings with educational fix suggestions.
+
+Consumed by [`.github/workflows/flaky-unit-test-detection.yml`](../../../.github/workflows/flaky-unit-test-detection.yml). Not a shipped built-in — this is a Tier 3 fully-custom mode defined entirely in this repo.
+
+## Files
+
+| File                   | Purpose                                                                          |
+| ---------------------- | -------------------------------------------------------------------------------- |
+| `mode.yaml`            | Mode identity — `id`, `finalizeToolName`, `outputFile`                           |
+| `system-prompt.md`     | AI role, tool-use guidance, and pattern list (uses `{{template_vars}}`)          |
+| `task-prompt.md`       | Per-run instructions, receives `{{changed_files}}`                               |
+| `finalize-schema.json` | JSON Schema the AI must satisfy when calling `finalize_flaky_unit_test_analysis` |
+| `fallback.json`        | Deterministic `conservative` / `empty` results when the AI can't complete        |
+
+The J1–J10 pattern reference is **not** duplicated here. It is loaded on demand via the analyzer's `load_skill` tool, from a copy of `mms-flaky-test-detection` synced by `yarn skills` (CI does this with the cloud-agent bootstrap; see workflow). This keeps a single source of truth in `MetaMask/skills` and prevents content drift.
+
+## Output artifact
+
+Written to `.ai-pr-analyzer/flaky-ai-analysis.json` (relative to the workspace, per `outputFile` in `mode.yaml`). Consumed by `.github/scripts/flaky-sticky-comment.ts`.
+
+Shape (matches `finalize-schema.json`):
+
+```json
+{
+  "analyzedFiles": ["app/components/Views/Foo/Foo.test.tsx"],
+  "findings": [
+    {
+      "file": "app/components/Views/Foo/Foo.test.tsx",
+      "line": 42,
+      "patternId": "J1",
+      "patternName": "Missing act() on async state update",
+      "severity": "critical",
+      "snippet": "refreshControl.props.onRefresh();",
+      "explanation": "Async prop callback triggers a state update outside act(), causing an intermittent race.",
+      "suggestedFix": "await act(async () => { await refreshControl.props.onRefresh(); });",
+      "historicalHintUsed": true
+    }
+  ],
+  "confidence": 78,
+  "reasoning": "1 of 1 analyzed file contains a J1 pattern matching its historical failure signature."
+}
+```
+
+`findings: []` with `analyzedFiles` populated means "reviewed, nothing found" — Stage 3 (sticky comment) uses that distinction for its four-state logic (create / update / no-op / all-clear).
+
+## Editing the mode
+
+- **Prompt tweaks**: edit `system-prompt.md` / `task-prompt.md`. The analyzer supports `{{prompt_context}}`, `{{changed_files}}`, `{{tools_section}}`, `{{skills_section}}`, `{{max_iterations}}`, `{{finalize_tool_name}}` and more — see [MetaMask/ai-analyzer docs/adding-a-new-mode.md](https://github.com/MetaMask/ai-analyzer/blob/v1/docs/adding-a-new-mode.md).
+- **Schema changes**: keep `finalize-schema.json` and `fallback.json` in sync (both must satisfy the same shape). Any downstream consumer (the sticky-comment script) must be updated too.
+- **Pattern reference changes**: do **not** edit `.ai-pr-analyzer/skills/mms-flaky-test-detection.md` — it is generated. Edit the source at [`MetaMask/skills/domains/coding/skills/flaky-test-detection/skill.md`](https://github.com/MetaMask/skills/blob/main/domains/coding/skills/flaky-test-detection/skill.md) and re-run `yarn skills`.

.ai-pr-analyzer/modes/flaky-unit-test-analysis/fallback.json

@@ -0,0 +1,14 @@
+{
+  "conservative": {
+    "analyzedFiles": [],
+    "findings": [],
+    "confidence": 0,
+    "reasoning": "AI analysis did not complete. Historical signal (if any) is still reported by the deterministic history check."
+  },
+  "empty": {
+    "analyzedFiles": [],
+    "findings": [],
+    "confidence": 100,
+    "reasoning": "No modified unit test files to analyze."
+  }
+}

.ai-pr-analyzer/modes/flaky-unit-test-analysis/finalize-schema.json

@@ -0,0 +1,69 @@
+{
+  "type": "object",
+  "properties": {
+    "analyzedFiles": {
+      "type": "array",
+      "items": { "type": "string" },
+      "description": "All modified unit test files that were reviewed"
+    },
+    "findings": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "properties": {
+          "file": { "type": "string" },
+          "line": { "type": "number" },
+          "patternId": {
+            "type": "string",
+            "enum": [
+              "J1",
+              "J2",
+              "J3",
+              "J4",
+              "J5",
+              "J6",
+              "J7",
+              "J8",
+              "J9",
+              "J10"
+            ]
+          },
+          "patternName": { "type": "string" },
+          "severity": {
+            "type": "string",
+            "enum": ["critical", "high", "medium"]
+          },
+          "snippet": {
+            "type": "string",
+            "description": "The exact current code being replaced, covering the same lines/scope as `suggestedFix`, formatted as real TypeScript with line breaks (use \\n between lines and preserve indentation). Must line up with `suggestedFix` so the two can be rendered as a diff (removed vs added lines) — do not paraphrase or summarize it."
+          },
+          "explanation": { "type": "string" },
+          "suggestedFix": {
+            "type": "string",
+            "description": "The corrected code snippet ONLY, formatted as real TypeScript with line breaks (use \\n between lines and preserve indentation). Do NOT write prose sentences here — keep all explanation in the `explanation` field."
+          },
+          "historicalHintUsed": { "type": "boolean" }
+        },
+        "required": [
+          "file",
+          "patternId",
+          "patternName",
+          "severity",
+          "snippet",
+          "explanation",
+          "suggestedFix",
+          "historicalHintUsed"
+        ]
+      }
+    },
+    "confidence": {
+      "type": "number",
+      "description": "Confidence in the assessment (0-100)"
+    },
+    "reasoning": {
+      "type": "string",
+      "description": "Detailed reasoning: what was investigated and why each finding was flagged"
+    }
+  },
+  "required": ["analyzedFiles", "findings", "confidence", "reasoning"]
+}

.ai-pr-analyzer/modes/flaky-unit-test-analysis/mode.yaml

@@ -0,0 +1,4 @@
+id: flaky-unit-test-analysis
+description: Detects flaky Jest patterns in modified unit test files and suggests fixes
+finalizeToolName: finalize_flaky_unit_test_analysis
+outputFile: .ai-pr-analyzer/flaky-ai-analysis.json

.ai-pr-analyzer/modes/flaky-unit-test-analysis/system-prompt.md

@@ -0,0 +1,33 @@
+You are a flaky-Jest-test detector. You analyze modified unit test files for patterns known to cause intermittent CI failures — not for general code quality or style.
+
+{{prompt_context}}
+
+GOAL: For each modified test file, identify concrete flaky-test patterns and produce an educational, actionable fix suggestion for each one found.
+
+{{reasoning_section}}
+
+{{tools_section}}
+
+{{skills_section}}
+
+Before analyzing any file, call load_skill with skill_name "mms-flaky-test-detection" to load the full pattern reference (J1-J10) — always do this first, in your first tool-call batch.
+
+HISTORICAL CONTEXT:
+Read .ai-pr-analyzer/flaky-history.json with read_file if present. Treat entries with "flaky": true as a HINT to inspect that file more carefully — never as a finding by itself. A file can have findings with no historical signal, and a file with a historical failure rate can have zero pattern findings (the failure may be environmental, not a code pattern).
+
+PATTERNS TO DETECT (see loaded skill for full detail and fix examples):
+
+- Missing act() around async state updates
+- Real timers where fake timers are needed
+- Missing jest.clearAllMocks()/resetAllMocks() between tests
+- waitFor() without a real assertion inside, or with an async callback
+- Incomplete mock store state
+- Arbitrary setTimeout/sleep used as a synchronization barrier
+- Non-deterministic data: Date.now(), Math.random(), unstubbed network
+- jest.useFakeTimers() combined with waitFor() (polling conflict)
+- Module-level mutable let bindings not reset in beforeEach
+- jest.spyOn() without restoreAllMocks()/mockRestore() afterward
+
+Do not invent findings — only report a pattern match you can point to with a concrete line and snippet from the file. If a file has no matches, omit it from findings rather than forcing one.
+
+Do not exceed {{max_iterations}} analysis iterations.

.ai-pr-analyzer/modes/flaky-unit-test-analysis/task-prompt.md

@@ -0,0 +1,31 @@
+Analyze ONLY the modified unit test files listed below for flaky-Jest patterns. Do not analyze production code files even if referenced by a test.
+
+{{changed_files}}
+
+FILE ACCESS RULES (strict):
+
+- The ONLY files you may read with read_file are:
+  1. The modified unit test files listed above (they end in `.test.ts`, `.test.tsx`, `.spec.ts`, or `.spec.tsx`).
+  2. `.ai-pr-analyzer/flaky-history.json` — historical failure hint, deliberately provided.
+- Do NOT read production code (any `.ts`/`.tsx` file without `.test.` or `.spec.` in the name), CI config, workflow files, mode configuration, or anything under `.github/`, `.agents/`, `node_modules/`, or `.ai-pr-analyzer/` (except the flaky-history JSON above). These describe or govern this workflow itself and would bias the analysis.
+- Do NOT use grep_codebase, find_related_files, list_directory, or get_git_diff to explore files outside the allowlist. J1-J10 patterns are properties of the test file itself — additional context is never needed.
+- If reaching a conclusion would require inspecting a file outside the allowlist, do NOT force a finding: omit it. Every finding you report MUST reference a file from the modified list above.
+
+For each file:
+
+1. Call load_skill("mms-flaky-test-detection") once, before reading files, if not already loaded.
+2. Read the file with read_file.
+3. Check .ai-pr-analyzer/flaky-history.json (read_file) for a historical hint on this file.
+4. Match against the J1-J10 patterns from the loaded skill.
+5. For every match, record: file, line, patternId, patternName, severity, snippet, explanation, suggestedFix, and whether the historical hint was used.
+   - `snippet` MUST be the exact current code being replaced, copied verbatim from the file (no paraphrasing or summarizing), covering the same lines/scope as `suggestedFix` so the two can be rendered as a before/after diff.
+   - `suggestedFix` MUST be the corrected code snippet ONLY, formatted as real TypeScript with actual line breaks (`\n`) and indentation — never a single-line prose paragraph. Keep all reasoning and instructions in `explanation`.
+
+If a file has no matches, do not invent findings — omit it from findings.
+
+INVESTIGATION STRATEGY:
+
+- Batch independent tool calls (e.g. reading multiple files, or a file plus the history JSON) in a single response.
+- Only call get_git_diff if you need to distinguish newly-added test code from pre-existing code in a large file — and only against files in the modified list above.
+
+Call {{finalize_tool_name}} with your complete result once all files are reviewed.