Skip to content

Commit a378893

Browse files
abretonc7sabretonc7s
committed
chore(perps): merge origin/main into perps analytics branch
Sync 37 commits from main. Resolve conflicts where main migrated the button-color A/B test to the canonical useABTest standard (abTestConfig.ts + enrichWithABTests registry) that superseded this branch's bespoke selector-based button-color wiring: - featureFlags/index.ts: drop obsolete selectPerpsButtonColorTestVariant / VALID_BUTTON_COLOR_VARIANTS (referenced deleted buttonColors.ts); keep main's canonical config. - PerpsMarketDetailsView / PerpsOrderView: drop manual ab_test_button_color event injection; the assignment is now auto-attached to PERPS_SCREEN_VIEWED and PERPS_UI_INTERACTION via enrichWithABTests(). Keep this branch's comprehensive Place Order press tracking and controller-owned latency note; avoid double-emitting the button-press interaction event. - utils/abTesting/*: accept main's deletion of the legacy framework. - docs/perps/*: take main's canonical A/B testing documentation. Preserve all perps analytics work (discovery/search funnel, abandon tracking, entry-CTA source, tradeAction, best-effort UTM).
2 parents 892b4f0 + b8d3e80 commit a378893

239 files changed

Lines changed: 8506 additions & 3705 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# `flaky-unit-test-analysis` mode
2+
3+
Custom [`MetaMask/ai-analyzer`](https://github.com/MetaMask/ai-analyzer) mode that reviews **only the modified Jest unit test files** in a PR for known flaky-test patterns (J1–J10 from the [`flaky-test-detection`](https://github.com/MetaMask/skills/blob/main/domains/coding/skills/flaky-test-detection/skill.md) skill) and emits structured findings with educational fix suggestions.
4+
5+
Consumed by [`.github/workflows/flaky-unit-test-detection.yml`](../../../.github/workflows/flaky-unit-test-detection.yml). Not a shipped built-in — this is a Tier 3 fully-custom mode defined entirely in this repo.
6+
7+
## Files
8+
9+
| File | Purpose |
10+
| ---------------------- | -------------------------------------------------------------------------------- |
11+
| `mode.yaml` | Mode identity — `id`, `finalizeToolName`, `outputFile` |
12+
| `system-prompt.md` | AI role, tool-use guidance, and pattern list (uses `{{template_vars}}`) |
13+
| `task-prompt.md` | Per-run instructions, receives `{{changed_files}}` |
14+
| `finalize-schema.json` | JSON Schema the AI must satisfy when calling `finalize_flaky_unit_test_analysis` |
15+
| `fallback.json` | Deterministic `conservative` / `empty` results when the AI can't complete |
16+
17+
The J1–J10 pattern reference is **not** duplicated here. It is loaded on demand via the analyzer's `load_skill` tool, from a copy of `mms-flaky-test-detection` synced by `yarn skills` (CI does this with the cloud-agent bootstrap; see workflow). This keeps a single source of truth in `MetaMask/skills` and prevents content drift.
18+
19+
## Output artifact
20+
21+
Written to `.ai-pr-analyzer/flaky-ai-analysis.json` (relative to the workspace, per `outputFile` in `mode.yaml`). Consumed by `.github/scripts/flaky-sticky-comment.ts`.
22+
23+
Shape (matches `finalize-schema.json`):
24+
25+
```json
26+
{
27+
"analyzedFiles": ["app/components/Views/Foo/Foo.test.tsx"],
28+
"findings": [
29+
{
30+
"file": "app/components/Views/Foo/Foo.test.tsx",
31+
"line": 42,
32+
"patternId": "J1",
33+
"patternName": "Missing act() on async state update",
34+
"severity": "critical",
35+
"snippet": "refreshControl.props.onRefresh();",
36+
"explanation": "Async prop callback triggers a state update outside act(), causing an intermittent race.",
37+
"suggestedFix": "await act(async () => { await refreshControl.props.onRefresh(); });",
38+
"historicalHintUsed": true
39+
}
40+
],
41+
"confidence": 78,
42+
"reasoning": "1 of 1 analyzed file contains a J1 pattern matching its historical failure signature."
43+
}
44+
```
45+
46+
`findings: []` with `analyzedFiles` populated means "reviewed, nothing found" — Stage 3 (sticky comment) uses that distinction for its four-state logic (create / update / no-op / all-clear).
47+
48+
## Editing the mode
49+
50+
- **Prompt tweaks**: edit `system-prompt.md` / `task-prompt.md`. The analyzer supports `{{prompt_context}}`, `{{changed_files}}`, `{{tools_section}}`, `{{skills_section}}`, `{{max_iterations}}`, `{{finalize_tool_name}}` and more — see [MetaMask/ai-analyzer docs/adding-a-new-mode.md](https://github.com/MetaMask/ai-analyzer/blob/v1/docs/adding-a-new-mode.md).
51+
- **Schema changes**: keep `finalize-schema.json` and `fallback.json` in sync (both must satisfy the same shape). Any downstream consumer (the sticky-comment script) must be updated too.
52+
- **Pattern reference changes**: do **not** edit `.ai-pr-analyzer/skills/mms-flaky-test-detection.md` — it is generated. Edit the source at [`MetaMask/skills/domains/coding/skills/flaky-test-detection/skill.md`](https://github.com/MetaMask/skills/blob/main/domains/coding/skills/flaky-test-detection/skill.md) and re-run `yarn skills`.

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/fallback.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"conservative": {
3+
"analyzedFiles": [],
4+
"findings": [],
5+
"confidence": 0,
6+
"reasoning": "AI analysis did not complete. Historical signal (if any) is still reported by the deterministic history check."
7+
},
8+
"empty": {
9+
"analyzedFiles": [],
10+
"findings": [],
11+
"confidence": 100,
12+
"reasoning": "No modified unit test files to analyze."
13+
}
14+
}

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/finalize-schema.json‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
{
2+
"type": "object",
3+
"properties": {
4+
"analyzedFiles": {
5+
"type": "array",
6+
"items": { "type": "string" },
7+
"description": "All modified unit test files that were reviewed"
8+
},
9+
"findings": {
10+
"type": "array",
11+
"items": {
12+
"type": "object",
13+
"properties": {
14+
"file": { "type": "string" },
15+
"line": { "type": "number" },
16+
"patternId": {
17+
"type": "string",
18+
"enum": [
19+
"J1",
20+
"J2",
21+
"J3",
22+
"J4",
23+
"J5",
24+
"J6",
25+
"J7",
26+
"J8",
27+
"J9",
28+
"J10"
29+
]
30+
},
31+
"patternName": { "type": "string" },
32+
"severity": {
33+
"type": "string",
34+
"enum": ["critical", "high", "medium"]
35+
},
36+
"snippet": {
37+
"type": "string",
38+
"description": "The exact current code being replaced, covering the same lines/scope as `suggestedFix`, formatted as real TypeScript with line breaks (use \\n between lines and preserve indentation). Must line up with `suggestedFix` so the two can be rendered as a diff (removed vs added lines) — do not paraphrase or summarize it."
39+
},
40+
"explanation": { "type": "string" },
41+
"suggestedFix": {
42+
"type": "string",
43+
"description": "The corrected code snippet ONLY, formatted as real TypeScript with line breaks (use \\n between lines and preserve indentation). Do NOT write prose sentences here — keep all explanation in the `explanation` field."
44+
},
45+
"historicalHintUsed": { "type": "boolean" }
46+
},
47+
"required": [
48+
"file",
49+
"patternId",
50+
"patternName",
51+
"severity",
52+
"snippet",
53+
"explanation",
54+
"suggestedFix",
55+
"historicalHintUsed"
56+
]
57+
}
58+
},
59+
"confidence": {
60+
"type": "number",
61+
"description": "Confidence in the assessment (0-100)"
62+
},
63+
"reasoning": {
64+
"type": "string",
65+
"description": "Detailed reasoning: what was investigated and why each finding was flagged"
66+
}
67+
},
68+
"required": ["analyzedFiles", "findings", "confidence", "reasoning"]
69+
}

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/mode.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
id: flaky-unit-test-analysis
2+
description: Detects flaky Jest patterns in modified unit test files and suggests fixes
3+
finalizeToolName: finalize_flaky_unit_test_analysis
4+
outputFile: .ai-pr-analyzer/flaky-ai-analysis.json

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/system-prompt.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
You are a flaky-Jest-test detector. You analyze modified unit test files for patterns known to cause intermittent CI failures — not for general code quality or style.
2+
3+
{{prompt_context}}
4+
5+
GOAL: For each modified test file, identify concrete flaky-test patterns and produce an educational, actionable fix suggestion for each one found.
6+
7+
{{reasoning_section}}
8+
9+
{{tools_section}}
10+
11+
{{skills_section}}
12+
13+
Before analyzing any file, call load_skill with skill_name "mms-flaky-test-detection" to load the full pattern reference (J1-J10) — always do this first, in your first tool-call batch.
14+
15+
HISTORICAL CONTEXT:
16+
Read .ai-pr-analyzer/flaky-history.json with read_file if present. Treat entries with "flaky": true as a HINT to inspect that file more carefully — never as a finding by itself. A file can have findings with no historical signal, and a file with a historical failure rate can have zero pattern findings (the failure may be environmental, not a code pattern).
17+
18+
PATTERNS TO DETECT (see loaded skill for full detail and fix examples):
19+
20+
- Missing act() around async state updates
21+
- Real timers where fake timers are needed
22+
- Missing jest.clearAllMocks()/resetAllMocks() between tests
23+
- waitFor() without a real assertion inside, or with an async callback
24+
- Incomplete mock store state
25+
- Arbitrary setTimeout/sleep used as a synchronization barrier
26+
- Non-deterministic data: Date.now(), Math.random(), unstubbed network
27+
- jest.useFakeTimers() combined with waitFor() (polling conflict)
28+
- Module-level mutable let bindings not reset in beforeEach
29+
- jest.spyOn() without restoreAllMocks()/mockRestore() afterward
30+
31+
Do not invent findings — only report a pattern match you can point to with a concrete line and snippet from the file. If a file has no matches, omit it from findings rather than forcing one.
32+
33+
Do not exceed {{max_iterations}} analysis iterations.

‎.ai-pr-analyzer/modes/flaky-unit-test-analysis/task-prompt.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
Analyze ONLY the modified unit test files listed below for flaky-Jest patterns. Do not analyze production code files even if referenced by a test.
2+
3+
{{changed_files}}
4+
5+
FILE ACCESS RULES (strict):
6+
7+
- The ONLY files you may read with read_file are:
8+
1. The modified unit test files listed above (they end in `.test.ts`, `.test.tsx`, `.spec.ts`, or `.spec.tsx`).
9+
2. `.ai-pr-analyzer/flaky-history.json` — historical failure hint, deliberately provided.
10+
- Do NOT read production code (any `.ts`/`.tsx` file without `.test.` or `.spec.` in the name), CI config, workflow files, mode configuration, or anything under `.github/`, `.agents/`, `node_modules/`, or `.ai-pr-analyzer/` (except the flaky-history JSON above). These describe or govern this workflow itself and would bias the analysis.
11+
- Do NOT use grep_codebase, find_related_files, list_directory, or get_git_diff to explore files outside the allowlist. J1-J10 patterns are properties of the test file itself — additional context is never needed.
12+
- If reaching a conclusion would require inspecting a file outside the allowlist, do NOT force a finding: omit it. Every finding you report MUST reference a file from the modified list above.
13+
14+
For each file:
15+
16+
1. Call load_skill("mms-flaky-test-detection") once, before reading files, if not already loaded.
17+
2. Read the file with read_file.
18+
3. Check .ai-pr-analyzer/flaky-history.json (read_file) for a historical hint on this file.
19+
4. Match against the J1-J10 patterns from the loaded skill.
20+
5. For every match, record: file, line, patternId, patternName, severity, snippet, explanation, suggestedFix, and whether the historical hint was used.
21+
- `snippet` MUST be the exact current code being replaced, copied verbatim from the file (no paraphrasing or summarizing), covering the same lines/scope as `suggestedFix` so the two can be rendered as a before/after diff.
22+
- `suggestedFix` MUST be the corrected code snippet ONLY, formatted as real TypeScript with actual line breaks (`\n`) and indentation — never a single-line prose paragraph. Keep all reasoning and instructions in `explanation`.
23+
24+
If a file has no matches, do not invent findings — omit it from findings.
25+
26+
INVESTIGATION STRATEGY:
27+
28+
- Batch independent tool calls (e.g. reading multiple files, or a file plus the history JSON) in a single response.
29+
- Only call get_git_diff if you need to distinguish newly-added test code from pre-existing code in a large file — and only against files in the modified list above.
30+
31+
Call {{finalize_tool_name}} with your complete result once all files are reviewed.

0 commit comments

Comments
 (0)