Commit 2b1fa19
authored
[Security Solution][On-week] Flaky test doctor (elastic#250630)
## Summary
Adds a comprehensive Cursor rule to guide AI-assisted analysis and
fixing of flaky Cypress tests in Security Solution. This rule codifies
team knowledge, best practices, and patterns learned from fixing real
flaky tests.
> 🧪 **On-Week Experiment:** This rule was created during an on-week as
an experiment. If it proves helpful and sees wide adoption, the plan is
to extend it to cover additional testing layers (API integration tests,
unit tests) and Scout (Playwright) tests.
## Problem
Flaky tests are a significant source of CI instability and developer
friction. When investigating flaky tests, there's often:
- No structured approach to diagnose root causes
- Tests at the wrong layer of the testing pyramid (E2E tests that should
be API/unit tests)
- Inconsistent application of team conventions
- Wasted effort fixing tests that should be deleted (duplicates)
- Fixes that introduce new anti-patterns
- New tests merged with patterns that will inevitably become flaky
## Solution
This rule provides:
### Structured Analysis Framework
- Step-by-step process: validity check → environment context → duplicate
coverage → **layer analysis** → classification → fix
- Mandatory checks before proposing any fix (functionality valid?
duplicates exist?)
- Information gathering strategy (what to self-investigate vs. ask the
user)
### Layer Analysis (Testing Pyramid)
The rule evaluates whether a Cypress test is at the appropriate layer.
When duplicate coverage exists at API or unit test levels, the rule
recommends deleting the Cypress test in favor of the faster, more
reliable lower-layer tests.
### Proactive Flakiness Detection for New Tests
The rule can identify patterns in new or existing tests that are likely
to cause flakiness if you ask for it.
This helps catch potential flakiness **before** tests are merged, not
after they start failing in CI.
## How to Use
### Activation
- **Automatic:** The rule activates when working with Cypress test files
(`*.cy.ts`)
- **Manual:** Say `@flaky_test_doctor` to explicitly activate the rule
### Interactive, Conversation-Guided Experience
This is **not a one-shot tool**. The Flaky Test Doctor works with you
through an interactive conversation:
1. **You provide initial context** → The AI analyzes the test and
codebase
2. **AI asks clarifying questions** → About environment, error messages,
screenshots
3. **Together you investigate** → AI searches code, you provide CI
feedback
4. **AI proposes a fix** → You review, test locally, provide feedback
5. **Iterate until solved** → Refine the fix based on test results
6. **Verify with Flaky Test Runner** → Confirm the fix works in CI
The AI will guide you through the diagnosis step by step, asking
questions when it needs information it can't find in the codebase (like
CI logs or screenshots), and explaining its reasoning along the way.
### For Best Results, Provide Context
- Link to the GitHub issue for the flaky test
- Which environment(s) the test is failing in (ESS, Serverless, MKI)
- Error message and stack trace from the failure
- Screenshots or videos showing the UI state at failure
> 💡 **The more context you provide, the faster and more accurate the
diagnosis.** The AI can search the codebase and git history, but it
cannot access CI dashboards, build logs, or screenshots directly.
### For Existing Flaky Tests
Ask about a flaky test, and the AI will:
1. Check if the functionality is still valid
2. Search for duplicate coverage
3. Evaluate if the test should be at a different layer (API/unit)
4. Classify the issue (bug vs. flakiness)
5. Propose a fix following team conventions
6. Recommend verification via Flaky Test Runner
### For New Tests
The AI will:
1. Review the test for anti-patterns that cause flakiness
2. Suggest improvements before the test is merged
3. Recommend the appropriate testing layer
4. Ensure team conventions are followed
Feedback and contributions welcome!1 parent 2fe7e83 commit 2b1fa19
3 files changed
Lines changed: 1562 additions & 0 deletions
File tree
- .github
- x-pack/solutions/security/test/security_solution_cypress/.cursor/rules
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2564 | 2564 | | |
2565 | 2565 | | |
2566 | 2566 | | |
| 2567 | + | |
2567 | 2568 | | |
2568 | 2569 | | |
2569 | 2570 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
0 commit comments