Skip to content

Commit 2b1fa19

Browse files
authored
[Security Solution][On-week] Flaky test doctor (elastic#250630)
## Summary Adds a comprehensive Cursor rule to guide AI-assisted analysis and fixing of flaky Cypress tests in Security Solution. This rule codifies team knowledge, best practices, and patterns learned from fixing real flaky tests. > 🧪 **On-Week Experiment:** This rule was created during an on-week as an experiment. If it proves helpful and sees wide adoption, the plan is to extend it to cover additional testing layers (API integration tests, unit tests) and Scout (Playwright) tests. ## Problem Flaky tests are a significant source of CI instability and developer friction. When investigating flaky tests, there's often: - No structured approach to diagnose root causes - Tests at the wrong layer of the testing pyramid (E2E tests that should be API/unit tests) - Inconsistent application of team conventions - Wasted effort fixing tests that should be deleted (duplicates) - Fixes that introduce new anti-patterns - New tests merged with patterns that will inevitably become flaky ## Solution This rule provides: ### Structured Analysis Framework - Step-by-step process: validity check → environment context → duplicate coverage → **layer analysis** → classification → fix - Mandatory checks before proposing any fix (functionality valid? duplicates exist?) - Information gathering strategy (what to self-investigate vs. ask the user) ### Layer Analysis (Testing Pyramid) The rule evaluates whether a Cypress test is at the appropriate layer. When duplicate coverage exists at API or unit test levels, the rule recommends deleting the Cypress test in favor of the faster, more reliable lower-layer tests. ### Proactive Flakiness Detection for New Tests The rule can identify patterns in new or existing tests that are likely to cause flakiness if you ask for it. This helps catch potential flakiness **before** tests are merged, not after they start failing in CI. ## How to Use ### Activation - **Automatic:** The rule activates when working with Cypress test files (`*.cy.ts`) - **Manual:** Say `@flaky_test_doctor` to explicitly activate the rule ### Interactive, Conversation-Guided Experience This is **not a one-shot tool**. The Flaky Test Doctor works with you through an interactive conversation: 1. **You provide initial context** → The AI analyzes the test and codebase 2. **AI asks clarifying questions** → About environment, error messages, screenshots 3. **Together you investigate** → AI searches code, you provide CI feedback 4. **AI proposes a fix** → You review, test locally, provide feedback 5. **Iterate until solved** → Refine the fix based on test results 6. **Verify with Flaky Test Runner** → Confirm the fix works in CI The AI will guide you through the diagnosis step by step, asking questions when it needs information it can't find in the codebase (like CI logs or screenshots), and explaining its reasoning along the way. ### For Best Results, Provide Context - Link to the GitHub issue for the flaky test - Which environment(s) the test is failing in (ESS, Serverless, MKI) - Error message and stack trace from the failure - Screenshots or videos showing the UI state at failure > 💡 **The more context you provide, the faster and more accurate the diagnosis.** The AI can search the codebase and git history, but it cannot access CI dashboards, build logs, or screenshots directly. ### For Existing Flaky Tests Ask about a flaky test, and the AI will: 1. Check if the functionality is still valid 2. Search for duplicate coverage 3. Evaluate if the test should be at a different layer (API/unit) 4. Classify the issue (bug vs. flakiness) 5. Propose a fix following team conventions 6. Recommend verification via Flaky Test Runner ### For New Tests The AI will: 1. Review the test for anti-patterns that cause flakiness 2. Suggest improvements before the test is merged 3. Recommend the appropriate testing layer 4. Ensure team conventions are followed Feedback and contributions welcome!
1 parent 2fe7e83 commit 2b1fa19

3 files changed

Lines changed: 1562 additions & 0 deletions

File tree

.github/CODEOWNERS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2564,6 +2564,7 @@ x-pack/solutions/security/test/security_solution_api_integration/test_suites/sou
25642564
## Security Solution sub teams - security-engineering-productivity
25652565
## NOTE: It's important to keep this above other teams' sections because test automation doesn't process
25662566
## the CODEOWNERS file correctly. See https://github.com/elastic/kibana/issues/173307#issuecomment-1855858929
2567+
/x-pack/solutions/security/test/security_solution_cypress/.cursor/rules/flaky_test_doctor.mdc @elastic/security-engineering-productivity
25672568
/x-pack/solutions/security/test/security_solution_cypress/* @elastic/security-engineering-productivity
25682569
/x-pack/solutions/security/test/security_solution_cypress/cypress/* @elastic/security-engineering-productivity
25692570
/x-pack/solutions/security/test/security_solution_cypress/cypress/tasks/login.ts @elastic/security-engineering-productivity

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ target
2121
.idea
2222
.claude
2323
.cursor
24+
!x-pack/solutions/security/test/security_solution_cypress/.cursor/
2425
.windsurf
2526
claude.md
2627
*.iml

0 commit comments

Comments
 (0)