Skip to content

[P2] Add Reproducible Rule Validation Harness #19

Description

@barvhaim

Roadmap Item 13: Add Reproducible Rule Validation Harness

Priority: P2
Area: Detection engineering
References: src/threatxtension/config/custom_semgrep_rules.yaml,
scripts/analyze_extension_patterns.py

Custom Semgrep rules are central to the project. They need a repeatable way to validate
true positives, false positives, and expected MITRE/CWE metadata.

Scope:

  • Add rule fixtures with positive and negative JavaScript examples.
  • Add tests that run Semgrep against fixture files.
  • Validate rule metadata, severity, CWE, MITRE, and category fields.
  • Document how to add a new rule safely.

Acceptance criteria:

  • CI fails when custom rule syntax breaks or a fixture no longer triggers.
  • Each high-value rule has at least one positive and one negative fixture.
  • Rule authoring docs exist under docs/.

Created from docs/roadmap.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    detection-engineeringDetection rules, validation, and tuningenhancementNew feature or requestsastStatic analysis and Semgrep detection worktestingAutomated tests and fixtures

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions