feat(eval): add evaluation harness for rule efficacy measurement (#158)

* feat(eval): add evaluation harness module for rule efficacy testing

Add core evaluation types and logic:
- EvalCase, EvalResult for test case representation
- RuleMetrics with precision/recall/F1 calculation
- EvalSummary aggregating results across cases
- EvalManifest for YAML manifest parsing
- evaluate_case() and evaluate_manifest() functions
- Output formatters: JSON, CSV, Markdown

This enables measuring validation rule effectiveness against labeled
test cases, supporting data-driven rule development.

Part of #111

* feat(cli): add 'agnix eval' subcommand for evaluation harness

Add CLI subcommand to run evaluation manifests:
- --format option: markdown (default), json, csv
- --filter option: filter cases by rule prefix
- --verbose option: show per-case pass/fail details
- Exit code 1 on any failing cases

Usage:
  agnix eval tests/eval/eval.yaml
  agnix eval tests/eval/eval.yaml --format json
  agnix eval tests/eval/eval.yaml --filter "MCP-" --verbose

Part of #111

* test(eval): add initial evaluation dataset with 39 test cases

Create evaluation manifest covering all rule families:
- AS-*, CC-SK-*: Skill validation rules (13 cases)
- MCP-*: MCP tool validation rules (8 cases)
- AGM-*: AGENTS.md rules (6 cases)
- XP-*: Cross-platform rules (5 cases)
- XML-*: XML validation rules (4 cases)
- REF-*: Reference/import rules (3 cases)

All 39 cases pass with 100% precision/recall.

Part of #111

* docs(eval): add evaluation harness documentation

Document the evaluation harness feature:
- Quick start examples
- Manifest format specification
- Metrics explanation (TP, FP, FN, precision, recall, F1)
- Output format examples (markdown, JSON, CSV)
- Usage guide for regression testing and rule development

Part of #111

* fix: add path traversal protection and comprehensive tests

- Add validate_path_within_base() to prevent path traversal attacks
- Add EvalError::PathTraversal variant for security errors
- Add tests for error paths (file not found, invalid YAML)
- Add tests for filter logic with various scenarios
- Add tests for evaluate_manifest_file entry point
- Add tests for path traversal detection
- Add tests for Display trait and empty results edge case

Security: Validates that manifest file paths stay within base directory

* fix(security): harden path traversal protection

- Reject absolute paths explicitly (only relative paths allowed)
- Reject paths containing .. segments before joining (defense-in-depth)
- Fail validation when canonicalization fails (no fallback to non-canonical)
- Add explicit test for absolute path rejection
- Strengthen test assertions to explicitly verify path-traversal detection

Security: Addresses edge cases with non-existent malicious paths and
platform-specific path behaviors.

* fix(security): allow relative paths with parent refs within repo

The previous security model was too strict, rejecting any path with ..
segments. This made the evaluation harness unusable for manifests in
subdirectories that reference sibling directories (e.g., tests/eval/
referencing ../fixtures/).

New security model:
- Reject absolute paths (prevents /etc/passwd access)
- Allow relative paths with .. (enables ../fixtures patterns)
- Require file to exist (canonicalization verifies)

This is still secure because:
1. Absolute paths are blocked
2. Non-existent files fail validation
3. Symlinks are resolved to real targets

Add test for relative paths with parent directory references.

* docs: add eval command to README and CHANGELOG

- Add agnix eval example to Quick Start section
- Add comprehensive changelog entry for evaluation harness feature

* fix: address review feedback - path traversal and filter logic

Security (Critical):
- Add canonical base directory check to prevent path traversal
- Return canonical path to prevent TOCTOU vulnerabilities
- Move eval.yaml to tests/ to avoid ../fixtures paths

Filter logic (High):
- Include cases with empty expected to detect FPs on clean files
- Use starts_with for filter matching (per CLI help text)

Tests:
- Update path traversal tests for stricter security
- Add test_relative_path_within_base_dir_allowed
- Update test_relative_path_escaping_base_dir_blocked

Docs:
- Update paths from tests/eval/eval.yaml to tests/eval.yaml

Author: avifenesh

CHANGELOG.md

@@ -71,6 +71,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Documented SHA pin reference in .github/workflows/README.md for maintainability
 
 ### Added
+- Evaluation harness with `agnix eval` command for measuring rule efficacy
+  - Load test cases from YAML manifests with expected rule IDs
+  - Calculate precision, recall, and F1 scores per rule and overall
+  - Output formats: markdown (default), JSON, CSV
+  - Filter by rule prefix (`--filter`)
+  - Verbose mode for per-case details (`--verbose`)
+  - 39 test cases covering AS-*, CC-SK-*, MCP-*, AGM-*, XP-*, XML-*, REF-* rules
+  - Path traversal protection (relative paths only)
+  - Documentation in knowledge-base/EVALUATION.md
 - MCP-008 rule for protocol version validation with configurable `mcp_protocol_version` option
 - 5 new parse error rules with normalized IDs (AS-016, CC-HK-012, CC-AG-007, CC-PL-006, MCP-007)
 - Auto-fix support for CC-MEM-005 and CC-MEM-007 memory rules

README.md

@@ -75,6 +75,9 @@ agnix --format sarif .
 
 # Generate config file
 agnix init
+
+# Evaluate rule efficacy
+agnix eval tests/eval.yaml
 ```
 
 ## Output

crates/agnix-cli/src/main.rs

@@ -7,6 +7,7 @@ use agnix_core::{
     apply_fixes,
     config::{LintConfig, TargetTool},
     diagnostics::DiagnosticLevel,
+    eval::{evaluate_manifest_file, EvalFormat},
     validate_project, ValidationResult,
 };
 use clap::{Parser, Subcommand, ValueEnum};
@@ -95,6 +96,25 @@ struct Cli {
     format: OutputFormat,
 }
 
+/// Output format for evaluation results
+#[derive(Debug, Clone, Copy, Default, ValueEnum)]
+pub enum EvalOutputFormat {
+    #[default]
+    Markdown,
+    Json,
+    Csv,
+}
+
+impl From<EvalOutputFormat> for EvalFormat {
+    fn from(f: EvalOutputFormat) -> Self {
+        match f {
+            EvalOutputFormat::Markdown => EvalFormat::Markdown,
+            EvalOutputFormat::Json => EvalFormat::Json,
+            EvalOutputFormat::Csv => EvalFormat::Csv,
+        }
+    }
+}
+
 #[derive(Subcommand)]
 enum Commands {
     /// Validate agent configs
@@ -110,6 +130,24 @@ enum Commands {
         #[arg(default_value = ".agnix.toml")]
         output: PathBuf,
     },
+
+    /// Evaluate rule efficacy against labeled test cases
+    Eval {
+        /// Path to evaluation manifest (YAML file)
+        path: PathBuf,
+
+        /// Output format (markdown, json, csv)
+        #[arg(long, short, value_enum, default_value_t = EvalOutputFormat::Markdown)]
+        format: EvalOutputFormat,
+
+        /// Filter to specific rule prefix (e.g., "AS-", "MCP-")
+        #[arg(long)]
+        filter: Option<String>,
+
+        /// Show detailed results for each case
+        #[arg(long, short)]
+        verbose: bool,
+    },
 }
 
 fn main() {
@@ -118,6 +156,12 @@ fn main() {
     let result = match &cli.command {
         Some(Commands::Validate { path }) => validate_command(path, &cli),
         Some(Commands::Init { output }) => init_command(output),
+        Some(Commands::Eval {
+            path,
+            format,
+            filter,
+            verbose,
+        }) => eval_command(path, *format, filter.as_deref(), *verbose),
         None => validate_command(&cli.path, &cli),
     };
 
@@ -373,3 +417,94 @@ fn init_command(output: &PathBuf) -> anyhow::Result<()> {
 
     Ok(())
 }
+
+fn eval_command(
+    path: &Path,
+    format: EvalOutputFormat,
+    filter: Option<&str>,
+    verbose: bool,
+) -> anyhow::Result<()> {
+    let config = LintConfig::default();
+
+    println!("{} {}", "Evaluating:".cyan().bold(), path.display());
+    if let Some(f) = filter {
+        println!("  {} {}", "filter:".dimmed(), f);
+    }
+    println!();
+
+    let (results, summary) = evaluate_manifest_file(path, &config, filter)?;
+
+    // Show verbose per-case results if requested
+    if verbose {
+        println!("{}", "Per-Case Results".cyan().bold());
+        println!("{}", "=".repeat(60).dimmed());
+
+        for result in &results {
+            let status = if result.passed() {
+                "PASS".green().bold()
+            } else {
+                "FAIL".red().bold()
+            };
+
+            println!("[{}] {}", status, result.case.file.display());
+
+            if let Some(desc) = &result.case.description {
+                println!("     {}", desc.dimmed());
+            }
+
+            if !result.passed() {
+                if !result.false_positives.is_empty() {
+                    println!(
+                        "     {} {:?}",
+                        "unexpected:".yellow(),
+                        result.false_positives
+                    );
+                }
+                if !result.false_negatives.is_empty() {
+                    println!("     {} {:?}", "missing:".red(), result.false_negatives);
+                }
+            }
+            println!();
+        }
+
+        println!("{}", "=".repeat(60).dimmed());
+        println!();
+    }
+
+    // Output summary in requested format
+    let eval_format: EvalFormat = format.into();
+    match eval_format {
+        EvalFormat::Json => {
+            let json = summary.to_json()?;
+            println!("{}", json);
+        }
+        EvalFormat::Csv => {
+            let csv = summary.to_csv();
+            println!("{}", csv);
+        }
+        EvalFormat::Markdown => {
+            let md = summary.to_markdown();
+            println!("{}", md);
+        }
+    }
+
+    // Print final status
+    println!();
+    if summary.cases_failed == 0 {
+        println!(
+            "{} All {} cases passed",
+            "SUCCESS".green().bold(),
+            summary.cases_run
+        );
+    } else {
+        println!(
+            "{} {}/{} cases failed",
+            "FAILED".red().bold(),
+            summary.cases_failed,
+            summary.cases_run
+        );
+        process::exit(1);
+    }
+
+    Ok(())
+}