[Feature] Add Modern Evaluation Metrics Framework

[anivar]

Problem Statement

any-guardrail currently has no evaluation framework or metrics. This makes it impossible to:

• Measure guardrail performance scientifically
• Compare different guardrail models
• Validate improvements
• Benchmark against competitors

Current State

• No evaluation metrics (checked codebase)
• Only basic pass/fail in GuardrailOutput.valid
• No benchmarking system
• No performance measurement

Proposed Solution

Add comprehensive evaluation framework with modern metrics:

from any_guardrail.evaluation import Evaluator

evaluator = Evaluator()
results = evaluator.evaluate(
    guardrail,
    test_data,
    metrics=['accuracy', 'f1', 'latency']
)

print(results)
# {
#   'accuracy': 0.92,
#   'f1_score': 0.89,
#   'latency_ms': 45,
#   'confusion_matrix': [[...]]
# }

Features to Add

1. Basic Metrics:

   • Accuracy, Precision, Recall, F1
   • Confusion matrices
   • ROC curves

2. Advanced Metrics:

   • Semantic similarity (BERTScore)
   • Cross-lingual consistency
   • Latency and throughput

3. Benchmarking:

   • Standard test sets
   • Comparison reports
   • CI/CD integration

Implementation Plan

# src/any_guardrail/evaluation/
├── metrics.py       # Core metrics
├── benchmarks.py    # Standard datasets
├── reporter.py      # Generate reports
└── __init__.py

Why This Matters

• Can't improve what you can't measure
• Users need to compare guardrail performance
• Required for research papers and publications
• Essential for production deployment decisions

References

• scikit-learn metrics
• BERTScore

[angpt]

Thank you for raising this issue! We agree that evaluation is a core component of utilizing guardrails.

We intend any-guardrail to be a lightweight, easy to use library for switching between guardrails, so researchers and practitioners can figure out which guardrails fit their downstream use case. However, providing specific metrics, benchmarks etc within any-guardrails is out of scope. Evaluations are heavily dependent on individual use cases, and quite honestly, are broader than guardrails themselves.