feat: add regulatory compliance evaluation metrics with NIST AI RMF alignment

[garyatwalAI]

Summary

Adds a new mlflow-regulatory-compliance plugin package (under cookbook/) providing four governance-focused evaluation metrics for LLM and ML model assessment: PII detection, legal privilege detection, factual grounding, and bias detection. Includes a composite NIST AI RMF compliance score and structured compliance report generator.

• PII Detection Metric: Detects 9 categories of personally identifiable information using pattern matching and validation algorithms (Luhn check for credit cards, area number validation for SSNs)
• Legal Privilege Detection Metric: Flags attorney-client, work product, and settlement communications using multi-layer detection with confidence scoring and false positive mitigation
• Factual Grounding Metric: Measures RAG faithfulness by checking if model claims are supported by provided context using token and n-gram overlap
• Bias Detection Metric: Detects demographic bias across 5 dimensions (gender, racial/ethnic, age, disability, socioeconomic) with configurable sensitivity levels
• NIST AI RMF Composite Score: Weighted aggregate score mapping to GOVERN, MAP, MEASURE, MANAGE functions
• Compliance Report Generator: Structured report with per-function scores, pass/fail status, and remediation recommendations
• Full MLflow Integration: Works with mlflow.evaluate() via standard make_metric() / extra_metrics interface

Motivation

Organisations deploying AI in regulated industries (legal services, insurance, financial services, healthcare) need to evaluate models against governance and compliance criteria — not just accuracy and latency. Current MLflow evaluation metrics cover model quality but lack coverage for regulatory compliance dimensions that are increasingly required by frameworks like the NIST AI Risk Management Framework, ISO 42001, the EU AI Act, and state-level AI laws in Colorado, Texas, and California.

This plugin fills that gap by providing compliance evaluation metrics that integrate directly into existing MLflow evaluation workflows, so teams can assess governance alongside performance without switching tools.

NIST AI RMF Mapping

NIST Function	Metric	What It Measures
GOVERN	Meta-assessment	Whether all governance evaluators are active
MAP	Factual Grounding	Risk of hallucination and ungrounded claims
MEASURE	PII + Bias Detection	Compliance with data protection and fairness requirements
MANAGE	Legal Privilege	Runtime prevention of privileged information disclosure

Example Usage

import mlflow
from mlflow_regulatory_compliance import (
    pii_detection_metric,
    legal_privilege_metric,
    factual_grounding_metric,
    bias_detection_metric,
    nist_composite_metric,
)

results = mlflow.evaluate(
    data=eval_df,
    predictions="predictions",
    extra_metrics=[
        pii_detection_metric,
        legal_privilege_metric,
        factual_grounding_metric,
        bias_detection_metric,
        nist_composite_metric,
    ],
)

print(f"NIST Compliance Score: {results.metrics['nist_compliance_score/v1/mean']:.2f}")

Tests

• PII detection — all 9 categories, Luhn validation, false positives, edge cases
• Legal privilege — all 3 categories, confidence scoring, false positives
• Factual grounding — grounded, partial, hallucinated inputs
• Bias detection — all 5 dimensions, sensitivity levels, custom terms
• NIST composite — weighting, thresholds, aggregation
• Compliance evaluator — metric selection, configuration
• Report generator — output format, status determination, recommendations
• Edge cases — empty input, None input, Unicode handling
• 159 tests passing, ruff lint clean

Compatibility

• MLflow >= 2.10
• Python >= 3.9
• No heavy ML dependencies in default mode (regex + pattern matching)

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[vercel]

@garyatwalAI is attempting to deploy a commit to the langfuse Team on Vercel.

A member of the Team first needs to authorize it.

[garyatwalAI]

Hi @marcklingen @jannikmaierhoefer — would appreciate a review when you get a chance. This adds an mlflow-regulatory-compliance plugin package to the cookbook with NIST AI RMF-aligned evaluation metrics (PII detection, legal privilege detection, factual grounding, bias detection) that integrate with mlflow.evaluate(). All 159 tests passing. Thanks!