Skip to content

Commit d4549f8

Browse files
committed
Add NeMoGuardrails tests
Signed-off-by: Rob Geada <rob@geada.net>
1 parent 2a8c802 commit d4549f8

File tree

7 files changed

+1495
-1
lines changed

7 files changed

+1495
-1
lines changed

tests/model_explainability/README.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,12 @@ model_explainability/
3030
│ ├── test_lm_eval.py # HuggingFace, offline, vLLM, S3 tests
3131
│ └── utils.py
3232
33+
├── nemo_guardrails/ # NeMo Guardrails tests
34+
│ ├── conftest.py # NeMo CR, ConfigMap, Secret fixtures
35+
│ ├── constants.py # Test data, entity types, policies
36+
│ ├── test_nemo_guardrails.py # API, chat/completions, guardrail/checks, multi-server tests
37+
│ ├── utils.py # Config generation, request helpers
38+
3339
├── trustyai_operator/ # TrustyAI Operator validation
3440
│ ├── test_trustyai_operator.py # Operator image validation
3541
│ └── utils.py
@@ -65,6 +71,7 @@ model_explainability/
6571
- **`evalhub/`** - EvalHub service health endpoint validation via kube-rbac-proxy
6672
- **`guardrails/`** - Guardrails Orchestrator tests with built-in regex detectors (PII), HuggingFace detectors (prompt injection, HAP), auto-configuration, and gateway routing. Includes OpenTelemetry/Tempo trace integration
6773
- **`lm_eval/`** - Language Model Evaluation tests covering HuggingFace models, local/offline tasks, vLLM integration, S3 storage, and OCI registry artifacts
74+
- **`nemo_guardrails/`** - NeMo Guardrails tests for LLM-as-a-judge (self-check policies), Presidio PII detection (email, SSN, credit card, person names), multi-server deployments, multi-configuration servers, authentication (kube-rbac-proxy), and secret mounting for API tokens
6875
- **`trustyai_operator/`** - TrustyAI operator container image validation (SHA256 digests, CSV relatedImages)
6976
- **`trustyai_service/`** - TrustyAI Service tests for drift detection (4 metrics), fairness metrics (SPD, DIR), database migration, multi-namespace support, and upgrade scenarios. Tests run against both PVC and database storage backends
7077

@@ -79,6 +86,7 @@ model_explainability/
7986
@pytest.mark.post_upgrade # Post-upgrade tests
8087
@pytest.mark.rawdeployment # KServe raw deployment mode
8188
@pytest.mark.skip_on_disconnected # Requires internet connectivity
89+
@pytest.mark.nemo_guardrails # NeMo Guardrails specific tests
8290
```
8391

8492
## Running Tests
@@ -95,9 +103,12 @@ uv run pytest tests/model_explainability/
95103
# Run TrustyAI Service tests
96104
uv run pytest tests/model_explainability/trustyai_service/
97105

98-
# Run Guardrails tests
106+
# Run Guardrails Orchestrator tests
99107
uv run pytest tests/model_explainability/guardrails/
100108

109+
# Run NeMo Guardrails tests
110+
uv run pytest tests/model_explainability/nemo_guardrails/
111+
101112
# Run LM Eval tests
102113
uv run pytest tests/model_explainability/lm_eval/
103114

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""NeMo Guardrails test suite for OpenDataHub/RHOAI."""

0 commit comments

Comments
 (0)