sync: main to incubation

BENCHMARK_METADATA_REFERENCE.md

@@ -0,0 +1,288 @@
+# Benchmark Metadata Reference
+
+This document is the reference for fields supported in:
+
+```python
+client.alpha.benchmarks.register(..., metadata={...})
+```
+
+It covers:
+
+- `garak_config` (detailed command config)
+- shield fields (`shield_ids`, `shield_config`)
+- runtime controls (`timeout`, remote-only retry/GPU keys)
+- deep-merge behavior when updating predefined/existing benchmarks
+
+## 1) Metadata Shape
+
+```python
+metadata = {
+    "garak_config": {
+        "system": {...},
+        "run": {...},
+        "plugins": {...},
+        "reporting": {...},
+    },
+    "timeout": 1800,
+    "shield_ids": ["Prompt-Guard-86M"],  # or use shield_config
+    "max_retries": 3,                    # remote mode only
+    "use_gpu": False,                    # remote mode only
+}
+```
+
+If `garak_config` is omitted, provider falls back to default Garak config (effectively broad/default probe selection), which can be very slow.
+
+### 1.1 Build `garak_config` via Python models (optional)
+
+You can construct config using typed models exported by this package:
+
+```python
+from llama_stack_provider_trustyai_garak import (
+    GarakCommandConfig,
+    GarakSystemConfig,
+    GarakRunConfig,
+    GarakPluginsConfig,
+    GarakReportingConfig,
+)
+```
+
+Example:
+
+```python
+garak_cfg = GarakCommandConfig(
+    system=GarakSystemConfig(parallel_attempts=20),
+    run=GarakRunConfig(generations=2, eval_threshold=0.5),
+    plugins=GarakPluginsConfig(probe_spec=["promptinject.HijackHateHumans"]),
+    reporting=GarakReportingConfig(taxonomy="owasp"),
+)
+
+metadata = {
+    "garak_config": garak_cfg.to_dict(),
+    "timeout": 900,
+}
+```
+
+## 2) Top-Level Metadata Keys
+
+| Key | Type | Default | Mode | Notes |
+|---|---|---|---|---|
+| `garak_config` | `dict` | default `GarakCommandConfig()` | inline + remote | Main Garak command schema. Recommended to always set. |
+| `timeout` | `int` (seconds) | provider default (`10800`) | inline + remote | Max scan runtime for a benchmark run. |
+| `shield_ids` | `list[str]` | `[]` | inline + remote | Shortcut for input shields only. |
+| `shield_config` | `dict` | `{}` | inline + remote | Explicit mapping: `{"input": [...], "output": [...]}`. |
+| `max_retries` | `int` | `3` | remote only | KFP pipeline retry count for scan step. |
+| `use_gpu` | `bool` | `False` | remote only | Requests GPU scheduling in KFP pipeline. |
+
+Notes:
+
+- If both `shield_ids` and `shield_config` are provided, `shield_ids` takes precedence.
+- Unknown top-level keys are passed as provider params but are ignored unless consumed by adapter logic.
+
+## 3) Shield Metadata Rules
+
+### `shield_ids`
+
+```python
+"shield_ids": ["Prompt-Guard-86M"]
+```
+
+- Must be a list.
+- Treated as input shields.
+- Easier syntax for common cases.
+
+### `shield_config`
+
+```python
+"shield_config": {
+    "input": ["Prompt-Guard-86M"],
+    "output": ["Llama-Guard-3-8B"]
+}
+```
+
+- Must be a dictionary.
+- Use when you need separate input/output shield chains.
+
+Validation behavior:
+
+- Provider validates shield IDs against Shields API.
+- If Shields API is not enabled and shield metadata is present, run fails.
+
+## 4) `garak_config` Detailed Schema
+
+`garak_config` has four primary sections:
+
+- `system`
+- `run`
+- `plugins`
+- `reporting`
+
+### 4.1 `garak_config.system`
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `parallel_attempts` | `bool \| int` | `16` | Parallel prompt attempts where supported. |
+| `max_workers` | `int` | `500` | Upper bound for requested worker count. |
+| `parallel_requests` | `bool \| int` | `False` | Parallel requests for generators lacking multi-response support. |
+| `verbose` | `int` (`0..2`) | `0` | CLI verbosity. |
+| `show_z` | `bool` | `False` | Show Z-scores in CLI output. |
+| `narrow_output` | `bool` | `False` | Improve output for narrow terminals. |
+| `lite` | `bool` | `True` | Lite mode caution output behavior. |
+| `enable_experimental` | `bool` | `False` | Enable experimental Garak flags. |
+
+### 4.2 `garak_config.run`
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `generations` | `int` | `1` | Number of generations per prompt. |
+| `probe_tags` | `str \| None` | `None` | Tag-based probe selection (e.g. `owasp:llm`). |
+| `eval_threshold` | `float` (`0..1`) | `0.5` | Detector threshold for hit/vulnerable decision. |
+| `soft_probe_prompt_cap` | `int` | `256` | Preferred prompt cap for autoscaling probes. Lower values reduce prompts per probe and make runs faster (with reduced coverage/comprehensiveness). |
+| `target_lang` | `str \| None` | `None` | BCP47 language target. |
+| `langproviders` | `list[str] \| None` | `None` | Providers for language conversion. |
+| `system_prompt` | `str \| None` | `None` | Default system prompt where applicable. |
+| `seed` | `int \| None` | `None` | Reproducibility seed. |
+| `deprefix` | `bool` | `True` | Remove prompt prefix echoed by model outputs. |
+
+Performance tuning tip:
+
+- Predefined benchmarks are comprehensive by default.
+- To speed up exploratory runs, override `garak_config.run.soft_probe_prompt_cap` with a smaller value.
+- For full security assessment/comparability, keep defaults (or use consistent cap across compared runs).
+
+### 4.3 `garak_config.plugins`
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `probe_spec` | `list[str] \| str` | `"all"` | Probe/module/class selection. |
+| `detector_spec` | `list[str] \| str \| None` | `None` | Detector override (`None` uses probe defaults). |
+| `extended_detectors` | `bool` | `True` | Include extended detector set. |
+| `buff_spec` | `list[str] \| str \| None` | `None` | Buff/module selection. |
+| `buffs_include_original_prompt` | `bool` | `True` | Keep original prompt when buffing. |
+| `buff_max` | `int \| None` | `None` | Cap output count from buffs. |
+| `target_type` | `str` | auto-managed | Provider sets this for openai/function mode. |
+| `target_name` | `str \| None` | auto-managed | Provider sets this to model or shield orchestrator. |
+| `probes` | `dict \| None` | `None` | Probe plugin config tree. |
+| `detectors` | `dict \| None` | `None` | Detector plugin config tree. |
+| `generators` | `dict \| None` | `None` | Generator plugin config tree. |
+| `buffs` | `dict \| None` | `None` | Buff plugin config tree. |
+| `harnesses` | `dict \| None` | `None` | Harness plugin config tree. |
+
+Provider behavior worth knowing:
+
+- `probe_spec`, `detector_spec`, `buff_spec` accept string or list, and are normalized before run.
+- If shield metadata is present, provider switches generator mode to function-based shield orchestration automatically.
+- Otherwise provider uses OpenAI-compatible generator mode.
+
+### 4.4 `garak_config.reporting`
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `taxonomy` | `str \| None` | `None` | Grouping taxonomy (`owasp`, `avid-effect`, `quality`, `cwe`). |
+| `show_100_pass_modules` | `bool` | `True` | Include fully passing entries in HTML report details. |
+| `show_top_group_score` | `bool` | `True` | Show top-level aggregate in grouped report sections. |
+| `group_aggregation_function` | `str` | `"lower_quartile"` | Group aggregation strategy in report. |
+| `report_dir` | `str \| None` | auto-managed | Provider-managed output location; usually leave unset. |
+| `report_prefix` | `str \| None` | auto-managed | Provider-managed output prefix; usually leave unset. |
+
+Please refer to [Garak configuration docs](https://reference.garak.ai/en/latest/configurable.html#config-files-yaml-and-json) for details about these controls.
+
+## 5) Deep-Merge Behavior (Updating Predefined/Existing Benchmarks)
+
+When registering with `provider_benchmark_id`, metadata is deep-merged:
+
+- base metadata comes from:
+  - predefined profile (`trustyai_garak::...`), or
+  - existing benchmark metadata
+- your new metadata overrides only specified keys
+
+Example:
+
+```python
+client.alpha.benchmarks.register(
+    benchmark_id="quick_promptinject_tuned",
+    dataset_id="garak",
+    scoring_functions=["garak_scoring"],
+    provider_id=garak_provider_id,
+    provider_benchmark_id="trustyai_garak::quick",
+    metadata={
+        "garak_config": {
+            "plugins": {"probe_spec": ["promptinject"]},
+            "system": {"parallel_attempts": 20},
+        },
+        "timeout": 1200,
+    },
+)
+```
+
+## 6) Practical Examples
+
+### Example A: Minimal custom benchmark
+
+```python
+metadata = {
+    "garak_config": {
+        "plugins": {"probe_spec": ["promptinject.HijackHateHumans"]},
+        "run": {"generations": 2, "eval_threshold": 0.5},
+        "reporting": {"taxonomy": "owasp"},
+    },
+    "timeout": 900,
+}
+```
+
+### Example B: Explicit input/output shield mapping
+
+```python
+metadata = {
+    "garak_config": {
+        "plugins": {"probe_spec": ["promptinject.HijackHateHumans"]},
+    },
+    "shield_config": {
+        "input": ["Prompt-Guard-86M"],
+        "output": ["Llama-Guard-3-8B"],
+    },
+    "timeout": 600,
+}
+```
+
+### Example C: Remote retry/GPU controls
+
+```python
+metadata = {
+    "garak_config": {
+        "run": {"probe_tags": "owasp:llm"},
+    },
+    "timeout": 7200,
+    "max_retries": 2,
+    "use_gpu": True,
+}
+```
+
+### Example D: Faster predefined benchmark variant
+
+```python
+metadata = {
+    "garak_config": {
+        "run": {
+            "soft_probe_prompt_cap": 100
+        }
+    },
+    "timeout": 7200,
+}
+
+# Register as a tuned variant of a predefined benchmark
+client.alpha.benchmarks.register(
+    benchmark_id="owasp_fast",
+    dataset_id="garak",
+    scoring_functions=["garak_scoring"],
+    provider_id=garak_provider_id,
+    provider_benchmark_id="trustyai_garak::owasp_llm_top10",
+    metadata=metadata,
+)
+```
+
+## 7) Legacy / Compatibility Notes
+
+- Prefer `metadata.garak_config.plugins.probe_spec` over old top-level `metadata.probes`.
+- Prefer `metadata.garak_config.run.eval_threshold` for threshold control.
+- Keep benchmark metadata focused on benchmark/run concerns.
+  KFP control-plane settings such as `experiment_name` belong in provider config (`kubeflow_config.experiment_name`, environment: `KUBEFLOW_EXPERIMENT_NAME`), not benchmark metadata.

COMPATIBILITY.md

@@ -6,14 +6,25 @@ This document tracks the compatibility of `llama-stack-provider-trustyai-garak`
 
 | Provider Version | Llama-Stack Version | Python Version | Key Dependencies | Status | Notes |
 |------------------|---------------------|----------------|------------------|---------|-------|
-| 0.1.3 | ==0.2.18 | >=3.12 | greenlet, httpx[http2], kfp, kfp-kubernetes, kfp-server-api, boto3, garak | Current | Latest stable release with thin dependencies and lazy kfp & s3 client init for remote mode |
+| 0.2.0 | >=0.5.0 | >=3.12 | kfp>=2.14.6, kfp-kubernetes>=2.14.6, kfp-server-api>=2.14.6, boto3>=1.35.88 | Current | Current release with updated metadata schema (`metadata.garak_config`) and remote/inline support |
+| 0.1.3 | ==0.2.18 | >=3.12 | greenlet, httpx[http2], kfp, kfp-kubernetes, kfp-server-api, boto3, garak | | Latest stable release with thin dependencies and lazy kfp & s3 client init for remote mode |
 | 0.1.2 | >=0.2.15 | >=3.12 | fastapi, opentelemetry-api, opentelemetry-exporter-otlp, aiosqlite, greenlet, uvicorn, ipykernel, httpx[http2], kfp, kfp-kubernetes, kfp-server-api, boto3, garak | | Release with both remote and inline implementation |
 | 0.1.1 | >=0.2.15 | >=3.12 | fastapi, opentelemetry-api, opentelemetry-exporter-otlp, aiosqlite, greenlet, uvicorn, ipykernel, httpx[http2], garak |  | Initial stable release with inline implementation |
 
 ## Dependency Details
 
 ### Core Dependencies
 
+#### Version 0.2.0 (latest)
+- **llama-stack-client**: >=0.5.0
+- **llama-stack-api**: >=0.5.0
+- **llama-stack** (server extra): >=0.5.0
+- **garak** (inline extra): ==0.14.0
+- **kfp**: >=2.14.6
+- **kfp-kubernetes**: >=2.14.6
+- **kfp-server-api**: >=2.14.6
+- **boto3**: >=1.35.88
+
 #### Version 0.1.3
 - **llama-stack**: == 0.2.18
 - **greenlet**: Latest compatible (3.2.4)
@@ -68,6 +79,16 @@ The provider is built and compatible with:
 - **Llama-Stack Version**: 0.2.18 (in container builds)
 - **Additional Runtime Dependencies**: torch, transformers, sqlalchemy, and others as specified in the Containerfile
 
+## Image Compatibility (Latest Deployments)
+
+Use the table below as a quick reference for image fields used in current remote deployments.
+
+| Use Case | Config Key / Field | Where to Set | Recommended Image | Alternative | Notes |
+|---|---|---|---|---|---|
+| LLS distro image (total remote) | `spec.distribution.image` | `lsd_remote/llama_stack_distro-setup/lsd-garak.yaml` | `quay.io/opendatahub/llama-stack@sha256:cf21d3919d265f8796ed600bfe3d2eb3ce797b35ab8e60ca9b6867e0516675e5` | `quay.io/rhoai/odh-llama-stack-core-rhel9:rhoai-3.4` | Pick image matching your RHOAI/ODH release stream |
+| Garak KFP base image (total remote) | `KUBEFLOW_GARAK_BASE_IMAGE` | `lsd_remote/llama_stack_distro-setup/lsd-config.yaml` | `quay.io/opendatahub/odh-trustyai-garak-lls-provider-dsp:dev` | `quay.io/rhoai/odh-trustyai-garak-lls-provider-dsp-rhel9:rhoai-3.4` | Injected into LSD env via `lsd-garak.yaml` |
+| Garak KFP base image (partial remote) | `kubeflow_config.garak_base_image` (env: `KUBEFLOW_GARAK_BASE_IMAGE`) | `demos/2-partial-remote/partial-remote.yaml` | `quay.io/opendatahub/odh-trustyai-garak-lls-provider-dsp:dev` | `quay.io/rhoai/odh-trustyai-garak-lls-provider-dsp-rhel9:rhoai-3.4` | Used by KFP components for scan/parse/validate steps |
+
 ## Breaking Changes
 
 ### Version 0.1.3

Containerfile

@@ -15,22 +15,25 @@ COPY . .
 # Build argument to specify architecture
 ARG TARGETARCH=x86_64
 
-# Install dependencies
-RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "x86_64" ]; then \
-        echo "Installing x86_64 dependencies ..."; \
-        pip install --no-cache-dir -r requirements-x86_64.txt; \
-    elif [ "$TARGETARCH" = "arm64" ] || [ "$TARGETARCH" = "aarch64" ]; then \
-        echo "Installing ARM64 dependencies ..."; \
-        pip install --no-cache-dir -r requirements-aarch64.txt; \
-    else \
-        echo "ERROR: Unsupported architecture: $TARGETARCH"; \
-        exit 1; \
-    fi
-
-# Install the package itself (--no-deps since dependencies already installed)
+# # Install dependencies
+# RUN if [ "$TARGETARCH" = "amd64" ] || [ "$TARGETARCH" = "x86_64" ]; then \
+#         echo "Installing x86_64 dependencies ..."; \
+#         pip install --no-cache-dir -r requirements-x86_64.txt; \
+#     elif [ "$TARGETARCH" = "arm64" ] || [ "$TARGETARCH" = "aarch64" ]; then \
+#         echo "Installing ARM64 dependencies ..."; \
+#         pip install --no-cache-dir -r requirements-aarch64.txt; \
+#     else \
+#         echo "ERROR: Unsupported architecture: $TARGETARCH"; \
+#         exit 1; \
+#     fi
+
+# Install cpu torch to reduce image size
+RUN pip install torch --index-url https://download.pytorch.org/whl/cpu
+
+# Install the package itself
 # Use [inline] to get garak dependency
-RUN pip install --no-cache-dir --no-deps -e ".[inline]"
-
+RUN pip install --no-cache-dir ".[inline]"
+RUN pip install --no-cache-dir -r requirements-inline-extra.txt
 # Set XDG environment variables to use /tmp (always writable) for garak to write to
 ENV XDG_CACHE_HOME=/tmp/.cache
 ENV XDG_DATA_HOME=/tmp/.local/share