feat(garak): Add integration tests for Garak remote provider#2
Merged
kpunwatk merged 3 commits intokpunwatk:garak_inlinefrom Apr 1, 2026
Merged
feat(garak): Add integration tests for Garak remote provider#2kpunwatk merged 3 commits intokpunwatk:garak_inlinefrom
kpunwatk merged 3 commits intokpunwatk:garak_inlinefrom
Conversation
Implement comprehensive integration tests for the remote mode of the llama_stack_garak_provider across three tiers: - smoke (TestGarakRemoteQuickScan): predefined quick benchmark registration, eval job submission, status polling, and result retrieval - tier1 (TestGarakRemoteCustomBenchmark): custom benchmark with explicit garak_config metadata, probe selection, and result validation - tier2 (TestGarakRemoteShieldScan): shield registration with FMS guardrails orchestrator, benchmark with shield_ids, and shielded eval execution Key changes: - Support distribution_image override in llama_stack_server_config fixture to use specific LlamaStack 0.5.x images - Pre-generate CR names for consistent LlamaStack service URL construction - Add deployment namespace to NetworkPolicy allowedFrom for KFP pod access - Add guardrails orchestrator service URL fixture for in-cluster communication - Use provider-qualified model IDs (vllm-inference/<model>) for LlamaStack 0.5.x - Add eval job utilities with enhanced status logging and result validation Made-with: Cursor
|
The following are automatically added/executed:
Available user actions:
Supported labels{'/lgtm', '/wip', '/verified', '/cherry-pick', '/hold', '/build-push-pr-image'} |
saichandrapandraju
changed the title
The guardrails_orchestrator_ssl_cert, guardrails_orchestrator_ssl_cert_secret, and patched_llamastack_deployment_tls_certs fixtures are no longer needed since the shield tests now use verify_ssl=False with the HTTPS route. This resolves the unused-code CI failure. Made-with: Cursor
kpunwatk
reviewed
Mar 9, 2026
| default_garak_image = "quay.io/trustyai/garak-remote-provider:latest" | ||
| # Garak uses KUBEFLOW_GARAK_BASE_IMAGE; Ragas uses KUBEFLOW_BASE_IMAGE | ||
| # quay.io/rhoai/odh-trustyai-garak-lls-provider-dsp-rhel9@sha256:75eb795e9e459c0f6951ee1fc3ee325ae593d6aab32eee203723d28880c7ca31 (3.4-ea.1 sha) | ||
| # quay.io/opendatahub/odh-trustyai-garak-lls-provider-dsp@sha256:a3b65a9fdb6996fdaac45286b17522806cdf5af133275806fef5f93265103fc9 |
Author
There was a problem hiding this comment.
those are just for reference and all those images have same content. As I used the tag I pasted the links with sha just to be safe
|
|
||
| Reads the predictor service port from the cluster instead of hardcoding | ||
| a container port. Works correctly regardless of KServe headed/headless mode. | ||
| """ |
Owner
There was a problem hiding this comment.
Author
There was a problem hiding this comment.
can we modify that fixture to dynamically use port from svc instead of hardcoding?
Owner
There was a problem hiding this comment.
yeah you're right the fixture hardcodes the 8032 port, we could have modified the fixture but does make sense if to create a new one as per the test requirement
| def guardrails_orchestrator_service_url( | ||
| admin_client: DynamicClient, | ||
| model_namespace: Namespace, | ||
| guardrails_orchestrator: Any, |
Author
There was a problem hiding this comment.
We are setting up guardrails orchestrator to use for shields testing.
Owner
There was a problem hiding this comment.
then we can use the existing guardrail_orch fixtures https://github.com/opendatahub-io/opendatahub-tests/blob/292742e7d6e0113c223c4a428fdc98030dfe15f0/tests/fixtures/guardrails.py#L128 ?
|
Status of building tag garak_inline: success. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement comprehensive integration tests for the remote mode of the llama_stack_garak_provider across three tiers:
Key changes:
Made-with: Cursor