Skip to content

Commit 070c714

Browse files
saichandrapandrajusaichandrapandraju
authored
Merge pull request #74 from trustyai-explainability/main
[pull] main from trustyai-explainability:main
2 parents d283004 + 6fe663b commit 070c714

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+4616
-2412
lines changed

‎.github/CODEOWNERS‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Default owners for the entire repository
2+
* @trustyai-explainability/developers

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Bug Report
2+
description: Report a bug in llama-stack-provider-trustyai-garak
3+
labels: ["bug"]
4+
body:
5+
- type: markdown
6+
attributes:
7+
value: |
8+
Thank you for reporting a bug. Please fill out the sections below
9+
to help us reproduce and fix the issue.
10+
11+
- type: textarea
12+
id: description
13+
attributes:
14+
label: Bug Description
15+
description: A clear and concise description of the bug.
16+
validations:
17+
required: true
18+
19+
- type: textarea
20+
id: reproduction
21+
attributes:
22+
label: Steps to Reproduce
23+
description: Minimal steps to reproduce the behavior.
24+
placeholder: |
25+
1. Register benchmark with config...
26+
2. Run eval with...
27+
3. Observe error...
28+
validations:
29+
required: true
30+
31+
- type: textarea
32+
id: expected
33+
attributes:
34+
label: Expected Behavior
35+
description: What you expected to happen.
36+
validations:
37+
required: true
38+
39+
- type: textarea
40+
id: actual
41+
attributes:
42+
label: Actual Behavior
43+
description: What actually happened, including any error messages.
44+
validations:
45+
required: true
46+
47+
- type: textarea
48+
id: logs
49+
attributes:
50+
label: Error Logs
51+
description: Paste relevant logs or stack traces.
52+
render: text
53+
54+
- type: dropdown
55+
id: execution-mode
56+
attributes:
57+
label: Execution Mode
58+
options:
59+
- Llama Stack Inline (local garak)
60+
- Llama Stack Remote (KFP pipelines)
61+
- Llama Stack (all modes)
62+
- Eval-Hub Simple (direct pod execution)
63+
- Eval-Hub KFP (KFP pipeline execution)
64+
- Eval-Hub (all modes)
65+
validations:
66+
required: true
67+
68+
- type: textarea
69+
id: environment
70+
attributes:
71+
label: Environment
72+
description: Provide environment details.
73+
placeholder: |
74+
- Provider version:
75+
- Python version:
76+
- Garak version:
77+
- Llama Stack version:
78+
- OS / Platform:
79+
- Kubernetes version (if remote):
80+
validations:
81+
required: true
82+
83+
- type: textarea
84+
id: config
85+
attributes:
86+
label: Benchmark / Garak Config
87+
description: Paste relevant benchmark config or garak_config if applicable.
88+
render: yaml

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
## Summary
2+
3+
<!-- Brief description of what this PR does and why. -->
4+
5+
## Changes
6+
7+
<!-- List the key changes. -->
8+
9+
-
10+
11+
## Testing Checklist
12+
13+
- [ ] Unit tests pass (`make test`)
14+
- [ ] Linting passes (`make lint`)
15+
- [ ] New/changed code has test coverage
16+
- [ ] No breaking changes to existing benchmark configs
17+
- [ ] Documentation updated (if applicable)
18+
19+
## Related Issues
20+
21+
<!-- Link any related issues: Fixes #123, Relates to #456 -->

‎.github/workflows/lint.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: Lint
2+
3+
on:
4+
pull_request:
5+
branches: [main]
6+
push:
7+
branches: [main]
8+
9+
jobs:
10+
ruff:
11+
name: Ruff Lint & Format Check
12+
runs-on: ubuntu-latest
13+
14+
steps:
15+
- name: Checkout code
16+
uses: actions/checkout@v4
17+
18+
- name: Set up Python
19+
uses: actions/setup-python@v4
20+
with:
21+
python-version: '3.12'
22+
23+
- name: Install tools
24+
run: pip install ruff mypy
25+
26+
- name: Ruff check
27+
run: ruff check src/ tests/
28+
29+
- name: Ruff format check
30+
run: ruff format --check src/ tests/
31+
32+
- name: Mypy type check
33+
run: mypy src/

‎.github/workflows/run-tests.yml‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Run Tests
1+
name: Tier 1 - Unit Tests
22

33
on:
44
pull_request:
@@ -7,26 +7,32 @@ on:
77
branches: [main]
88

99
jobs:
10-
test:
11-
name: Run Tests
10+
unit-tests:
11+
name: Unit Tests
1212
runs-on: ubuntu-latest
13-
13+
1414
steps:
1515
- name: Checkout code
1616
uses: actions/checkout@v4
1717

1818
- name: Set up Python
1919
uses: actions/setup-python@v5
2020
with:
21-
python-version: '3.12'
21+
python-version: "3.12"
2222

2323
- name: Install dependencies
2424
run: |
2525
python -m pip install --upgrade pip
26-
python -m pip install -e ".[dev,inline,server]"
26+
python -m pip install --no-cache-dir -e ".[test]"
27+
28+
- name: Smoke-test imports
29+
run: |
30+
python -c "import numpy; print('numpy OK')"
31+
python -c "import pandas; print('pandas OK')"
32+
python -c "import llama_stack_provider_trustyai_garak; print('provider OK')"
2733
2834
- name: Run tests
2935
env:
3036
PYTHONPATH: src
3137
run: |
32-
pytest tests -v
38+
pytest tests -v

‎.github/workflows/security.yml‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,27 +10,28 @@ jobs:
1010
trivy-scan:
1111
name: Trivy Security Scan
1212
runs-on: ubuntu-latest
13+
container:
14+
image: registry.access.redhat.com/ubi9/python-312:latest
15+
options: --user root
1316
permissions:
1417
contents: read
1518
security-events: write
1619
actions: read
1720

1821
steps:
22+
1923
- name: Checkout code
2024
uses: actions/checkout@v4
2125

22-
- name: Set up Python
23-
uses: actions/setup-python@v4
24-
with:
25-
python-version: '3.12'
26-
27-
- name: Install dependencies
26+
- name: Install runtime deps
2827
run: |
2928
python -m pip install --upgrade pip
30-
python -m pip install -e ".[dev]"
29+
python -m pip install --no-cache-dir \
30+
-r requirements.txt
31+
python -m pip install --no-cache-dir --no-deps .
3132
3233
- name: Run Trivy filesystem scan
33-
uses: aquasecurity/trivy-action@master
34+
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
3435
with:
3536
scan-type: 'fs'
3637
scan-ref: '.'
@@ -40,7 +41,7 @@ jobs:
4041
exit-code: '0' # Don't fail on this scan, we'll check results separately
4142

4243
- name: Run Trivy dependency scan
43-
uses: aquasecurity/trivy-action@master
44+
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
4445
with:
4546
scan-type: 'fs'
4647
scan-ref: '.'
@@ -51,7 +52,7 @@ jobs:
5152
exit-code: '0' # Don't fail on this scan, we'll check results separately
5253

5354
- name: Check for critical vulnerabilities
54-
uses: aquasecurity/trivy-action@master
55+
uses: aquasecurity/trivy-action@57a97c7e7821a5776cebc9bb87c984fa69cba8f1 # v0.35.0
5556
with:
5657
scan-type: 'fs'
5758
scan-ref: '.'

‎.github/workflows/validate-deps.yml‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
name: Validate Dependencies
2+
3+
on:
4+
pull_request:
5+
branches: [main]
6+
push:
7+
branches: [main]
8+
9+
permissions:
10+
contents: write
11+
12+
jobs:
13+
sync-requirements:
14+
name: Auto-sync requirements.txt
15+
if: github.event_name == 'pull_request'
16+
runs-on: ubuntu-latest
17+
18+
steps:
19+
- name: Checkout PR branch
20+
uses: actions/checkout@v4
21+
with:
22+
ref: ${{ github.head_ref }}
23+
token: ${{ secrets.GITHUB_TOKEN }}
24+
25+
- name: Set up Python
26+
uses: actions/setup-python@v4
27+
with:
28+
python-version: '3.12'
29+
30+
- name: Install uv
31+
run: python -m pip install --upgrade pip uv
32+
33+
- name: Regenerate requirements.txt
34+
run: |
35+
uv pip compile \
36+
--python-platform linux \
37+
--extra inline \
38+
--emit-index-url \
39+
--default-index https://console.redhat.com/api/pypi/public-rhai/rhoai/3.4/cpu-ubi9-test/simple/ \
40+
pyproject.toml \
41+
--index-url https://console.redhat.com/api/pypi/public-rhai/rhoai/3.4/cpu-ubi9-test/simple/ \
42+
-o requirements.txt
43+
44+
- name: Commit if changed
45+
run: |
46+
git diff --quiet requirements.txt && exit 0
47+
git config user.name "github-actions[bot]"
48+
git config user.email "github-actions[bot]@users.noreply.github.com"
49+
git add requirements.txt
50+
git commit -m "chore: auto-sync requirements.txt from pyproject.toml"
51+
git push
52+
53+
check-garak-drift:
54+
name: Check garak midstream version drift
55+
runs-on: ubuntu-latest
56+
57+
steps:
58+
- name: Checkout code
59+
uses: actions/checkout@v4
60+
61+
- name: Compare pyproject.toml garak version with latest midstream tag
62+
run: |
63+
PYPROJECT_VER=$(grep -oP 'garak==\K[^\s"]+' pyproject.toml)
64+
echo "pyproject.toml garak version: $PYPROJECT_VER"
65+
66+
LATEST_TAG=$(git ls-remote --tags \
67+
https://github.com/trustyai-explainability/garak.git \
68+
| grep 'refs/tags/v' \
69+
| grep -v '\^{}' \
70+
| sed 's|.*refs/tags/v||' \
71+
| sort -V \
72+
| tail -1)
73+
echo "Latest midstream tag: $LATEST_TAG"
74+
75+
if [ "$PYPROJECT_VER" != "$LATEST_TAG" ]; then
76+
echo "::error::Garak version drift detected!"
77+
echo " pyproject.toml pins: $PYPROJECT_VER"
78+
echo " Latest midstream: $LATEST_TAG"
79+
echo "Update pyproject.toml, regenerate requirements.txt, and commit."
80+
exit 1
81+
fi
82+
83+
echo "Garak version is up-to-date with midstream."
84+
85+
container-build:
86+
name: Container Build + Import Validation
87+
runs-on: ubuntu-latest
88+
89+
steps:
90+
- name: Checkout code
91+
uses: actions/checkout@v4
92+
93+
- name: Build container image
94+
run: |
95+
docker build -f Containerfile -t provider-smoke-test:ci .
96+
97+
- name: Verify full import chain
98+
run: |
99+
docker run --rm provider-smoke-test:ci bash -c "\
100+
python -c \"import numpy; print('numpy OK')\" && \
101+
python -c \"import pandas; print('pandas OK')\" && \
102+
python -c \"import garak; print('garak OK')\" && \
103+
python -c \"import sdg_hub; print('sdg-hub OK')\" && \
104+
python -c \"import llama_stack_provider_trustyai_garak; print('provider OK')\""
105+
106+
- name: Verify garak version matches pyproject.toml
107+
run: |
108+
EXPECTED=$(grep -oP 'garak==\K[^\s"]+' pyproject.toml)
109+
INSTALLED=$(docker run --rm provider-smoke-test:ci python -c "from importlib.metadata import version; print(version('garak'))")
110+
echo "Expected: $EXPECTED"
111+
echo "Installed: $INSTALLED"
112+
if [ "$EXPECTED" != "$INSTALLED" ]; then
113+
echo "::error::Garak version mismatch! Containerfile installs $INSTALLED but pyproject.toml expects $EXPECTED"
114+
exit 1
115+
fi
116+
echo "Garak version in container matches pyproject.toml."

‎.pre-commit-config.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
repos:
2+
- repo: local
3+
hooks:
4+
- id: sync-requirements
5+
name: Regenerate requirements.txt from pyproject.toml
6+
entry: bash -c 'uv pip compile --python-platform linux --extra inline --emit-index-url --default-index https://console.redhat.com/api/pypi/public-rhai/rhoai/3.4/cpu-ubi9-test/simple/ pyproject.toml --index-url https://console.redhat.com/api/pypi/public-rhai/rhoai/3.4/cpu-ubi9-test/simple/ -o requirements.txt'
7+
language: system
8+
files: ^pyproject\.toml$
9+
pass_filenames: false
10+
- repo: https://github.com/astral-sh/ruff-pre-commit
11+
rev: v0.11.4
12+
hooks:
13+
- id: ruff
14+
args: [--fix, --exit-non-zero-on-fix]
15+
- id: ruff-format

0 commit comments

Comments
 (0)