Skip to content

Commit ed5cc4d

Browse files
LeoRoccoBreedtclaude
LeoRoccoBreedt
and
claude
committed
ci: add PR test, scheduled runs, and compliance check workflows
Replaces legacy e2e-tests.yml and test-scripts-matrix.yml with three focused workflows: - pr-examples-test.yml: detects changed example folders that have run_examples.sh, runs each in a matrix job using real Opik credentials and a cheap model (OPIK_EXAMPLES_MODEL). Blocks the PR on failure. - scheduled-examples.yml: weekly curated run (Monday 06:00 UTC) across Python 3.12 and 3.13. Example list lives in .github/ci-examples.json and is easy to edit. Also supports workflow_dispatch. - example-compliance-check.yml: secrets-free check on new example folders — enforces run_examples.sh with set -e and OPIK_PROJECT_NAME, pyproject.toml (no requirements.txt), README.md, and OPIK_EXAMPLES_MODEL if litellm is a dependency. Updates CONTRIBUTING.md to document the run_examples.sh convention, the litellm/OPIK_EXAMPLES_MODEL pattern for LLM-calling examples, and CI credential expectations. Required GitHub secrets/variables before workflows go live: Secrets: OPIK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY Variables: OPIK_WORKSPACE, OPIK_EXAMPLES_MODEL, OPIK_ENVIRONMENT Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 3bfe126 commit ed5cc4d

7 files changed

Lines changed: 249 additions & 141 deletions

File tree

‎.github/ci-examples.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
[
2+
"use-cases/f1_radio_rag",
3+
"integrations/otel/distributed_tracing",
4+
"integrations/otel/offline_evaluation",
5+
"scripts/automate_annotation_queue",
6+
"use-cases/governance_observability"
7+
]

‎.github/workflows/e2e-tests.yml‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

‎.github/workflows/example-compliance-check.yml‎

Lines changed: 65 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,23 @@
11
name: Example Compliance Check
22

3-
# Verifies that any new or modified example folder in a PR satisfies the contribution requirements.
3+
# Verifies that new example folders added in a PR follow repo conventions.
4+
# Only checks folders that are new (not previously on the base branch).
45
# Rules enforced:
5-
# 1. run_examples.sh must exist
6+
# 1. run_examples.sh must exist, have set -e, and export OPIK_PROJECT_NAME
67
# 2. README.md must exist
7-
# 3. At least one dependency manifest must exist (requirements.txt, pyproject.toml, or uv.lock)
8-
# 4. run_examples.sh should contain "set -e" (warning, not a hard failure)
8+
# 3. pyproject.toml must exist; requirements.txt must not
9+
# 4. If litellm is a dependency, OPIK_EXAMPLES_MODEL must be referenced in a .py file
910
#
1011
# This check is secrets-free so it runs safely on PRs from forks.
1112

1213
on:
1314
pull_request:
1415
paths:
15-
- "code-snippets/**"
1616
- "examples/**"
17+
- "integrations/**"
18+
- "scripts/**"
19+
- "use-cases/**"
20+
- "guides/**"
1721

1822
permissions:
1923
contents: read
@@ -33,11 +37,19 @@ jobs:
3337
3438
folders=""
3539
while IFS= read -r file; do
36-
if [[ "$file" == code-snippets/* ]] || [[ "$file" == examples/* ]]; then
37-
# Extract the two-segment path (e.g. examples/foo or code-snippets/bar)
38-
parent=$(echo "$file" | cut -d'/' -f1,2)
39-
folders="$folders $parent"
40-
fi
40+
top=$(echo "$file" | cut -d'/' -f1)
41+
case "$top" in
42+
integrations)
43+
parent=$(echo "$file" | cut -d'/' -f1,2,3)
44+
;;
45+
examples|scripts|use-cases|guides)
46+
parent=$(echo "$file" | cut -d'/' -f1,2)
47+
;;
48+
*)
49+
continue
50+
;;
51+
esac
52+
folders="$folders $parent"
4153
done <<< "$changed_files"
4254
4355
# Deduplicate and trim
@@ -62,6 +74,13 @@ jobs:
6274
continue
6375
fi
6476
77+
# Only enforce on NEW folders (not yet on the base branch).
78+
# Modifying an existing example does not require adding run_examples.sh.
79+
if git ls-tree "origin/${{ github.base_ref }}" "$folder" 2>/dev/null | grep -q .; then
80+
echo " Skipping '$folder' (existing folder — compliance enforced on new examples only)"
81+
continue
82+
fi
83+
6584
echo ""
6685
echo "Checking: $folder"
6786
@@ -79,17 +98,43 @@ jobs:
7998
echo " OK README.md"
8099
fi
81100
82-
if [[ ! -f "$folder/requirements.txt" ]] && \
83-
[[ ! -f "$folder/pyproject.toml" ]] && \
84-
[[ ! -f "$folder/uv.lock" ]]; then
85-
echo "::error::$folder is missing a dependency manifest (requirements.txt, pyproject.toml, or uv.lock)"
101+
if [[ ! -f "$folder/pyproject.toml" ]]; then
102+
echo "::error::$folder is missing pyproject.toml (uv projects only — no requirements.txt)"
86103
FAILED=1
87104
else
88-
echo " OK dependency manifest"
105+
echo " OK pyproject.toml"
106+
fi
107+
108+
if [[ -f "$folder/requirements.txt" ]]; then
109+
echo "::error file=$folder/requirements.txt::$folder has requirements.txt — declare deps in pyproject.toml instead"
110+
FAILED=1
111+
fi
112+
113+
if [[ -f "$folder/run_examples.sh" ]]; then
114+
if grep -q "export OPIK_PROJECT_NAME" "$folder/run_examples.sh"; then
115+
echo " OK OPIK_PROJECT_NAME in run_examples.sh"
116+
else
117+
echo "::error file=$folder/run_examples.sh::$folder/run_examples.sh must export OPIK_PROJECT_NAME"
118+
FAILED=1
119+
fi
120+
121+
if ! grep -q "set -e" "$folder/run_examples.sh"; then
122+
echo "::error file=$folder/run_examples.sh::$folder/run_examples.sh must contain 'set -e'"
123+
FAILED=1
124+
else
125+
echo " OK set -e in run_examples.sh"
126+
fi
89127
fi
90128
91-
if [[ -f "$folder/run_examples.sh" ]] && ! grep -q "set -e" "$folder/run_examples.sh"; then
92-
echo "::warning file=$folder/run_examples.sh::$folder/run_examples.sh does not contain 'set -e' — failures may go undetected"
129+
# If litellm is declared as a dependency, OPIK_EXAMPLES_MODEL must be referenced.
130+
if grep -q "litellm" "$folder/pyproject.toml" 2>/dev/null; then
131+
if grep -rq "OPIK_EXAMPLES_MODEL" "$folder" --include="*.py" 2>/dev/null; then
132+
echo " OK OPIK_EXAMPLES_MODEL referenced (litellm dep detected)"
133+
else
134+
echo "::error::$folder uses litellm but no .py file references OPIK_EXAMPLES_MODEL"
135+
echo "::error::Add: os.environ.get(\"OPIK_EXAMPLES_MODEL\", \"<default-model>\") to your config"
136+
FAILED=1
137+
fi
93138
fi
94139
done
95140
@@ -109,7 +154,7 @@ jobs:
109154
echo "Folders checked: \`${{ steps.changed-folders.outputs.folders }}\`" >> "$GITHUB_STEP_SUMMARY"
110155
echo "" >> "$GITHUB_STEP_SUMMARY"
111156
echo "**Rules:**" >> "$GITHUB_STEP_SUMMARY"
112-
echo "- \`run_examples.sh\` must exist" >> "$GITHUB_STEP_SUMMARY"
157+
echo "- \`run_examples.sh\` must exist, contain \`set -e\`, and export \`OPIK_PROJECT_NAME\`" >> "$GITHUB_STEP_SUMMARY"
113158
echo "- \`README.md\` must exist" >> "$GITHUB_STEP_SUMMARY"
114-
echo "- At least one of \`requirements.txt\`, \`pyproject.toml\`, or \`uv.lock\` must exist" >> "$GITHUB_STEP_SUMMARY"
115-
echo "- \`run_examples.sh\` should contain \`set -e\`" >> "$GITHUB_STEP_SUMMARY"
159+
echo "- \`pyproject.toml\` must exist; \`requirements.txt\` is not allowed" >> "$GITHUB_STEP_SUMMARY"
160+
echo "- If \`litellm\` is a dependency, a \`.py\` file must reference \`OPIK_EXAMPLES_MODEL\`" >> "$GITHUB_STEP_SUMMARY"

‎.github/workflows/pr-examples-test.yml‎

Lines changed: 45 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
name: PR Example Tests
22

3-
# Runs only the example folders that changed in this PR, using a single Python version.
4-
# This gives fast feedback without running the full cross-version matrix.
5-
# The full matrix (all Python versions, all examples) runs on the daily schedule via e2e-tests.yml.
3+
# Runs only the example folders changed in this PR using a cheap model and real Opik
4+
# credentials. A passing run means the code executes correctly and traces are visible
5+
# in the opik-examples workspace. Failures block the PR.
66

77
on:
88
pull_request:
99
paths:
10-
- "code-snippets/**"
1110
- "examples/**"
11+
- "integrations/**"
12+
- "scripts/**"
13+
- "use-cases/**"
14+
- "guides/**"
1215

1316
permissions:
1417
contents: read
@@ -23,6 +26,7 @@ jobs:
2326
- uses: actions/checkout@v4
2427
with:
2528
fetch-depth: 0
29+
2630
- name: Detect changed runnable example folders
2731
id: detect
2832
run: |
@@ -32,30 +36,32 @@ jobs:
3236
folders_json="["
3337
3438
while IFS= read -r file; do
35-
if [[ "$file" == code-snippets/* ]] || [[ "$file" == examples/* ]]; then
36-
folder=$(echo "$file" | cut -d'/' -f1,2)
37-
38-
# Only include folders that exist and have run_examples.sh
39-
if [[ ! -d "$folder" ]] || [[ ! -f "$folder/run_examples.sh" ]]; then
39+
top=$(echo "$file" | cut -d'/' -f1)
40+
case "$top" in
41+
integrations)
42+
folder=$(echo "$file" | cut -d'/' -f1,2,3)
43+
;;
44+
examples|scripts|use-cases|guides)
45+
folder=$(echo "$file" | cut -d'/' -f1,2)
46+
;;
47+
*)
4048
continue
41-
fi
49+
;;
50+
esac
51+
52+
if [[ -z "$folder" ]] || [[ ! -d "$folder" ]] || [[ ! -f "$folder/run_examples.sh" ]]; then
53+
continue
54+
fi
4255
43-
# Deduplicate
44-
already_seen=false
45-
for s in "${seen[@]}"; do
46-
if [[ "$s" == "$folder" ]]; then
47-
already_seen=true
48-
break
49-
fi
50-
done
56+
already_seen=false
57+
for s in "${seen[@]}"; do
58+
[[ "$s" == "$folder" ]] && already_seen=true && break
59+
done
5160
52-
if [[ "$already_seen" == false ]]; then
53-
seen+=("$folder")
54-
if [[ "${#seen[@]}" -gt 1 ]]; then
55-
folders_json="$folders_json,"
56-
fi
57-
folders_json="$folders_json\"$folder\""
58-
fi
61+
if [[ "$already_seen" == false ]]; then
62+
seen+=("$folder")
63+
[[ "${#seen[@]}" -gt 1 ]] && folders_json="$folders_json,"
64+
folders_json="$folders_json\"$folder\""
5965
fi
6066
done <<< "$changed_files"
6167
@@ -75,32 +81,27 @@ jobs:
7581
needs: detect-changes
7682
if: needs.detect-changes.outputs.has_changes == 'true'
7783
runs-on: ubuntu-latest
78-
env:
79-
PYTHONIOENCODING: utf-8
8084
strategy:
8185
fail-fast: false
8286
matrix:
83-
scripts: ${{ fromJson(needs.detect-changes.outputs.folders) }}
84-
python-version: ["3.11"]
87+
folder: ${{ fromJson(needs.detect-changes.outputs.folders) }}
8588
steps:
8689
- uses: actions/checkout@v4
87-
- uses: actions/setup-python@v5
90+
91+
- uses: astral-sh/setup-uv@v5
8892
with:
89-
python-version: ${{ matrix.python-version }}
90-
cache: "pip"
91-
- name: Upgrade pip
92-
run: python -m pip install --upgrade pip
93-
- name: Test changed example
94-
uses: nick-fields/retry@v3
93+
version: "latest"
94+
enable-cache: true
95+
python-version: "3.12"
96+
97+
- name: Run example
9598
env:
9699
OPIK_API_KEY: ${{ secrets.OPIK_API_KEY }}
97100
OPIK_WORKSPACE: ${{ vars.OPIK_WORKSPACE }}
98-
OPIK_BASE_URL: "https://www.comet.com"
99-
OPIK_OTLP_ENDPOINT: "https://www.comet.com/opik/api/v1/private/otel/v1/traces"
101+
OPIK_ENVIRONMENT: ${{ vars.OPIK_ENVIRONMENT }}
102+
OPIK_EXAMPLES_MODEL: ${{ vars.OPIK_EXAMPLES_MODEL }}
100103
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
101-
with:
102-
timeout_minutes: 30
103-
max_attempts: 2
104-
command: |
105-
cd ${{ matrix.scripts }}
106-
bash run_examples.sh
104+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
105+
run: |
106+
cd ${{ matrix.folder }}
107+
bash run_examples.sh

‎.github/workflows/scheduled-examples.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Scheduled Example Tests
2+
3+
# Runs a curated set of examples on a schedule to verify they stay working.
4+
# The list of examples is in .github/ci-examples.json.
5+
#
6+
# To change the schedule: edit the cron expression below.
7+
# To add or remove examples: edit .github/ci-examples.json.
8+
9+
on:
10+
schedule:
11+
# Every Monday at 06:00 UTC. Edit this line to change the cadence.
12+
- cron: "0 6 * * 1"
13+
workflow_dispatch:
14+
15+
permissions:
16+
contents: read
17+
18+
jobs:
19+
load-examples:
20+
runs-on: ubuntu-latest
21+
outputs:
22+
folders: ${{ steps.load.outputs.folders }}
23+
steps:
24+
- uses: actions/checkout@v4
25+
26+
- name: Load curated example list
27+
id: load
28+
run: |
29+
folders=$(cat .github/ci-examples.json)
30+
echo "folders=$folders" >> "$GITHUB_OUTPUT"
31+
32+
run-examples:
33+
needs: load-examples
34+
runs-on: ubuntu-latest
35+
strategy:
36+
fail-fast: false
37+
matrix:
38+
folder: ${{ fromJson(needs.load-examples.outputs.folders) }}
39+
python-version: ["3.12", "3.13"]
40+
steps:
41+
- uses: actions/checkout@v4
42+
43+
- uses: astral-sh/setup-uv@v5
44+
with:
45+
version: "latest"
46+
enable-cache: true
47+
python-version: ${{ matrix.python-version }}
48+
49+
- name: Run example
50+
env:
51+
OPIK_API_KEY: ${{ secrets.OPIK_API_KEY }}
52+
OPIK_WORKSPACE: ${{ vars.OPIK_WORKSPACE }}
53+
OPIK_ENVIRONMENT: ${{ vars.OPIK_ENVIRONMENT }}
54+
OPIK_EXAMPLES_MODEL: ${{ vars.OPIK_EXAMPLES_MODEL }}
55+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
56+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
57+
run: |
58+
cd ${{ matrix.folder }}
59+
bash run_examples.sh

0 commit comments

Comments
 (0)