ci: add PR test, scheduled runs, and compliance check workflows - #16
Merged
Conversation
Replaces legacy e2e-tests.yml and test-scripts-matrix.yml with three focused workflows: - pr-examples-test.yml: detects changed example folders that have run_examples.sh, runs each in a matrix job using real Opik credentials and a cheap model (OPIK_EXAMPLES_MODEL). Blocks the PR on failure. - scheduled-examples.yml: weekly curated run (Monday 06:00 UTC) across Python 3.12 and 3.13. Example list lives in .github/ci-examples.json and is easy to edit. Also supports workflow_dispatch. - example-compliance-check.yml: secrets-free check on new example folders — enforces run_examples.sh with set -e and OPIK_PROJECT_NAME, pyproject.toml (no requirements.txt), README.md, and OPIK_EXAMPLES_MODEL if litellm is a dependency. Updates CONTRIBUTING.md to document the run_examples.sh convention, the litellm/OPIK_EXAMPLES_MODEL pattern for LLM-calling examples, and CI credential expectations. Required GitHub secrets/variables before workflows go live: Secrets: OPIK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY Variables: OPIK_WORKSPACE, OPIK_EXAMPLES_MODEL, OPIK_ENVIRONMENT Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es.sh export Scripts (single .py, no config module) set it via export in run_examples.sh. Use-cases and guides with a config.py define it as a Python constant and pass it explicitly to opik.track(). The compliance check accepts either pattern. Updates CONTRIBUTING.md and the step summary to document both approaches. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clarifies inclusion criteria (self-contained, cheap, no GPU, no seed data) and that contributors do not need to add their example to the scheduled list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LeoRoccoBreedt
force-pushed
the
lrb/ci-workflows
branch
from
June 25, 2026 16:45
f144b49 to
224ffc7
Compare
Clearer name — each example folder has exactly one run.sh scoped to that folder, so there is no ambiguity about what it runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LeoRoccoBreedt
marked this pull request as ready for review
June 26, 2026 10:38
…on in scheduled runs
LeoRoccoBreedt
force-pushed
the
lrb/ci-workflows
branch
from
June 26, 2026 10:43
e6aff49 to
4148f29
Compare
Address review of #16 (high/medium/low) and make the templates + scaffold consistent with the conventions this PR introduces, so a freshly scaffolded example passes its own compliance check. pr-test.yml: split the single test job into detect-changes (+has_secrets) / lint / dry-run / live-run. lint + dry-run are secrets-free, so fork PRs (which GitHub withholds secrets from) get real signal; live-run is gated on secrets being present. Add a concurrency group + per-job timeouts; quote the matrix folder. scheduled.yml: guard run-examples on a non-empty list — an empty matrix array errored ("Matrix vector 'folder' does not contain any values"), so the weekly cron and workflow_dispatch failed with the empty scheduled.json. Add a timeout; quote the folder. compliance.yml: match a quoted litellm dependency instead of a bare substring; group the step-summary redirects (shellcheck SC2129). actionlint now clean. scheduled.json: add a trailing newline. templates: add run.sh to both templates — it doubles as the secrets-free dry-run smoke test CI runs. use-case config.py now reads OPIK_EXAMPLES_MODEL, so a fresh use-case scaffold satisfies the new litellm compliance rule. scaffold.py rewrites run.sh's command on --command override and no longer prints a phantom "run-all" next step. CONTRIBUTING.md: dry-run is required again (the secrets-free CI job enforces it); fix the "CI always has real Opik credentials" claim (false for forks); fix the run.sh snippet; add ruff check/format to the checklist. .github: add pull_request_template.md (embeds the checklist), CODEOWNERS (workflows + scheduled.json are maintainer-managed), dependabot for actions. Verified: actionlint clean on all three workflows; scaffolded use-case and script projects pass ruff check + format and `bash run.sh` exits 0 in DRY_RUN with no credentials; compliance positive and negative cases behave as expected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fschlz
approved these changes
Jun 30, 2026
fschlz
left a comment
fschlz
left a comment
Collaborator
There was a problem hiding this comment.
looking good overall
I pushed some changes to harden the CI flow
This was referenced Jun 30, 2026
LeoRoccoBreedt
added a commit
that referenced
this pull request
Jul 1, 2026
## Summary Adds CI support to `use-cases/f1_radio_rag` — the first example to go through the new PR test workflow (from PR #16). ## What changed - `run.sh` — runs `ingest` then `ask`, logging one RAG trace to Opik. Lightweight alternative to the full eval suite for CI. - `config.py` — reads `OPIK_EXAMPLES_MODEL` so CI routes to a cheap model (`openai/gpt-4o-mini`). Removes `LLM_READY` guard — missing keys now fail loudly via litellm rather than silently skipping. - `cli.py` — removes the `LLM_READY` dry-run branch in `ask`. ## Tested locally Ran `bash run.sh` with real credentials. Trace appeared in Opik workspace under project `f1-radio-rag`. **Depends on PR #16 merging first** for the PR test workflow to fire. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the legacy `e2e-tests.yml` and `test-scripts-matrix.yml` with three focused workflows and documents the conventions in `CONTRIBUTING.md`. Infrastructure only — no example code changes. A follow-up PR adds `run.sh` to the first example to prove the pipeline end-to-end.
What changed
Deleted
Added / updated
What to review
Tested
The full CI flow was validated end-to-end by retargeting PR #19 (
f1_radio_rag) at this branch and triggering `pr-test.yml`. The matrix job ran successfully: ingested 30 radio messages into ChromaDB, retrieved context, called Claude Haiku, and logged a trace to the `f1-radio-rag` project in Opik.GitHub secrets/variables required before workflows go live
🤖 Generated with Claude Code