ci: add PR test, scheduled runs, and compliance check workflows

.claude/skills/scaffold-example/SKILL.md

@@ -87,11 +87,14 @@ uv run <command> eval        # use-case: [DRY RUN] lists the dataset items + ass
 uv run <command> optimize    # use-case: [DRY RUN] line
 uv run <command> --dry-run   # script: [DRY RUN] line
 uv run ruff check .          # clean
+bash run.sh                  # the exact dry-run smoke test CI runs (exits 0 without creds)
 ```
 
-With credentials set (`ANTHROPIC_API_KEY`, `OPIK_API_KEY`, `OPIK_WORKSPACE`), the same commands
-talk to Claude + Opik and `eval` prints a pass rate + experiment URL. Running the eval and
-optimization workflows in depth is covered by the `run-evals` and `run-optimizations` skills.
+The scaffold includes a working `run.sh` at the project root — this is what the CI `dry-run` and
+`live-run` jobs execute. With credentials set (`ANTHROPIC_API_KEY`, `OPIK_API_KEY`,
+`OPIK_WORKSPACE`), the same commands talk to Claude + Opik and `eval` prints a pass rate +
+experiment URL. Running the eval and optimization workflows in depth is covered by the `run-evals`
+and `run-optimizations` skills.
 
 ## What this skill does NOT do
 

.claude/skills/scaffold-example/scripts/scaffold.py

@@ -150,6 +150,14 @@ def main() -> int:
         text = text.replace(f'{kebab} = "{entry}"', f'{command} = "{entry}"')
         pyproject.write_text(text, encoding="utf-8")
 
+        # run.sh invokes the CLI by the kebab name (from the sentinel rewrite); point it
+        # at the overridden command so `bash run.sh` matches [project.scripts].
+        run_sh = dest / "run.sh"
+        if run_sh.is_file():
+            text = run_sh.read_text(encoding="utf-8")
+            text = text.replace(f"uv run {kebab}", f"uv run {command}")
+            run_sh.write_text(text, encoding="utf-8")
+
     # Description.
     if args.description:
         pyproject = dest / "pyproject.toml"
@@ -173,11 +181,11 @@ def main() -> int:
     if kind == "package":
         print(f"  2. uv run {command} eval        # DRY_RUN smoke test (no creds needed)")
         print("  3. Fill the TODO stubs: app.py (real logic), prompts.py, data/cases.json, README.md")
-        print(f"  4. uv run ruff check . && uv run {command} run-all")
+        print("  4. uv run ruff check . && bash run.sh   # what CI runs (dry-run without creds)")
     else:
         print(f"  2. uv run {command} --dry-run   # DRY_RUN smoke test (no creds needed)")
         print(f"  3. Fill the TODO in {pkg}.py (real logic) and the README.md sections")
-        print("  4. uv run ruff check .")
+        print("  4. uv run ruff check . && bash run.sh   # what CI runs (dry-run without creds)")
     return 0
 
 

.github/CODEOWNERS

@@ -0,0 +1,6 @@
+# Maintainer-owned CI + automation. Edits to these paths require a maintainer review.
+# The scheduled-run curation list and the workflows are intentionally not contributor-editable.
+/.github/workflows/      @fschlz @LeoRoccoBreedt
+/.github/scheduled.json  @fschlz @LeoRoccoBreedt
+/.github/CODEOWNERS      @fschlz @LeoRoccoBreedt
+/.github/dependabot.yml  @fschlz @LeoRoccoBreedt

.github/dependabot.yml

@@ -0,0 +1,11 @@
+version: 2
+updates:
+  # Keep the GitHub Actions pinned in .github/workflows/ (checkout, setup-uv, …) current,
+  # so a "stay-working" examples repo doesn't silently drift onto stale/abandoned action versions.
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+    commit-message:
+      prefix: "ci"

.github/pull_request_template.md

@@ -0,0 +1,18 @@
+## What & why
+
+<!-- What does this PR change, and why? Link any related issue. -->
+
+## Checklist
+
+<!-- See CONTRIBUTING.md for details. Tick what applies; delete rows that don't. -->
+
+- [ ] Example is in the right bucket (`integrations` / `guides` / `use-cases` / `scripts`)
+- [ ] Folder name is `lowercase_with_underscores`
+- [ ] `README.md` has all required sections; index tables updated if examples were added/renamed/removed
+- [ ] Dry-run works with no credentials — `bash run.sh` exits cleanly (this is what CI's secrets-free job runs)
+- [ ] `uv run ruff check .` and `uv run ruff format --check .` are clean
+- [ ] No credentials or `.env` files committed
+- [ ] Dependencies declared in `pyproject.toml` (uv project); no `requirements.txt`, no committed `uv.lock`
+- [ ] `run.sh` exists and starts with `set -e`
+- [ ] `OPIK_PROJECT_NAME` is set — exported in `run.sh` (scripts) or defined in `config.py` (use-cases/guides)
+- [ ] Examples that call LLMs use litellm and read `OPIK_EXAMPLES_MODEL`

.github/scheduled.json

@@ -0,0 +1 @@
+[]

.github/workflows/compliance.yml

@@ -0,0 +1,171 @@
+name: Example Compliance Check
+
+# Verifies that new example folders added in a PR follow repo conventions.
+# Only checks folders that are new (not previously on the base branch).
+# Rules enforced:
+#   1. run.sh must exist and contain set -e
+#   2. OPIK_PROJECT_NAME must be set — either exported in run.sh (scripts)
+#      or referenced in a .py file (use-cases/guides with a config module)
+#   3. README.md must exist
+#   4. pyproject.toml must exist; requirements.txt must not
+#   5. If litellm is a dependency, OPIK_EXAMPLES_MODEL must be referenced in a .py file
+#
+# This check is secrets-free so it runs safely on PRs from forks.
+
+on:
+  pull_request:
+    paths:
+      - "examples/**"
+      - "integrations/**"
+      - "scripts/**"
+      - "use-cases/**"
+      - "guides/**"
+
+permissions:
+  contents: read
+
+jobs:
+  compliance:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Find changed example folders
+        id: changed-folders
+        run: |
+          changed_files=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD")
+
+          folders=""
+          while IFS= read -r file; do
+            top=$(echo "$file" | cut -d'/' -f1)
+            case "$top" in
+              integrations)
+                parent=$(echo "$file" | cut -d'/' -f1,2,3)
+                ;;
+              examples|scripts|use-cases|guides)
+                parent=$(echo "$file" | cut -d'/' -f1,2)
+                ;;
+              *)
+                continue
+                ;;
+            esac
+            folders="$folders $parent"
+          done <<< "$changed_files"
+
+          # Deduplicate and trim
+          folders=$(echo "$folders" | tr ' ' '\n' | sort -u | grep -v '^$' | tr '\n' ' ' | xargs)
+          echo "folders=$folders" >> "$GITHUB_OUTPUT"
+
+          if [[ -z "$folders" ]]; then
+            echo "No example folders changed."
+          else
+            echo "Changed example folders: $folders"
+          fi
+
+      - name: Check compliance
+        if: steps.changed-folders.outputs.folders != ''
+        run: |
+          FAILED=0
+
+          for folder in ${{ steps.changed-folders.outputs.folders }}; do
+            # Skip if the folder was deleted entirely
+            if [[ ! -d "$folder" ]]; then
+              echo "Skipping deleted folder: $folder"
+              continue
+            fi
+
+            # Only enforce on NEW folders (not yet on the base branch).
+            # Modifying an existing example does not require adding run.sh.
+            if git ls-tree "origin/${{ github.base_ref }}" "$folder" 2>/dev/null | grep -q .; then
+              echo "  Skipping '$folder' (existing folder — compliance enforced on new examples only)"
+              continue
+            fi
+
+            echo ""
+            echo "Checking: $folder"
+
+            if [[ ! -f "$folder/run.sh" ]]; then
+              echo "::error file=$folder/run.sh::$folder is missing run.sh"
+              FAILED=1
+            else
+              echo "  OK  run.sh"
+            fi
+
+            if [[ ! -f "$folder/README.md" ]]; then
+              echo "::error file=$folder/README.md::$folder is missing README.md"
+              FAILED=1
+            else
+              echo "  OK  README.md"
+            fi
+
+            if [[ ! -f "$folder/pyproject.toml" ]]; then
+              echo "::error::$folder is missing pyproject.toml (uv projects only — no requirements.txt)"
+              FAILED=1
+            else
+              echo "  OK  pyproject.toml"
+            fi
+
+            if [[ -f "$folder/requirements.txt" ]]; then
+              echo "::error file=$folder/requirements.txt::$folder has requirements.txt — declare deps in pyproject.toml instead"
+              FAILED=1
+            fi
+
+            if [[ -f "$folder/run.sh" ]]; then
+              if ! grep -q "set -e" "$folder/run.sh"; then
+                echo "::error file=$folder/run.sh::$folder/run.sh must contain 'set -e'"
+                FAILED=1
+              else
+                echo "  OK  set -e in run.sh"
+              fi
+
+              # OPIK_PROJECT_NAME may be exported in run.sh (scripts)
+              # or defined as a constant in a .py file (use-cases/guides with config.py).
+              if grep -q "export OPIK_PROJECT_NAME" "$folder/run.sh"; then
+                echo "  OK  OPIK_PROJECT_NAME in run.sh"
+              elif grep -rq "OPIK_PROJECT_NAME" "$folder" --include="*.py" 2>/dev/null; then
+                echo "  OK  OPIK_PROJECT_NAME in .py config"
+              else
+                echo "::error::$folder must set OPIK_PROJECT_NAME — either export it in run.sh or define it in a .py config file"
+                FAILED=1
+              fi
+            fi
+
+            # If litellm is declared as a dependency, OPIK_EXAMPLES_MODEL must be referenced.
+            # Match a quoted dependency entry ("litellm", "litellm>=...", "litellm[extra]")
+            # so a bare mention in a comment doesn't trip the check.
+            if grep -qE '"litellm' "$folder/pyproject.toml" 2>/dev/null; then
+              if grep -rq "OPIK_EXAMPLES_MODEL" "$folder" --include="*.py" 2>/dev/null; then
+                echo "  OK  OPIK_EXAMPLES_MODEL referenced (litellm dep detected)"
+              else
+                echo "::error::$folder uses litellm but no .py file references OPIK_EXAMPLES_MODEL"
+                echo "::error::Add: os.environ.get(\"OPIK_EXAMPLES_MODEL\", \"<default-model>\") to your config"
+                FAILED=1
+              fi
+            fi
+          done
+
+          echo ""
+          if [[ "$FAILED" -ne 0 ]]; then
+            echo "Compliance check failed. Fix the errors above before merging."
+            exit 1
+          fi
+
+          echo "All compliance checks passed."
+
+      - name: Write step summary
+        if: always() && steps.changed-folders.outputs.folders != ''
+        run: |
+          {
+            echo "## Example Compliance Check"
+            echo ""
+            echo "Folders checked: \`${{ steps.changed-folders.outputs.folders }}\`"
+            echo ""
+            echo "**Rules:**"
+            echo "- \`run.sh\` must exist and contain \`set -e\`"
+            echo "- \`OPIK_PROJECT_NAME\` must be set — via \`export\` in \`run.sh\` (scripts) or as a constant in a \`.py\` config file (use-cases/guides)"
+            echo "- \`README.md\` must exist"
+            echo "- \`pyproject.toml\` must exist; \`requirements.txt\` is not allowed"
+            echo "- If \`litellm\` is a dependency, a \`.py\` file must reference \`OPIK_EXAMPLES_MODEL\`"
+          } >> "$GITHUB_STEP_SUMMARY"