Skip to content

Commit 3db0717

Browse files
Merge branch 'main' into LeoRoccoBreedt/multimodal-online-evaluation
2 parents 4def564 + 719ba37 commit 3db0717

26 files changed

Lines changed: 1819 additions & 206 deletions
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: Community Entry Check
2+
3+
# Lightweight, non-executing gate for community showcase entries.
4+
# It does NOT run contributor code. It validates entry structure, scans for
5+
# committed secrets, and confirms hosted entries use Opik. Secrets-free, so it
6+
# runs on forks. The showcase index (community/README.md) is regenerated
7+
# separately on merge by community-index.yml — contributors run nothing.
8+
#
9+
# Community entries are intentionally exempt from the verified-bucket workflows
10+
# (compliance / pr-test / test-notebooks / scheduled), which are path-scoped to
11+
# examples|integrations|scripts|use-cases|guides and never match community/**.
12+
13+
on:
14+
pull_request:
15+
paths:
16+
- "community/**"
17+
18+
permissions:
19+
contents: read
20+
21+
jobs:
22+
check:
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 10
25+
steps:
26+
- uses: actions/checkout@v7
27+
28+
- uses: astral-sh/setup-uv@v7
29+
with:
30+
version: "latest"
31+
enable-cache: true
32+
python-version: "3.12"
33+
34+
- name: Install tooling
35+
run: |
36+
cd community/_ci
37+
uv sync
38+
39+
- name: Unit-test the check tooling
40+
run: |
41+
cd community/_ci
42+
uv run pytest -q
43+
44+
- name: Lint the check tooling
45+
run: |
46+
cd community/_ci
47+
uv run ruff check .
48+
uv run ruff format --check .
49+
50+
- name: Validate all community entries
51+
run: |
52+
cd community/_ci
53+
uv run python check_entry.py
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: Community Index
2+
3+
# Regenerates the community showcase index (community/README.md) from every
4+
# entry's meta.yaml after changes land on main. This keeps the index current
5+
# with zero manual action — contributors never run the generator, and no one
6+
# hand-edits community/README.md.
7+
#
8+
# It runs build_index.py (which only parses YAML — it never executes contributor
9+
# code) and commits the result back to main only if it changed. The path filter
10+
# excludes community/README.md so this workflow's own index commit does not
11+
# re-trigger it.
12+
13+
on:
14+
push:
15+
branches: [main]
16+
paths:
17+
- "community/**"
18+
- "!community/README.md"
19+
20+
permissions:
21+
contents: write
22+
23+
concurrency:
24+
group: community-index
25+
cancel-in-progress: false
26+
27+
jobs:
28+
regenerate:
29+
runs-on: ubuntu-latest
30+
timeout-minutes: 10
31+
steps:
32+
- uses: actions/checkout@v7
33+
34+
- uses: astral-sh/setup-uv@v7
35+
with:
36+
version: "latest"
37+
enable-cache: true
38+
python-version: "3.12"
39+
40+
- name: Regenerate showcase index
41+
run: |
42+
cd community/_ci
43+
uv sync
44+
uv run python build_index.py
45+
46+
- name: Commit index if it changed
47+
run: |
48+
if [[ -n "$(git status --porcelain community/README.md)" ]]; then
49+
git config user.name "github-actions[bot]"
50+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
51+
git add community/README.md
52+
git commit -m "chore: regenerate community showcase index"
53+
git push
54+
else
55+
echo "Showcase index already up to date."
56+
fi

.github/workflows/test-notebooks.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ jobs:
124124
python-version: "3.12"
125125

126126
- name: Execute notebooks (live, real Opik credentials)
127-
uses: nick-fields/retry@v3
127+
uses: nick-fields/retry@v4
128128
env:
129129
OPIK_API_KEY: ${{ secrets.OPIK_API_KEY }}
130130
OPIK_WORKSPACE: ${{ vars.OPIK_WORKSPACE }}

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,6 @@ poetry.lock
2626

2727
# Local planning/process artifacts (superpowers brainstorm specs & implementation plans) — kept local, never committed
2828
docs/superpowers/
29+
30+
# Community proof screenshots are intentionally committed (overrides the global *.png rule above)
31+
!community/**/*.png

CONTRIBUTING.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
Thank you for contributing. This repo is a reference library for Opik users — the goal is examples that are easy to find, easy to run, and easy to adapt.
44

5+
> **Contributing community showcase work instead of a verified example?** The
6+
> `community/` folder has its own, lighter-weight contract — see
7+
> [community/CONTRIBUTING.md](community/CONTRIBUTING.md). The rules below apply
8+
> to the verified `integrations/`, `guides/`, `use-cases/`, and `scripts/`
9+
> buckets.
10+
511
## Recommended workflow
612

713
This is the loop we follow for non-trivial contributions. The slash-commands in brackets come from Claude Code plugins (see below) and are optional but recommended. **Start every contribution with `/brainstorming`** to agree on scope, and **finish by reviewing your own PR with `/review`** before asking a human.

README.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,16 @@ Standalone scripts for automating and managing Opik resources.
5757
| [scripts/usage_stats](scripts/usage_stats/) | Fetch trace and span counts per project and visualise trends |
5858
| [scripts/leaderboard_dashboard](scripts/leaderboard_dashboard/) | Create an Experiment Leaderboard dashboard via the REST API |
5959

60+
## Community
61+
62+
Work built with Opik by the open-source community. Unlike the buckets above,
63+
these entries are **community-contributed and not maintainer-verified** — they
64+
showcase what people are building. Standout real-world projects get hosted
65+
in-repo and spotlighted in our community forums.
66+
67+
Browse them in [community/](community/), or add your own via
68+
[community/CONTRIBUTING.md](community/CONTRIBUTING.md).
69+
6070
## Contributing
6171

6272
See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution guide and example template.

community/CONTRIBUTING.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Contributing to the Community folder
2+
3+
This folder showcases work the open-source community has built with Opik. It is
4+
deliberately **lighter-weight than the main repo contract** — the strict
5+
`run.sh` / dry-run / litellm / CI rules in the root
6+
[CONTRIBUTING.md](../CONTRIBUTING.md) do **not** apply here. Community entries
7+
are not executed by CI; a maintainer reviews (and, for hosted entries, runs)
8+
them by hand.
9+
10+
There is one thing we always require: **proof you actually logged with Opik**
11+
either Comet cloud or the self-hosted open-source platform.
12+
13+
## Two kinds of entry
14+
15+
- **Listed** (default): a folder describing your work with links out to your own
16+
repo/blog/notebook. No code needs to live here.
17+
- **Hosted**: standout, real-world projects we promote into this repo with their
18+
code included. You submit as *listed*; a maintainer sets `hosted: true` and
19+
moves your code in when promoting. We also spotlight promoted work in our
20+
community forums.
21+
22+
## Add your entry
23+
24+
1. Copy `templates/entry-template/` to `community/<your-handle>_<project>/`
25+
(lowercase, underscores, e.g. `jane_support_agent`).
26+
2. Fill in `meta.yaml` (all fields) and `README.md` (all four sections).
27+
3. Replace `opik-proof.png` with a real screenshot of your Opik traces or
28+
dashboard.
29+
4. If you want it considered for hosting, include your code in the folder — it
30+
must genuinely use Opik (`import opik`, `@opik.track`, ...).
31+
5. Open a PR. A maintainer reviews it.
32+
33+
That's it — you don't need to run anything. The showcase index
34+
(`community/README.md`) is regenerated automatically when your entry merges, so
35+
your project appears there without any manual step on your part.
36+
37+
## What the automated check enforces
38+
39+
`community/_ci/check_entry.py` runs on your PR (a hard gate). It does **not**
40+
run your code. It checks:
41+
42+
- `meta.yaml` has all required fields, at least one link, and a valid
43+
`opik_platform`.
44+
- `README.md` has all four sections filled in (no leftover `TODO`).
45+
- `opik-proof.png` exists and is referenced from your README.
46+
- No `.env` file or hardcoded API keys are committed.
47+
- Hosted entries contain code that uses Opik.
48+
- The folder name is `lowercase_with_underscores`.

community/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Community
2+
3+
<!-- This file is generated by community/_ci/build_index.py. Do not edit by hand. -->
4+
5+
Work built with Opik by the open-source community. These entries are
6+
**community-contributed and not maintainer-verified** — they are showcased here
7+
to help people discover what others are building. Standout, real-world projects
8+
are hosted in-repo (see the Hosted column).
9+
10+
To add your own, see [CONTRIBUTING.md](CONTRIBUTING.md).
11+
12+
_No community contributions yet — be the first!_

community/_ci/build_index.py

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
from __future__ import annotations
2+
3+
import sys
4+
from pathlib import Path
5+
6+
from check_entry import COMMUNITY_DIR, discover_entries
7+
from entry_rules import load_meta
8+
9+
_HEADER = """\
10+
# Community
11+
12+
<!-- This file is generated by community/_ci/build_index.py. Do not edit by hand. -->
13+
14+
Work built with Opik by the open-source community. These entries are
15+
**community-contributed and not maintainer-verified** — they are showcased here
16+
to help people discover what others are building. Standout, real-world projects
17+
are hosted in-repo (see the Hosted column).
18+
19+
To add your own, see [CONTRIBUTING.md](CONTRIBUTING.md).
20+
"""
21+
22+
_TABLE_HEADER = (
23+
"| Project | Author | Description | Platform | Links | Hosted | Tags |\n"
24+
"|---|---|---|---|---|---|---|"
25+
)
26+
27+
28+
def load_entries(community_dir: Path) -> list[dict]:
29+
entries: list[dict] = []
30+
for entry in discover_entries(community_dir):
31+
data, errors = load_meta(entry)
32+
if errors:
33+
continue
34+
entries.append(
35+
{
36+
"slug": entry.name,
37+
"title": str(data.get("title", entry.name)),
38+
"description": str(data.get("description", "")),
39+
"author": str(data.get("author", "")),
40+
"links": data.get("links") or {},
41+
"opik_platform": str(data.get("opik_platform", "")),
42+
"tags": data.get("tags") or [],
43+
"hosted": bool(data.get("hosted", False)),
44+
}
45+
)
46+
entries.sort(key=lambda e: e["title"].casefold())
47+
return entries
48+
49+
50+
def _escape_pipes(value: str) -> str:
51+
return value.replace("|", "\\|")
52+
53+
54+
def _links_cell(links: dict) -> str:
55+
parts = [
56+
f"[{_escape_pipes(str(label))}]({_escape_pipes(str(url))})"
57+
for label, url in links.items()
58+
if str(url).startswith("http")
59+
]
60+
return "<br>".join(parts) if parts else ""
61+
62+
63+
def _row(entry: dict) -> str:
64+
project = f"[{_escape_pipes(entry['title'])}]({entry['slug']}/)"
65+
author = (
66+
f"[@{_escape_pipes(entry['author'])}](https://github.com/{_escape_pipes(entry['author'])})"
67+
if entry["author"]
68+
else ""
69+
)
70+
tags = _escape_pipes(", ".join(str(t) for t in entry["tags"]))
71+
hosted = "hosted" if entry["hosted"] else ""
72+
cells = [
73+
project,
74+
author,
75+
_escape_pipes(entry["description"]),
76+
_escape_pipes(entry["opik_platform"]),
77+
_links_cell(entry["links"]),
78+
hosted,
79+
tags,
80+
]
81+
return "| " + " | ".join(cells) + " |"
82+
83+
84+
def render_index(entries: list[dict]) -> str:
85+
lines = [_HEADER]
86+
if not entries:
87+
lines.append("_No community contributions yet — be the first!_\n")
88+
else:
89+
lines.append(_TABLE_HEADER)
90+
lines.extend(_row(e) for e in entries)
91+
lines.append("")
92+
return "\n".join(lines).rstrip("\n") + "\n"
93+
94+
95+
def write_index(community_dir: Path) -> None:
96+
(community_dir / "README.md").write_text(
97+
render_index(load_entries(community_dir)), encoding="utf-8"
98+
)
99+
100+
101+
def check_index(community_dir: Path) -> bool:
102+
readme = community_dir / "README.md"
103+
current = readme.read_text(encoding="utf-8") if readme.is_file() else ""
104+
return current == render_index(load_entries(community_dir))
105+
106+
107+
def main(argv: list[str] | None = None) -> int:
108+
args = sys.argv[1:] if argv is None else argv
109+
if "--check" in args:
110+
if check_index(COMMUNITY_DIR):
111+
print("community/README.md is up to date.")
112+
return 0
113+
print("community/README.md is stale. Run: python build_index.py")
114+
return 1
115+
write_index(COMMUNITY_DIR)
116+
print("Wrote community/README.md")
117+
return 0
118+
119+
120+
if __name__ == "__main__":
121+
raise SystemExit(main())

0 commit comments

Comments
 (0)