bench-evals-lab is a meta repository for assorted evaluations and benchmarks. It supports both Python and TypeScript evals.
Each eval or benchmark lives in its own fully self-contained folder: it owns its dependencies, documentation, run command, and committed results.
Current and planned top-level contents:
youtube-transcript-analysis/— seed eval (Gemini model comparison).template-python/— template for Python evals.template-typescript/— template for TypeScript evals.docs/EVAL_TEMPLATE.md— checklist and starting structure for a new eval.
- Every eval folder is self-contained, including its dependency manifest and README.
- Each eval has a documented one-command run path.
- Commit result artifacts under that eval's
results/directory. - Read secrets only from environment variables, such as
GEMINI_API_KEY; never commit secrets or.envfiles.
- Copy the appropriate Python or TypeScript template.
- Rename the copied folder for the eval.
- Fill in its configuration, scripts, README, and one-command run path.
- Add a row for it to the index below.
More evals will be added over time.
| Eval | Language | Status | Notes |
|---|---|---|---|
video-summarization |
Python | Seeded 2026-08-20 | yt-insights stage-1 eval: YouTube video → SummarySchema JSON via native Gemini video ingestion (4 candidates, 3.1-pro fact-sheet + dual LLM-as-judge scoring) |
youtube-transcript-analysis |
Python | Seeded 2026-08-15 | Gemini model comparison (YouTube URL → structured analysis) with dual LLM-as-judge scoring (gemini + deepseek) |
summary-structuring |
Python | Seeded 2026-08-20 | yt-insights stage-2 eval: markdown summary → SummarySchema JSON (4 candidates, dual LLM-as-judge fidelity scoring) |
template-python |
Python | Template | Copy-paste starter for Python evals |
template-typescript |
TypeScript | Template | Copy-paste starter for TS evals |
- Python evals use a virtual environment or
uv, with only the dependencies they need. - TypeScript evals use
npmandtsx; do not add frameworks unless the eval needs them.