Lightweight Python-based evaluation harness for AI coding assistants.
- Package Manager: uv
- Install:
make(oruv sync && uv tool install .) - Run Eval:
pitlane run examples/simple-codegen-eval.yaml - Test:
uv run pytest - Pre-commit:
uv run pre-commit run --all-files
- Unit tests (fast, default):
uv run pytest -m "not integration and not e2e" - Unit + integration:
uv run pytest -m "not e2e" - E2E only (on-demand, requires all CLIs):
uv run pytest -m e2e -v --tb=long - E2E single assistant:
uv run pytest -m e2e -v -k claude_code - All tests:
uv run pytest
For specific guidance, see:
- Development Conventions - TDD workflow, code quality, testing
- YAML Configuration - Benchmark structure, assertions
- Assistant Implementation - Creating new assistants
- Architecture & Design - Key decisions, project structure