Skip to content

CLAUDE.md

Latest commit

 

History

History
72 lines (56 loc) · 2.81 KB

CLAUDE.md

File metadata and controls

72 lines (56 loc) · 2.81 KB
Raw

Claude Code Language Benchmark

Overview

Benchmark that has Claude Code (Opus) implement "MiniGit" (a minimal git clone) in multiple languages, comparing generation time, LOC, token usage, and pass rate.

Repository Structure

SPEC-v1.txt          # MiniGit v1 spec (init/add/commit/log)
SPEC-v2.txt          # MiniGit v2 spec (v1 + status/diff/checkout/reset/rm/show)
test-v1.sh           # v1 test suite (11 tests)
test-v2.sh           # v2 test suite (30 tests)
benchmark.rb         # Benchmark runner (Ruby)
report.rb            # Report generator (results.json -> report.md)
plot.py              # Graph generator (results.json -> figures/*.png)
results/
  results.json       # Raw result data
  meta.json          # Environment metadata
  report.md          # Generated report
figures/             # Generated graphs

The data branch (orphan) contains:

generated/           # Source code generated by Claude (binaries removed)
  minigit-{lang}-{trial}-v1/
  minigit-{lang}-{trial}-v2/
logs/                # Claude JSON output logs
  minigit-{lang}-{trial}-v1.json
  minigit-{lang}-{trial}-v2.json

How It Works

  1. Run ruby benchmark.rb
  2. For each language x trial:
    • v1: Create generated/minigit-{lang}-{trial}-v1/, copy spec and tests, invoke claude -p
    • v2: Copy v1 result to minigit-{lang}-{trial}-v2/, invoke claude -p to extend
  3. Run test scripts independently to verify
  4. Measure wall-clock time, LOC, token usage, and cost
  5. Run ruby report.rb to generate the report
  6. Run python3 plot.py to generate graphs

Key Commands

ruby benchmark.rb                                    # All languages x 3 trials
ruby benchmark.rb --lang python --trials 1           # Single language test
ruby benchmark.rb --trials 10 --start 11             # Trials 11-20
ruby benchmark.rb --dry-run                           # Dry run
ruby report.rb                                        # Generate report
python3 plot.py                                       # Generate graphs

Supported Languages (LANGUAGES hash in benchmark.rb)

rust, go, c, typescript, javascript, java, perl, python, python/mypy, ruby, ruby/steep, lua, scheme, ocaml, haskell

To add a language, add an entry to the LANGUAGES hash. Tests just call ./minigit, so the implementation only needs to produce an executable with that name.

MiniGit Technical Notes

  • Custom hash function "MiniHash" (FNV-1a variant, 64-bit, 16-char hex output)
  • Data stored under .minigit/ (objects/, commits/, index, HEAD)
  • No external libraries allowed, stdlib only
  • Exact string matching required (determinism rules)

Notes

  • This is not a git repository for MiniGit itself; individual implementations under generated/ may use git init as part of their build process
  • The data branch is an orphan branch with no common history with main