Skip to content

Add per-run score fingerprinting for contest reproduction - #283

Open
jaisinha77777 wants to merge 1 commit into
google-deepmind:mainfrom
jaisinha77777:add-run-fingerprinting
Open

Add per-run score fingerprinting for contest reproduction#283
jaisinha77777 wants to merge 1 commit into
google-deepmind:mainfrom
jaisinha77777:add-run-fingerprinting

Conversation

@jaisinha77777

Copy link
Copy Markdown

Summary

Addresses #271.

`run.py` writes a full structured event log per run, but the actual per-agent `focal_scores`/`background_scores` dict is only printed to stdout and then discarded — there's no durable, parseable record of exactly what a completed run scored. That makes the concern raised in #271 concrete: two runs can land on the same aggregate Elo while actually differing per-scenario per-agent, and there's currently no way to tell those cases apart after the fact.

Changes

  • `run.py` now also writes `simulation_scores_{game}_{scenario}.json` alongside the existing HTML/JSON structured log, containing the per-agent focal/background scores plus run identity (game, scenario, model name, api type, UTC timestamp, and the exact command invoked).
  • New `fingerprint_run.py`, a companion script that reads one or more of those score files and computes:
    • `content_hash`: a SHA-256 Merkle root over every `(scenario, role, agent, score)` tuple across the supplied runs. This is what makes drift detectable at a finer grain than aggregate Elo — two runs with the same total score but a different per-agent split, or the same scores under a different scenario, produce different `content_hash` values.
    • `bundle_hash`: `content_hash` combined with each run's identity (model, timestamp, command), so two runs with identical scores but different provenance are distinguishable from runs that are genuinely identical.

Usage

python -m examples.games.run --game=haggling --scenario=fruitville ...
# writes /tmp/simulation_scores_haggling_fruitville.json

python -m examples.games.fingerprint_run \
    --scores /tmp/simulation_scores_*.json \
    --output /tmp/fingerprint.json

Test plan

  • `examples/games/fingerprint_run_test.py`: 19 tests covering Merkle-root determinism/order-independence, sensitivity to per-agent score changes and to scenario changes, the "matching aggregate but different per-agent split" case Contest run provenance: making model drift detectable with per-run output fingerprinting #271 specifically calls out, focal/background collision-avoidance, bundle vs content hash independence from provenance, glob/path resolution, and the CLI end to end (via `main()` with a temp dir, no simulation run required).
  • `python -m pytest examples/games/fingerprint_run_test.py -q` → 19 passed.
  • Manually exercised the CLI end-to-end against a synthetic score file (no LLM/API keys required) and confirmed the output JSON report.
  • `python -m pyink --check` on all changed/new files → clean.

Open questions from the issue

The issue itself left a few questions for maintainers unanswered (whether the original 2024 contest logs exist to compute a reference hash against, and whether this belongs in #265 instead). I didn't block on those — this PR is scoped to the mechanism itself (computing and persisting the fingerprint), which is useful regardless of how those get resolved. Happy to adjust the file layout/integration point based on maintainer feedback.

…pmind#271)

run.py already writes a structured event log per run but discards the
focal/background scores dict after printing it to stdout, so there was
no durable, parseable record of what a completed run actually scored.

Adds a simulation_scores_{game}_{scenario}.json output alongside the
existing HTML/JSON log, capturing per-agent scores plus run identity
(game, scenario, model name, api type, timestamp, command).

Adds fingerprint_run.py, a companion script that reads one or more of
those score files and computes:
- content_hash: a SHA-256 Merkle root over every (scenario, role,
  agent, score) tuple across the supplied runs. Two runs with matching
  aggregate scores but different per-scenario per-agent outcomes will
  not produce the same content_hash, which is what makes model drift
  detectable at a finer grain than an aggregate Elo comparison.
- bundle_hash: content_hash combined with each run's identity, so runs
  with identical scores but different provenance (model, timestamp,
  command) are distinguishable from runs that are truly identical.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant