Add per-run score fingerprinting for contest reproduction - #283
Open
jaisinha77777 wants to merge 1 commit into
Open
Add per-run score fingerprinting for contest reproduction#283jaisinha77777 wants to merge 1 commit into
jaisinha77777 wants to merge 1 commit into
Conversation
…pmind#271) run.py already writes a structured event log per run but discards the focal/background scores dict after printing it to stdout, so there was no durable, parseable record of what a completed run actually scored. Adds a simulation_scores_{game}_{scenario}.json output alongside the existing HTML/JSON log, capturing per-agent scores plus run identity (game, scenario, model name, api type, timestamp, command). Adds fingerprint_run.py, a companion script that reads one or more of those score files and computes: - content_hash: a SHA-256 Merkle root over every (scenario, role, agent, score) tuple across the supplied runs. Two runs with matching aggregate scores but different per-scenario per-agent outcomes will not produce the same content_hash, which is what makes model drift detectable at a finer grain than an aggregate Elo comparison. - bundle_hash: content_hash combined with each run's identity, so runs with identical scores but different provenance (model, timestamp, command) are distinguishable from runs that are truly identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #271.
`run.py` writes a full structured event log per run, but the actual per-agent `focal_scores`/`background_scores` dict is only printed to stdout and then discarded — there's no durable, parseable record of exactly what a completed run scored. That makes the concern raised in #271 concrete: two runs can land on the same aggregate Elo while actually differing per-scenario per-agent, and there's currently no way to tell those cases apart after the fact.
Changes
Usage
Test plan
Open questions from the issue
The issue itself left a few questions for maintainers unanswered (whether the original 2024 contest logs exist to compute a reference hash against, and whether this belongs in #265 instead). I didn't block on those — this PR is scoped to the mechanism itself (computing and persisting the fingerprint), which is useful regardless of how those get resolved. Happy to adjust the file layout/integration point based on maintainer feedback.