Contest evaluation robustness: detecting scenario-version drift and evaluation independence

[topeuph-ai]

Reading through the CHANGELOG, I noticed two things that seem worth raising together ahead of any future contest edition.

---

1. Scenario version drift is currently undetectable (concrete, Project B)

v2.3.0 shipped the NeurIPS 2023 contest scenarios, and v2.4.0 fixed two misconfigured ones (issue #246: wrong puppet bot in clean_up_20, duplicate territory__rooms_6). Both were present during the actual contest evaluation.

There's currently no way to tell, from a leaderboard score alone, whether an evaluation run used the corrected or buggy scenario. Two runs producing identical aggregate scores could have completely different per-episode per-scenario returns — one on the correct substrate, one silently running against the wrong bot.

The evaluation pipeline already produces exactly the right granularity: evaluate_saved_models returns a per-episode DataFrame with focal_player_returns per scenario. A SHA-256 Merkle root over those per-episode per-scenario returns — what I'd call a content_hash — would be different for "ran on v2.3.0 buggy scenario" vs "ran on v2.4.0 fixed scenario", even if the aggregate scores happened to match.

For the next contest edition, requiring submitters to include a content_hash alongside their AICrowd score would make scenario-version drift detectable rather than unknowable. I've been building exactly this format for a different reproducibility context — valichord_attestation (pip-installable, no infrastructure). The adapter for Melting Pot would be ~50 lines reading the evaluation DataFrame.

---

2. Evaluation independence across contest rounds (question, Project A)

This one is more speculative, so I'll frame it as a question.

The NeurIPS 2023 contest involved agents evaluated against background populations on held-out test scenarios. If the contest ran multiple submission rounds with visible intermediate rankings, teams could adapt their training strategy after seeing where competitors stood — the social game at the evaluation layer mirrors the social games the benchmark itself tests.

In multi-agent settings, the independence of evaluation teams matters as much as the independence of the agents. If Team A can observe Team B's intermediate scores before finalising their own submission, Team A can anchor on that signal, and the leaderboard no longer reflects independent performance estimates.

Has this been a concern in practice? The obvious mitigation is requiring teams to commit a sealed agent checkpoint before any team's evaluation results are revealed — a commit-reveal at the submission layer rather than the gameplay layer. We've been building a protocol for exactly this kind of problem in a different domain (ValiChord), and the structural similarity to Melting Pot's contest design struck me.

Happy to prototype the content_hash script (point 1) and share it regardless of interest in the broader question — just wanted to flag both while the next contest edition is (presumably) still being planned.

[duenez]

For (1), all results used the v2.3.0 of Melting Pot. All the contestants saw only those scenarios. The fix only occurred after the fact, because we did not want to pollute the results during the competition. The fix is so that researchers going forward can get the scenarios as they were intended, while allowing for full comparison with previous published results.

For (2), this is not a concern because the results on the leaderboard during the competition were on scenarios that were sampled from the same distribution, but were distinct, from the actual evaluation scenarios for the final score. In other words, we had 3 sets of scenarios: training ones that were public and available to everyone during the competition; test ones that were semi-private and provided scores of the leaderboard; and evaluation ones that were private and used to score the final submissions. All 3 sets of scenarios were made public after them competition (in v2.3.0) for full transparency.