AutomataBench

AutomataBench is a benchmark for reversible space-time reconstruction. Each instance gives a reversible binary 2x2 Margolus block cellular automaton, a toroidal grid, a time horizon, and a set of revealed cells from its space-time evolution. The task is to recover an initial state whose simulated trace matches every observation.

The public GitHub repo contains the task schema, verifier, scoring harness, prompt utilities, baseline solver, docs, and all public splits.

See docs/BENCHMARK.md for the task format, scoring policy, submission instructions, and model-reporting guidance.

Install

pip install -e .

Verify Answers

automata-bench-verify path/to/public_split.jsonl

Run Baseline and Score

automata-bench-baseline --data-dir data --split public_dev --output predictions.jsonl
automata-bench --data-dir data --split public_dev --predictions predictions.jsonl

Prediction rows should look like:

{"id": "sample_easy_000001", "initial_state": ["0100", "0111", "1000", "1101"]}

The initial_state may be row strings or row arrays of 0/1 integers.

Visualize a Problem

automata-bench-visualize \
  --data data/sample.jsonl \
  --id sample_easy_000001 \
  --out sample_easy_000001.html

Add --show-answer for local sample/dev debugging when answers are included.

Dataset

The public dataset artifact is intended to live at:

AutomataBench/automata-bench

with splits:

sample
public_dev
public_eval

All public splits include answers in v1. Official leaderboard scores should use a separate non-public evaluation set, not these public splits.

The checked-in public splits are:

sample:      60 rows, 20 easy / 20 medium / 20 hard
public_dev:  300 rows, 100 easy / 100 medium / 100 hard
public_eval: 300 rows, 75 easy / 100 medium / 125 hard

sample is a balanced quick-inspection excerpt from public data. public_dev and public_eval are public practice and reproducibility splits with answers; because answers are public, neither split is used for trusted official leaderboard scoring.

All public instances are marked metadata.unique_solution = true. Uniqueness was certified during generation with a SAT encoding by finding the reference initial state, blocking it, and proving the blocked formula UNSAT. The verifier checks that a submitted initial state simulates to a trace matching every observation; because instances are unique, satisfying the observations implies exact reconstruction.

The held-out private evaluation set used for the initial organizer-run results was checked on 2026-06-24 to have zero rule_id overlap with sample, public_dev, and public_eval.

The public_dev and public_eval splits were also checked on 2026-06-24 to have zero exact-instance overlap and zero rule_id overlap.

Links

• Hugging Face dataset: AutomataBench/automata-bench
• Website: https://automatabench.com
• Contact: data@automatabench.com

License

The source code in this repository is licensed under the Apache License 2.0.

The public AutomataBench dataset files and documentation are licensed under the Creative Commons Attribution 4.0 International License.

The AutomataBench name, logo, website, official leaderboard, and non-public evaluation or data assets are not licensed under these public licenses.

For larger datasets, custom-generated evaluation suites, or commercial licensing, contact: data@automatabench.com.