A curated list of benchmarks for evaluating AI agents. Scoped to three domains: games, robotics, and computer use.
Interactive environments — primarily 2D — for training and evaluating decision-making agents on exploration, planning, and generalization.
| Preview | Name | Year | Description |
|---|---|---|---|
 |
Terra Nova | 2025 | Civilization-V-inspired challenge environment combining partial observability, credit assignment, and huge action spaces. |
 |
AutumnBench | 2025 | 43 interactive grid-world environments and 129 tasks evaluating world-model learning via reward-free exploration. |
 |
PuzzleJAX | 2025 | GPU-accelerated puzzle game engine benchmarking RL, tree search, and LLM reasoning across hundreds of games. |
| Kaggle Game Arena | 2025 | Head-to-head leaderboard pitting LLMs against each other in strategic games (chess, Werewolf, poker). | |
| ARC-AGI-3 | 2025 | Interactive reasoning benchmark where agents explore novel games, acquire goals on the fly, and adapt. | |
 |
POPGym Arcade | 2025 | Atari-inspired hardware-accelerated pixel environments for analyzing agent memory under partial observability. |
 |
PUZZLES | 2024 | 40 Simon Tatham logic puzzles of adjustable size benchmarking algorithmic and logical reasoning in RL. |
 |
Overcooked Generalisation Challenge | 2024 | Overcooked-AI extension benchmarking RL cooperation with novel partners across procedurally generated kitchens. |
 |
DiscoveryWorld | 2024 | Text-based environment with 120 tasks requiring agents to form hypotheses, run experiments, and conclude discoveries. |
 |
Craftax | 2024 | JAX-based open-world benchmark extending Crafter with NetHack-inspired mechanics for fast open-ended RL research. |
 |
XLand-MiniGrid | 2023 | JAX-based grid-world suite with millions of pre-sampled tasks for meta-reinforcement learning research. |
 |
Minigrid & Miniworld | 2023 | Modular 2D grid-world and 3D first-person environments for fast goal-oriented RL research. |
 |
Pgx | 2023 | JAX-based suite of GPU/TPU-accelerated board games (chess, Go, shogi, backgammon) for RL. |
 |
MiniHack | 2021 | Sandbox for designing custom NetHack-based RL environments targeting exploration, transfer, and specific capabilities. |
 |
Crafter | 2021 | Open-world 2D survival game evaluating diverse agent capabilities via semantically meaningful achievements. |
 |
Alchemy | 2021 | 3D Unity meta-RL benchmark with resampled latent causal structure for structure learning and hypothesis testing. |
| Virtual Tools | 2020 | Tool-use puzzle game probing flexible, creative physical problem solving and reasoning. | |
 |
NetHack Learning Environment | 2020 | Scalable, procedurally-generated roguelike environment for long-horizon exploration, planning, and skill-acquisition RL. |
 |
Procgen | 2019 | 16 procedurally-generated game-like environments benchmarking RL sample efficiency and generalization. |
 |
SMAC | 2019 | StarCraft II micromanagement benchmark for cooperative multi-agent deep reinforcement learning. |
 |
Hanabi Learning Environment | 2019 | Cooperative card game benchmark requiring multi-agent coordination and theory-of-mind under partial information. |
 |
Atari (ALE) | 2012 | Interface to hundreds of Atari 2600 games; canonical general-agent and RL evaluation platform. |
Physics-based simulation (and real-robot) benchmarks covering manipulation, locomotion, and physical reasoning.
| Preview | Name | Year | Simulator | Description |
|---|---|---|---|---|
 |
CaP-X | 2026 | MuJoCo / Isaac Sim | Code-as-policy robot-manipulation benchmark with 100+ tasks evaluating LLMs/VLMs generating control code from natural language. |
| KinDER | 2026 | PyBullet, MuJoCo, Pymunk | 25 procedurally-generated 2D/3D environments isolating kinematic and dynamic physical reasoning challenges. | |
 |
VLMgineer | 2025 | PyBullet | Framework using VLMs with evolutionary search to co-design physical tools and action plans. |
 |
VLABench | 2025 | MuJoCo (dm_control) | 100 language-conditioned manipulation task categories evaluating VLAs on long-horizon reasoning and world knowledge. |
 |
OGBench | 2025 | MuJoCo | Offline goal-conditioned RL benchmark with 8 environments probing stitching and long-horizon reasoning. |
 |
ManiSkill-HAB | 2025 | SAPIEN / PhysX (ManiSkill3) | GPU-accelerated low-level manipulation benchmark for in-home object rearrangement with RL/IL baselines. |
 |
EmbodiedBench | 2025 | AI2-THOR, Habitat, CoppeliaSim | 1,128 tasks across four environments evaluating multi-modal LLMs as vision-driven embodied agents. |
 |
RoboCasa | 2024 | MuJoCo (robosuite) | Large-scale kitchen-focused simulation with 100 tasks and diverse assets for generalist robots. |
 |
Kinetix | 2024 | Jax2D (JAX) | Open-ended procedurally-generated 2D physics-based tasks for training general RL agents. |
 |
I-PHYRE | 2024 | Pymunk (Chipmunk2D) | Interactive physical reasoning requiring intuitive physics, multi-step planning, and in-situ intervention. |
 |
Embodied Agent Interface | 2024 | OmniGibson, VirtualHome | Unified interface benchmarking LLM modules (goals, subgoals, actions, transitions) for embodied decision making. |
 |
DittoGym | 2024 | Taichi (MPM) | RL benchmark for reconfigurable soft robots requiring fine-grained morphology changes within an episode. |
 |
BEHAVIOR-1k | 2024 | OmniGibson (Omniverse/PhysX 5) | 1,000 human-grounded everyday activities simulated with realistic physics via OmniGibson. |
 |
LIBERO | 2023 | MuJoCo (robosuite) | Lifelong robot-manipulation benchmark probing declarative and procedural knowledge transfer across tasks. |
 |
FurnitureBench | 2023 | Isaac Gym | Reproducible real-world furniture assembly benchmark for long-horizon complex robot manipulation. |
 |
ProcTHOR | 2022 | AI2-THOR (Unity) | Procedurally generates large-scale interactive houses for embodied-AI navigation, rearrangement, and manipulation. |
 |
CALVIN | 2021 | PyBullet | Open-source simulated benchmark for long-horizon language-conditioned robot manipulation from onboard sensors. |
 |
ALFRED | 2020 | AI2-THOR (Unity) | Grounded natural-language instructions mapped to egocentric action sequences for household tasks. |
| robosuite | 2020 | MuJoCo | Modular MuJoCo-powered simulation framework with standardized robot-manipulation benchmark environments for reproducible robot learning. | |
 |
Habitat | 2019 | Habitat-Sim (Magnum) | High-performance photorealistic 3D simulator for embodied navigation, instruction following, and question answering. |
 |
PHYRE | 2019 | Box2D | 2D classical-mechanics puzzles benchmarking sample-efficient physical reasoning. |
 |
DM Control | 2018 | MuJoCo | MuJoCo-powered continuous-control tasks with standardized structure; de facto RL benchmark. |
 |
AI2-THOR | 2017 | Unity (PhysX) | Near-photorealistic interactive 3D indoor scenes for visual AI, navigation, and object interaction. |
Benchmarks for agents that interact with software — browsers, desktop GUIs, terminals, and codebases.
| Preview | Name | Year | Description |
|---|---|---|---|
| BrowseComp | 2025 | 1,266 short-answer questions requiring persistent browsing for hard-to-find, entangled information. | |
 |
TheAgentCompany | 2024 | Simulated software company evaluating agents on end-to-end professional workplace tasks. |
 |
τ-bench | 2024 | Simulated user–agent conversations testing tool use and policy adherence (retail, airline). |
 |
AndroidWorld | 2024 | 116 dynamic, parameterized tasks across 20 real Android apps. |
 |
OSWorld | 2024 | 369 real computer tasks across Ubuntu, Windows, and macOS for multimodal agents. |
 |
WorkArena | 2024 | 33 ServiceNow-based enterprise tasks evaluating knowledge-worker web agents via BrowserGym. |
 |
VisualWebArena | 2024 | Realistic visually-grounded web tasks for evaluating multimodal agents. |
 |
GAIA | 2023 | Real-world assistant questions requiring reasoning, multimodality, web browsing, and tool use. |
 |
SWE-bench | 2023 | 2,294 real GitHub issues from 12 Python repos requiring multi-file code edits. |
 |
AgentBench | 2023 | Multi-dimensional LLM-as-agent evaluation across 8 distinct interactive environments. |
 |
WebArena | 2023 | Self-hostable realistic multi-site web environments for reproducible agent evaluation. |
 |
Mind2Web | 2023 | Generalist web-agent tasks spanning 137 real websites across 31 domains. |
 |
WebShop | 2022 | Simulated e-commerce site with 1.18M products for instruction-following shopping agents. |
| MiniWoB++ | 2017 | Low-level keyboard/mouse web tasks with crowdsourced demos; foundational GUI-agent benchmark. |
PRs welcome. Please:
- Link the benchmark Name to its paper or primary reference.
- Keep the description to one or two sentences.
- Preserve reverse chronological order within each table.
Several curated lists overlap with this collection but emphasize different slices of the agent-evaluation landscape:
- Awesome-General-Agents-Benchmark — 50+ benchmarks across general reasoning, agent tasks, domain skills (math/science/coding/web), multimodal, and safety; annotated with top-performer scores and human baselines.
- awesome-ai-benchmarks — 114+ entries spanning programming, multimodal, translation, agent reasoning, and creative evaluation; paired with a searchable site at aibenchmarks.net.
- ai-agent-benchmark-compendium — 50+ benchmarks for function-calling/tool use, general assistant/reasoning, coding, and computer interaction; heavy overlap with our Computer Use table.
- Awesome-Robotic-Benchmarks — 30+ robotics benchmarks covering manipulation, locomotion, navigation, HRI, safety, simulation, and generalist tasks; finer sub-categories than our Robotics table.
In contrast, this list is scoped to interactive decision-making environments, with a focus on games, robotics, and computer use.