Mimic open spiel's 'strictMode: false' in custom Game Arena envs (#1390)

* If any of the agents raise an exception or timeout, error out the
whole game
* If an agent makes an invalid move, set "DONE" status on all agents and
use default forfeit rewards

Author: lipovetz

.agents/skills/create-environment/SKILL.md

@@ -1,5 +1,34 @@
 # Create / Update an Environment
 
+## Step 0: Ask whether this is a Game Arena environment
+
+**Before writing any code, ask the user** — the answer changes how failures and
+illegal moves must be scored (Step 3b), and there is no way to infer it from the
+game rules alone.
+
+```
+AskUserQuestion({
+  questions: [{
+    question: "Is this a Game Arena environment (LLM-vs-LLM, results feed the Elo leaderboard)?",
+    header: "Env type",
+    multiSelect: false,
+    options: [
+      {label: "Game Arena",
+       description: "Agents are language models behind a harness; episodes are scored into a cross-model Elo rating. Requires the failure semantics in Step 3b."},
+      {label: "Regular",
+       description: "Classic Kaggle simulation competition or a standalone env. Per-agent INVALID/ERROR statuses are fine; use the framework defaults."},
+    ]
+  }]
+})
+```
+
+Signals that it is Game Arena, worth mentioning when you ask: the env ships a
+`harness.py`, agents receive natural-language prompts, or the user talks about
+comparing models rather than comparing submissions.
+
+If **Regular**, skip Step 3b and use the plain patterns in Step 3.
+If **Game Arena**, Step 3b is mandatory.
+
 ## Step 1: Create the directory and files
 
 Create `kaggle_environments/envs/<name>/` with:
@@ -145,7 +174,8 @@ from .agents import agents  # or define inline
 
 ### Common interpreter patterns
 
-**Validate and penalize invalid actions:**
+**Validate and penalize invalid actions** (regular envs only -- Game Arena envs
+must use Step 3b instead):
 ```python
 if state[i].action < 0 or state[i].action >= max_val:
     state[i].status = "INVALID"
@@ -175,6 +205,151 @@ state[0].observation.board = board  # shared fields (if marked shared in spec)
 state[0].observation.lastOpponentAction = state[1].action
 ```
 
+## Step 3b: Failure and illegal-move handling (Game Arena environments)
+
+Skip this section for regular environments.
+
+Game Arena episodes are scored into a **cross-model Elo leaderboard**, so every
+episode either produces a trustworthy result or must produce none at all. That
+forces a distinction the framework does not make for you:
+
+| What happened | Framework status | Correct outcome |
+|---|---|---|
+| Agent process raised, or the model provider errored | `ERROR` | **Void the episode.** Not a game result. |
+| Agent exceeded `actTimeout` | `TIMEOUT` | **Void the episode.** Same as above. |
+| Agent returned a well-formed action that breaks the rules | `INVALID`, or your own rule check | **Scored forfeit.** Offender loses, opponent wins. |
+
+The reason for the split: a crash or a timeout is a *broken participant*, not a
+model playing badly. Scoring it as a loss injects infrastructure flakiness into
+the ratings -- a model on a slow provider would rank below one on a fast
+provider for reasons that have nothing to do with gameplay. An illegal move is
+the opposite: the model *did* play, it just played badly, and failing to follow
+the action format is a genuine capability signal that belongs in the rating.
+
+`open_spiel_env` with `strictMode: false` (the default) is the reference
+implementation -- see `open_spiel_env.py`, the `agent_error` / `invalid_action`
+branches. `word_association` and `word_art` implement the same semantics.
+**New Game Arena environments must match it.** Do not add a config flag for
+this; the behavior is not per-env negotiable.
+
+### The pattern
+
+Define a module-level forfeit reward matching open_spiel's convention:
+
+```python
+# Statuses core.py assigns when an agent crashes or times out, as opposed to
+# returning a well-formed-but-illegal action.
+_FRAMEWORK_FAILURE_STATUSES = ("ERROR", "TIMEOUT")
+
+# Reward applied to the forfeiting side. The opponent receives the negation.
+DEFAULT_INVALID_ACTION_REWARD = -1
+```
+
+**Void on crash/timeout.** Force every seat to `ERROR`, except seats already
+`TIMEOUT` (which voids the episode identically and is more informative in the
+replay). `core.py` then nulls all rewards automatically:
+
+```python
+def _abort_on_agent_failure(state):
+    """Void the episode if the framework marked any seat ERROR or TIMEOUT.
+    Returns True if the episode was ended."""
+    if not any(s.status in _FRAMEWORK_FAILURE_STATUSES for s in state):
+        return False
+    for s in state:
+        if s.status != "TIMEOUT":
+            s.status = "ERROR"
+    return True
+```
+
+**Forfeit on an illegal move.** Every seat ends `DONE` -- including the
+offender -- so the episode scores normally:
+
+```python
+def forfeit(state, offending_seat):
+    for i in range(len(state)):
+        state[i].status = "DONE"
+        state[i].reward = (
+            DEFAULT_INVALID_ACTION_REWARD
+            if i == offending_seat
+            else -DEFAULT_INVALID_ACTION_REWARD
+        )
+```
+
+Never leave a seat in `INVALID` at episode end. `core.py` nulls the reward of
+any `ERROR`/`INVALID`/`TIMEOUT` agent, so an `INVALID` terminal status is
+indistinguishable from a crash downstream -- which is exactly the collapse this
+section exists to prevent.
+
+### Wiring it into the interpreter
+
+Check for framework failures **before** your action-processing code runs, so a
+crash cannot be laundered into a scored forfeit:
+
+```python
+def interpreter(state, env):
+    if env.done:
+        return state
+
+    # A crashed or timed-out seat is a broken participant, not a player
+    # making an illegal move. Void before process_action can rescore it.
+    if _abort_on_agent_failure(state):
+        return state
+
+    forfeited = process_action(state, env.configuration)
+    ...
+```
+
+### Env-shape adjustments
+
+The two rules above are fixed; how they map onto your env is not. Decide these
+explicitly and write the reasoning into a docstring:
+
+* **Team games.** Scope the forfeit to the offender's *team*, not the lone
+  seat -- crediting the offender's own partner would reward a team for its own
+  foul. `word_association` (2v2) gives both offending seats
+  `DEFAULT_INVALID_ACTION_REWARD` and both opponents its negation.
+* **Multi-game episodes.** A forfeit is terminal for the whole episode. Return
+  a flag from your action processor and gate the next-game rollover on it;
+  otherwise the rollover resets the forfeiting seats to `ACTIVE` and overwrites
+  the forfeit rewards. (`word_association` had exactly this bug.)
+* **Running-score rewards.** If `reward` is a cumulative point total rather
+  than a win/loss value, overwriting it with a flat ±1 discards every completed
+  round. `word_art` deliberately diverges here: on `INVALID` it ends all seats
+  `DONE` and **keeps the accumulated points**. Crash/timeout still voids.
+
+### Visualizers
+
+A visualizer that computes terminal state as `status === 'DONE'` will silently
+render nothing on a voided episode. Handle `ERROR`/`TIMEOUT` explicitly and show
+why the episode was voided:
+
+```ts
+const isVoided = step.some((p) => p?.status === 'ERROR' || p?.status === 'TIMEOUT');
+const isGameOver = step.every((p) => p?.status === 'DONE') || isVoided;
+```
+
+### Required tests
+
+```python
+@pytest.mark.parametrize("crash_seat", range(NUM_AGENTS))
+def test_agent_crash_voids_the_episode(crash_seat):
+    def crash(observation, configuration):
+        raise RuntimeError("provider exploded")
+    agents = [legal_agent] * NUM_AGENTS
+    agents[crash_seat] = crash
+    env = make("<name>")
+    env.run(agents)
+    assert [s.status for s in env.state] == ["ERROR"] * NUM_AGENTS
+    assert [s.reward for s in env.state] == [None] * NUM_AGENTS
+```
+
+Cover, at minimum:
+- crash on every seat -> all `ERROR`, all rewards `None`
+- `DeadlineExceeded` on every seat (from `kaggle_environments.errors`) -> the
+  offending seat keeps `TIMEOUT`, the rest are `ERROR`, all rewards `None`
+- an illegal-but-well-formed action -> all `DONE` with ±1 rewards, never `None`
+- if the env supports multi-game episodes: a forfeit does not start a new game
+
 ## Step 4: Write agents
 
 Agent functions receive `(observation, configuration)` as Struct objects and return an action:
@@ -213,6 +388,8 @@ def test_rewards():
 
 
 def test_invalid_action():
+    # Regular envs only. Game Arena envs assert the Step 3b shape instead:
+    #   statuses == ["DONE", "DONE"], rewards == [-1, 1]
     env = make("<name>")
     env.run([bad_agent, good_agent])
     json = env.toJSON()
@@ -241,10 +418,22 @@ Follow the `create-visualizer` skill to build a web-based replay visualizer for
 
 ## Checklist
 
+- [ ] Asked the user whether this is a Game Arena or regular environment (Step 0)
 - [ ] `<name>.json` spec is valid JSON with all required top-level keys
 - [ ] `<name>.py` exports `specification`, `interpreter`, `renderer`, `html_renderer`
 - [ ] Interpreter handles: normal play, invalid actions, game-over conditions
 - [ ] Rewards are set correctly for all outcomes (win/lose/draw/invalid)
 - [ ] `__init__.py` exists (can be empty)
 - [ ] Tests cover: normal completion, rewards, invalid actions, renderer output
 - [ ] `uv run ruff check --fix . && uv run ruff format .` passes
+
+Game Arena environments additionally:
+
+- [ ] Crash/timeout voids the episode (all seats `ERROR`/`TIMEOUT`, all rewards `None`)
+- [ ] Illegal move is a scored forfeit (all seats `DONE`, ±`DEFAULT_INVALID_ACTION_REWARD`)
+- [ ] No seat can end an episode in `INVALID`
+- [ ] Failure check runs before action processing in the interpreter
+- [ ] Team scoping, multi-game terminality, and running-score handling decided and documented
+- [ ] Visualizer renders a voided episode instead of silently showing nothing
+- [ ] Per-seat crash and timeout tests exist, plus an illegal-move forfeit test
+- [ ] No config flag was added to make this behavior optional

kaggle_environments/envs/word_art/visualizer/default/src/renderer.ts

@@ -122,7 +122,12 @@ export function renderer(options: RendererOptions) {
   const blueAttemptsUsed: number = obs0.blue_attempts_used ?? 0;
   const yellowAttemptsUsed: number = obs0.yellow_attempts_used ?? 0;
 
-  const isDone = currentStep?.every?.((p: any) => p?.status === 'DONE') ?? false;
+  // A crashed or timed-out seat voids the episode: every seat ends ERROR
+  // (a timed-out one keeps TIMEOUT) and all rewards are nulled. Those steps
+  // are terminal too, so fold them into isDone or the final panel never
+  // renders and the view sticks on a half-played round.
+  const isVoided = currentStep?.some?.((p: any) => p?.status === 'ERROR' || p?.status === 'TIMEOUT') ?? false;
+  const isDone = (currentStep?.every?.((p: any) => p?.status === 'DONE') ?? false) || isVoided;
 
   // Detect a round-transition step: on the sub-step where both teams
   // finish their guesses, the env immediately clears the round state and
@@ -422,7 +427,10 @@ export function renderer(options: RendererOptions) {
   statusBar.className = 'wa-status-bar sketched-border';
   if (isDone) {
     let outcome: string;
-    if (blueScore > yellowScore) outcome = `Blue wins ${blueScore}–${yellowScore}!`;
+    if (isVoided) {
+      const failed = currentStep.find((p: any) => p?.status === 'ERROR' || p?.status === 'TIMEOUT');
+      outcome = `Episode voided — an agent ${failed?.status === 'TIMEOUT' ? 'timed out' : 'crashed'}`;
+    } else if (blueScore > yellowScore) outcome = `Blue wins ${blueScore}–${yellowScore}!`;
     else if (yellowScore > blueScore) outcome = `Yellow wins ${yellowScore}–${blueScore}!`;
     else outcome = `Tie ${blueScore}–${yellowScore}`;
     const final = document.createElement('span');

kaggle_environments/envs/word_art/word_art.json

@@ -2,7 +2,7 @@
   "name": "word_art",
   "title": "Word Art",
   "description": "A 2v2 cooperative-vs-competitive game. Each round a secret word is shown to one artist per team; that artist draws ASCII art and passes it to their teammate, who has up to max_attempts guesses. Point value awarded on a correct guess is looked up in guess_points by attempt index (element 0 = attempt 1, element 1 = attempt 2, ...); failing all attempts scores 0. After N rounds the higher total wins. The artist and guesser within each team swap every round.",
-  "version": "1.1.0",
+  "version": "1.2.0",
   "agents": [4],
   "configuration": {
     "num_rounds": {

kaggle_environments/envs/word_art/word_art.py

@@ -583,9 +583,10 @@ def initialize_game(state, env):
     env.word_art_state = _WordArtState(sampled)
 
 
-# Statuses the kaggle framework sets when an agent crashes or times out.
-# These end the episode -- see _abort_on_agent_failure.
-_TERMINAL_FAILURE_STATUSES = ("TIMEOUT", "ERROR", "INVALID")
+# Statuses core.py sets when an agent crashes or times out, as opposed to
+# submitting a well-formed-but-illegal action. These void the episode --
+# see _abort_on_agent_failure.
+_FRAMEWORK_FAILURE_STATUSES = ("TIMEOUT", "ERROR")
 
 
 def _abort_on_agent_failure(state):
@@ -596,20 +597,36 @@ def _abort_on_agent_failure(state):
     making a bad move, and word_art cannot score a 2v2 game around one. The
     alternative -- skipping that seat and playing on -- yields an episode
     where one model contributes no art and no guesses for the rest of the
-    game while the scoreboard still reports a winner. Ending here keeps the
-    failure loud: the offending seat retains its status, so core.py nulls its
-    reward and the replay shows an errored episode rather than a lopsided one.
+    game while the scoreboard still reports a winner.
+
+    Two distinct outcomes, matching open_spiel_env's non-strict path:
+
+    * TIMEOUT/ERROR voids the episode. Every seat is forced to ERROR (a
+      TIMEOUT seat keeps TIMEOUT, which voids it the same way) so core.py
+      nulls all four rewards and the replay reads as an errored episode
+      rather than a decided one.
+    * INVALID -- a well-formed action that broke the action schema -- ends
+      the episode as a normal completion: all seats DONE, keeping whatever
+      round points each team had already banked. Unlike open_spiel, the
+      accumulated score is preserved rather than overwritten with a flat
+      +/-1, because word_art's reward channel is a running point total and
+      discarding it would throw away every completed round.
 
     Note this is deliberately NOT the illegalMoveForfeit path. A model that
     answers unparseably forfeits the turn and plays on; a model whose agent
     raised has no working turn to fall back to.
     """
-    if not any(s.status in _TERMINAL_FAILURE_STATUSES for s in state):
-        return False
-    for s in state:
-        if s.status not in _TERMINAL_FAILURE_STATUSES:
+    if any(s.status in _FRAMEWORK_FAILURE_STATUSES for s in state):
+        for s in state:
+            if s.status != "TIMEOUT":
+                s.status = "ERROR"
+        return True
+    if any(s.status == "INVALID" for s in state):
+        for s in state:
             s.status = "DONE"
-    return True
+            s.reward = s.reward or 0
+        return True
+    return False
 
 
 def _set_art_statuses(state, round_idx):

kaggle_environments/envs/word_association/visualizer/default/src/renderer.tsx

@@ -591,10 +591,26 @@ export const GameRenderer: React.FC<GameRendererProps> = (options: GameRendererP
     }
   }, [step]);
 
-  const isGameOver = currentEnvStep[0].status === 'DONE';
+  // A crashed or timed-out seat ends the episode with ERROR/TIMEOUT rather
+  // than DONE, so the terminal check must cover all three or the results
+  // panel silently never renders on a voided episode.
+  const isVoided = currentEnvStep.some((p: any) => p?.status === 'ERROR' || p?.status === 'TIMEOUT');
+  const isGameOver = currentEnvStep[0].status === 'DONE' || isVoided;
   let winnerText: React.ReactNode = null;
 
-  if (isGameOver) {
+  if (isVoided) {
+    // Rewards are nulled on a voided episode, so there is no winner to show.
+    const failedSeat = currentEnvStep.findIndex((p: any) => p?.status === 'ERROR' || p?.status === 'TIMEOUT');
+    const reason = currentEnvStep[failedSeat]?.status === 'TIMEOUT' ? 'timed out' : 'crashed';
+    winnerText = (
+      <div style={{ display: 'flex', flexDirection: 'column', alignItems: 'center', lineHeight: '1.2' }}>
+        <span style={{ color: '#ff6b6b' }}>⚠ EPISODE VOIDED ⚠</span>
+        <span style={{ fontSize: '14px', color: '#aaaaaa', marginTop: '4px', fontWeight: 'normal' }}>
+          (An agent {reason} — no result recorded)
+        </span>
+      </div>
+    );
+  } else if (isGameOver) {
     const trapIndex = renderState.roles.findIndex((role) => role === 'trap');
     const trapRevealed = trapIndex !== -1 && renderState.revealed[trapIndex];
 

kaggle_environments/envs/word_association/word_association.json

@@ -2,7 +2,7 @@
   "name": "word_association",
   "title": "Word Association",
   "description": "A game of word association and deduction, adapted for AI agents.",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "agents": [4],
   "configuration": {
     "board_size": {