You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Restructure page: drop Abstract, bullet-point Why WorldMark
- Remove Abstract section; its content now drives Why WorldMark as two
problem bullets (✗) and four solution bullets (✓)
- Hero stat cards expanded into detailed lists (models, case breakdown,
metric families)
- Action Suite: trajectory and VLM-selection figures side by side
- Capability radar shown at full width
- Remove Qualitative Examples section
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
<li>Feed-forward only—bit-identical across runs</li>
380
+
</ul>
381
+
</div>
334
382
</div>
335
383
</header>
336
384
@@ -345,34 +393,26 @@ <h1><span class="wm">WorldMark</span>: A Unified Benchmark Suite for Interactive
345
393
</figure>
346
394
</section>
347
395
348
-
<!-- ── Abstract ── -->
349
-
<sectionclass="wrap" id="abstract">
350
-
<pclass="kicker">Abstract</p>
351
-
<h2>The user acts, and the world responds</h2>
352
-
<p>
353
-
Unlike text- or image-driven video generation, an interactive world model is driven by <em>actions</em>. Two obstacles stand in the way of fair and comprehensive evaluation.
354
-
First, models take actions in incompatible formats—captions, camera trajectories, action functions—so no shared protocol has been established.
355
-
Second, while existing benchmarks have advanced world memory and visual quality, action following is reduced to trajectory or direction error, which collapses a whole path into one number:
356
-
not how quickly the world reacts to a command switch, nor how cleanly it moves along the commanded axis.
357
-
</p>
358
-
<p>
359
-
<spanclass="wm">WorldMark</span> removes both obstacles. Per-model adapters translate a shared WASD-style vocabulary into each model's native control format,
360
-
so <strong>ten heterogeneous models receive semantically identical instructions</strong> across 500 standardized cases spanning styles, viewpoints, and difficulty tiers; a new model costs one adapter.
361
-
On this common ground we characterize action dynamics through a <strong>control-systems lens</strong>—direction accuracy, direction purity, response latency, and motion stability, each resolved per axis—alongside suites for world memory and visual quality.
362
-
Together they expose differences existing protocols cannot see: the fastest responders are often the least stable; per-axis resolution reveals models that follow translation almost perfectly while barely responding to rotation;
363
-
and the model with the best perceptual quality ranks last in translational direction accuracy and latency. We release all data, evaluation code, and model outputs.
364
-
</p>
365
-
</section>
366
-
367
396
<!-- ── Comparison ── -->
368
397
<sectionclass="wrap" id="comparison">
369
398
<pclass="kicker">Why WorldMark</p>
370
-
<h2>What existing benchmarks cannot measure</h2>
371
-
<pclass="muted">
372
-
Visual quality and memory are well served; action following is not. Prior protocols report a single direction or trajectory score,
373
-
which cannot in principle capture <em>response dynamics</em>—latency, persistence, stability—and none is fully reproducible run-to-run.
374
-
</p>
375
-
<divclass="cmp-scroll" style="margin-top: 20px;">
399
+
<h2>The user acts, and the world responds</h2>
400
+
401
+
<pclass="blist-head">Two obstacles stand in the way of fair evaluation</p>
402
+
<ulclass="blist bad">
403
+
<li><b>Actions arrive in incompatible formats.</b> Captions, camera trajectories, pose strings, action functions—no shared protocol exists, so models cannot be driven with identical instructions on identical scenes.</li>
404
+
<li><b>Action following is collapsed into one number.</b> Existing benchmarks advanced world memory and visual quality, but reduce control to trajectory or direction error—which shows neither how quickly the world reacts to a command switch, nor how cleanly it moves along the commanded axis.</li>
<li><b>One vocabulary, ten native interfaces.</b> Per-model adapters translate a shared WASD-style vocabulary into each model's own control format, so ten heterogeneous models receive <strong>semantically identical instructions</strong> across 500 standardized cases spanning styles, viewpoints, and difficulty tiers. A new model costs one adapter.</li>
410
+
<li><b>Action dynamics through a control-systems lens.</b> An action command is a <em>step input</em>: we measure direction accuracy, direction purity, response latency, and motion stability—each resolved separately for translation and rotation—alongside suites for world memory and visual quality.</li>
411
+
<li><b>Deterministic by construction.</b> Every estimator is feed-forward, so scores are bit-identical across runs: no SLAM bundle-adjustment variance, no sampled VLM judgments.</li>
412
+
<li><b>Differences existing protocols cannot see.</b> The fastest responders are often the least stable; per-axis resolution reveals models that follow translation almost perfectly while barely responding to rotation; and the model with the best perceptual quality ranks last in translational accuracy and latency.</li>
<divclass="tier med"><b>Medium · 40s, two-segment</b>Adds exactly one switch, without which there is no transient; provides the first round trips.</div>
445
485
<divclass="tier hard"><b>Hard · 60s, three-segment</b>Patrol routes and 360° rotations run past most models' conditioning window, where revisit and global memory fail.</div>
<figcaption>The 15 standardized action sequences, from elementary translations and rotations to combined and cyclic trajectories.</figcaption>
450
-
</figure>
451
-
452
487
<p>
453
488
For each image, a VLM selects the five most plausible sequences from the library, identifying physical constraints—lateral obstacles in a corridor make prolonged strafing implausible—so
454
489
models are not penalized for refusing to walk through a wall. Across 100 images this yields <strong>500 evaluation cases</strong>.
455
490
</p>
456
-
<figure>
457
-
<divclass="figframe"><imgsrc="static/images/vlm_traj.jpg" alt="Context-aware action selection via VLM reasoning" loading="lazy"></div>
458
-
<figcaption>Context-aware action selection. A VLM analyzes the initial image to identify physical constraints and selects plausible action sequences from the predefined library.</figcaption>
<figcaption>The 15 standardized action sequences, from elementary translations and rotations to combined and cyclic trajectories.</figcaption>
495
+
</figure>
496
+
<figure>
497
+
<divclass="figframe"><imgsrc="static/images/vlm_traj.jpg" alt="Context-aware action selection via VLM reasoning" loading="lazy"></div>
498
+
<figcaption>Context-aware action selection. A VLM analyzes the initial image to identify physical constraints and selects plausible action sequences from the predefined library.</figcaption>
499
+
</figure>
500
+
</div>
460
501
</section>
461
502
462
503
<!-- ── Metrics ── -->
@@ -654,30 +695,13 @@ <h2>Ten models, four splits</h2>
654
695
655
696
<divclass="wrap" style="padding: 0;">
656
697
<figure>
657
-
<divclass="figframe"><imgsrc="static/images/capability_profiles.png" alt="Radar capability profile of each model over thirteen columns, Real vs Stylized overlaid" loading="lazy"></div>
698
+
<divclass="figframe"style="max-width: 100%;"><imgsrc="static/images/capability_profiles.png" alt="Radar capability profile of each model over thirteen columns, Real vs Stylized overlaid" loading="lazy"></div>
658
699
<figcaption>Capability profile of each model over the thirteen reported columns, First-Person Real (solid) against Stylized (dashed). All axes run 0–100 in the same order in every panel; the gap between the two outlines is that model's stylization penalty. Lyra 2.0 is the only near-convex profile—one all-round model and no second—and the stylization penalty falls hardest on the models that lead on Real.</figcaption>
659
700
</figure>
660
701
</div>
661
702
</section>
662
703
663
704
<!-- ── Qualitative examples ── -->
664
-
<sectionclass="wrap" id="examples">
665
-
<pclass="kicker">Qualitative Examples</p>
666
-
<h2>What each failure looks like</h2>
667
-
<pclass="muted">
668
-
One high- and one low-scoring case per metric, holding scene, command, and sampling fixed within each pair—the only difference is the model.
669
-
The Action Dynamics rows carry a signal panel as well as frames, because their failures are in the <em>motion</em> rather than the pixels: six stills cannot show that a model kept turning after the command changed.
670
-
</p>
671
-
<figure>
672
-
<divclass="figframe"><imgsrc="static/images/metric_examples_action.jpg" alt="High- and low-scoring examples for each Action Dynamics metric with flow signal panels" loading="lazy"></div>
673
-
<figcaption>Action Dynamics examples. The panel at left is the signal the metric reads; shading marks what the flow on the commanded axis <em>should</em> do, and the dashed rule in Response Latency is the command switch. Green marks the high-scoring model, red the low-scoring one.</figcaption>
674
-
</figure>
675
-
<figure>
676
-
<divclass="figframe"><imgsrc="static/images/metric_examples_rest.jpg" alt="High- and low-scoring examples for memory and quality metrics" loading="lazy"></div>
677
-
<figcaption>Memory and quality examples, drawn from the span each metric actually reads: adjacent frames for Local Memory, outbound and return halves for Revisit Memory, the full trajectory otherwise. These failures are visible in the frames themselves.</figcaption>
0 commit comments