|
313 | 313 | <div class="topbar-inner"> |
314 | 314 | <a class="brand" href="#top"><img src="static/images/favicon.png" alt="">WorldMark</a> |
315 | 315 | <div class="topnav"> |
316 | | - <a href="#comparison">Why WorldMark</a> |
| 316 | + <a href="#overview">What is WorldMark</a> |
| 317 | + <a href="#examples">Examples</a> |
317 | 318 | <a href="#suite">Benchmark</a> |
318 | 319 | <a href="#metrics">Metrics</a> |
319 | 320 | <a href="#results">Results</a> |
@@ -373,24 +374,42 @@ <h1><span class="wm">WorldMark</span>: A Unified Benchmark Suite for Interactive |
373 | 374 |
|
374 | 375 | <!-- ── Teaser ── --> |
375 | 376 | <section class="wrap" id="overview" style="padding-top: 28px;"> |
| 377 | + <p class="kicker">What is WorldMark?</p> |
| 378 | + <h2>The user acts, and the world responds</h2> |
| 379 | + <p> |
| 380 | + An interactive world model is not a video generator: it is an environment. You press a key, and the world |
| 381 | + is supposed to move that way, keep moving while you hold it, stop when you let go, and still be there when |
| 382 | + you turn back. <span class="wm">WorldMark</span> is a benchmark that measures exactly that. |
| 383 | + </p> |
| 384 | + <p> |
| 385 | + It drives <strong>ten heterogeneous models</strong>—caption-, pose-, keyboard-, and trajectory-controlled—from |
| 386 | + <strong>one shared WASD-style action vocabulary</strong> over <strong>500 standardized cases</strong>, and scores the |
| 387 | + result with <strong>nine deterministic metrics</strong> covering action dynamics, world memory, and visual quality. |
| 388 | + </p> |
376 | 389 | <figure> |
377 | 390 | <div class="figframe"><img src="static/images/teaser.png" alt="WorldMark overview: per-model adapters translate one shared action vocabulary into each model's native control format; a round-trip probe compares outbound and return views at equal accumulated motion."></div> |
378 | 391 | <figcaption> |
379 | 392 | Models take actions in incompatible formats—keyboard actions, text prompts, camera trajectories—so per-model adapters translate one shared vocabulary into each native format. |
380 | 393 | For a frame on the outbound leg and its return-leg counterpart at equal accumulated motion, the two views should match but diverge to different degrees. |
381 | 394 | </figcaption> |
382 | 395 | </figure> |
| 396 | + |
| 397 | + <p class="blist-head">Two obstacles to fair evaluation</p> |
| 398 | + <ul class="blist bad"> |
| 399 | + <li><b>Incompatible action formats.</b> Captions, trajectories, pose strings, action functions—no shared protocol, so no two models get identical instructions.</li> |
| 400 | + <li><b>Action following collapsed into one number.</b> Trajectory or direction error shows neither how fast the world reacts to a command switch, nor how cleanly it moves along the commanded axis.</li> |
| 401 | + </ul> |
| 402 | + |
| 403 | + <p class="blist-head">Our contributions</p> |
| 404 | + <ul class="blist good"> |
| 405 | + <li><b>A unified action interface</b> that drives ten models with incompatible control formats from a single action vocabulary, together with a standardized suite of 500 cases spanning visual styles, viewpoints, and difficulty tiers; supporting a new model requires one adapter.</li> |
| 406 | + <li><b>An action-centric metric suite</b> that treats each command as a <em>step input</em>, after control-systems practice, and characterizes <em>how</em> a world responds—when motion begins, whether it stays on the commanded axis, and whether it holds—rather than only how far it strays, resolved per motion axis and complemented by memory metrics at three timescales and two visual quality scores.</li> |
| 407 | + <li><b>A systematic evaluation</b> that turns these measurements into actionable findings: a response–stability trade-off no single action metric can capture, command axes that fail independently, and a negative correlation between perceptual quality and responsiveness that contradicts appearance-based evaluation.</li> |
| 408 | + </ul> |
383 | 409 | </section> |
384 | 410 |
|
385 | 411 | <!-- ── Qualitative video examples ── --> |
386 | 412 | <section class="wrap" id="examples"> |
387 | | - <p class="kicker">Qualitative Examples</p> |
388 | | - <h2>What each metric actually sees</h2> |
389 | | - <p class="muted"> |
390 | | - A high- and a low-scoring model per metric, on the <strong>same scene under the same command</strong>—the only |
391 | | - difference is the model. The on-screen keys show the command being issued. |
392 | | - </p> |
393 | | - |
394 | 413 | <h3 class="ex-h">Action Dynamics <span>Does the world go where it is told—and when?</span></h3> |
395 | 414 | <div class="ex-tabs" data-group="action" role="tablist"> |
396 | 415 | <button class="ex-tab active" data-group="action" data-ex="1">Direction Accuracy</button> |
@@ -486,26 +505,6 @@ <h3 class="ex-h">Visual Quality <span>How does each frame look?</span></h3> |
486 | 505 |
|
487 | 506 | </section> |
488 | 507 |
|
489 | | - <!-- ── Comparison ── --> |
490 | | - <section class="wrap" id="comparison"> |
491 | | - <p class="kicker">Why WorldMark</p> |
492 | | - <h2>The user acts, and the world responds</h2> |
493 | | - |
494 | | - <p class="blist-head">Two obstacles to fair evaluation</p> |
495 | | - <ul class="blist bad"> |
496 | | - <li><b>Incompatible action formats.</b> Captions, trajectories, pose strings, action functions—no shared protocol, so no two models get identical instructions.</li> |
497 | | - <li><b>Action following collapsed into one number.</b> Trajectory or direction error shows neither how fast the world reacts to a command switch, nor how cleanly it moves along the commanded axis.</li> |
498 | | - </ul> |
499 | | - |
500 | | - <p class="blist-head">How <span class="wm">WorldMark</span> removes them</p> |
501 | | - <ul class="blist good"> |
502 | | - <li><b>One vocabulary, ten native interfaces.</b> Per-model adapters deliver semantically identical instructions across 500 standardized cases. A new model costs one adapter.</li> |
503 | | - <li><b>Action as a step response.</b> Direction accuracy, purity, latency, and stability—each resolved per motion axis—plus world memory and visual quality.</li> |
504 | | - <li><b>Deterministic by construction.</b> Feed-forward estimators only: bit-identical across runs, no SLAM or sampled-VLM variance.</li> |
505 | | - <li><b>Differences one score cannot see.</b> The fastest responders are often the least stable, and the best-looking model is among the least responsive.</li> |
506 | | - </ul> |
507 | | - |
508 | | - </section> |
509 | 508 |
|
510 | 509 | <!-- ── Benchmark suite ── --> |
511 | 510 | <section class="wrap" id="suite"> |
@@ -830,11 +829,11 @@ <h2>BibTeX</h2> |
830 | 829 |
|
831 | 830 | <!-- ── Right side nav ── --> |
832 | 831 | <ul class="side-nav"> |
833 | | - <li><a href="#comparison">Why WorldMark</a></li> |
| 832 | + <li><a href="#overview">What is WorldMark</a></li> |
| 833 | + <li><a href="#examples">Examples</a></li> |
834 | 834 | <li><a href="#suite">Benchmark Suite</a></li> |
835 | 835 | <li><a href="#metrics">Metrics</a></li> |
836 | 836 | <li><a href="#results">Results</a></li> |
837 | | - <li><a href="#examples">Qualitative Examples</a></li> |
838 | 837 | <li><a href="#findings">Key Findings</a></li> |
839 | 838 | <li><a href="#arena">Arena</a></li> |
840 | 839 | <li><a href="#bibtex">BibTeX</a></li> |
|
0 commit comments