|
| 1 | +# Known traps |
| 2 | + |
| 3 | +Every entry below is a mistake that has actually been made, or a hazard found by reading |
| 4 | +source that contradicted a reasonable assumption. Each maps to a checklist item. |
| 5 | +Read this before concluding the probe. |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## 1. Letter semantics are not portable → B2 |
| 10 | + |
| 11 | +The same letter means different things across repos. Observed: one repo maps `a`/`d` to |
| 12 | +**yaw rotation** and `j`/`l` to **strafe translation**; several others do the exact opposite. |
| 13 | +Another repo uses `a`/`d` for translation and spells rotation as `left_rot`/`right_rot`. |
| 14 | + |
| 15 | +An adapter that copies a letter string from a sibling repo produces a video that moves when |
| 16 | +it should turn — and passes every numeric test. |
| 17 | + |
| 18 | +**Always grep the key→action dict and paste it into `PROBE.md`.** |
| 19 | + |
| 20 | +## 2. A parameter named "speed" may not be a speed → C1, D1 |
| 21 | + |
| 22 | +Observed: a repo whose JSON takes `action_speed_list: [4, 6]` alongside `action_seq`. |
| 23 | +Reading the consumer showed those numbers are **relative frame-count weights** |
| 24 | +(`weight / total_weight * num_frames`); the actual motion magnitude came from a default |
| 25 | +keyword argument elsewhere in the call chain. |
| 26 | + |
| 27 | +**Trace every control number to the function that consumes it.** The same repo had a second |
| 28 | +entry point where a similarly-named argument *was* a magnitude — see trap 7. |
| 29 | + |
| 30 | +## 3. Duration units differ, and the unit is often not "frame" → C1 |
| 31 | + |
| 32 | +Observed units for the same-looking `<action>-<number>` syntax: |
| 33 | +pixel frames · **latents** (×4 pixel frames) · fixed-length action blocks · 2-second text |
| 34 | +lines · autoregressive chunks. |
| 35 | + |
| 36 | +`w-31` meaning 31 latents = 124 frames, not 31 frames, is a 4× duration error. |
| 37 | + |
| 38 | +## 4. README fps ≠ config fps ≠ writer fps → C2 |
| 39 | + |
| 40 | +Observed in the same repo: README says 25 FPS, config constant says `sample_fps = 16`, |
| 41 | +and the actual `export_to_video(..., fps=17)` call writes 17. Wall-clock duration is set by |
| 42 | +the **writer**; the config value may only affect training-time conditioning. |
| 43 | + |
| 44 | +**Confirmed empirically**: one model whose README documents 24 fps produced benchmark videos |
| 45 | +at **19 fps** (1145 frames ≈ 60.3 s). Had the plan trusted the README, the requested duration |
| 46 | +would have been wrong by 26%. |
| 47 | + |
| 48 | +Record all values you find, state which governs duration, and **measure the produced file** |
| 49 | +rather than trusting any of them. |
| 50 | + |
| 51 | +## 5. Absolute motion scale may be discarded before conditioning → D3 |
| 52 | + |
| 53 | +Observed in more than one repo: per-frame translations are divided by their max norm before |
| 54 | +being turned into the geometric condition. Consequence: a trajectory at speed 0.05/frame and |
| 55 | +one at 0.5/frame produce **identical** conditioning if the shape is the same. |
| 56 | + |
| 57 | +Where this happens, **absolute speed is not controllable** through the geometry path — only |
| 58 | +trajectory shape and *relative* speed variation within a clip. If your benchmark claims |
| 59 | +matched speeds across models, this must be disclosed. Some repos feed the raw action vector |
| 60 | +through a second path, which partially restores magnitude information; check whether both |
| 61 | +paths exist. |
| 62 | + |
| 63 | +## 6. First chunk length ≠ later chunk length → C4 |
| 64 | + |
| 65 | +Observed: first clip 57 frames, every subsequent clip 40. Total is |
| 66 | +`57 + (N-1)*40`, not `N*40`. Segments therefore cannot all be exactly equal; pick a policy |
| 67 | +and record it (see `output_spec.md`). |
| 68 | + |
| 69 | +## 7. Two code paths, same-looking control, different semantics → A3 |
| 70 | + |
| 71 | +Observed: one repo ships an autoregressive-forcing entry point where the per-segment number |
| 72 | +is a frame-count weight, **and** a separate helper where the analogous number is a magnitude |
| 73 | +with a fixed 33-frame duration, **and** the two use different hardcoded intrinsics. |
| 74 | + |
| 75 | +Determine which path the launch script actually calls. Never mix constants across paths. |
| 76 | + |
| 77 | +## 8. Repo bugs that only fire in multi-GPU or looped use → F1, F5 |
| 78 | + |
| 79 | +Observed: |
| 80 | +- a keyword-argument name mismatch in the sequence-parallel attention wrapper — single-GPU |
| 81 | + path never touches it, any multi-GPU run crashes immediately; |
| 82 | +- `destroy_process_group()` + `exit()` at the end of the per-sample generate function — a |
| 83 | + batch loop renders exactly one sample and the process exits with no error; |
| 84 | +- a pinned dependency incompatible with the installed torch, which only surfaces at import. |
| 85 | + |
| 86 | +Fix in `patches/`, and check the patch is applied (`git diff --stat`) at the start of every |
| 87 | +run rather than assuming. |
| 88 | + |
| 89 | +## 9. Intrinsics: classify before you inject → E1–E3 |
| 90 | + |
| 91 | +Observed hardcoded values implying FOVs of ~63°, ~79°, ~89° and exactly 90° across different |
| 92 | +repos, some normalised and some in pixels, some rescaled internally against a **fixed |
| 93 | +reference resolution** rather than the actual working one. |
| 94 | + |
| 95 | +Two symmetric mistakes: |
| 96 | + |
| 97 | +- **Injecting where nothing is wanted.** Only a genuinely camera-conditioned model |
| 98 | + (case 4 in `output_spec.md §4`) consumes `arena_inputs/*_intrinsics/`. A model with no |
| 99 | + camera model, a hardcoded/derived virtual pinhole, or its own runtime estimator needs |
| 100 | + nothing from us — do not manufacture an injection point, and do not patch a hardcoded |
| 101 | + constant unless the user asks. |
| 102 | +- **Passing ours through raw.** Our values are in **original-image pixels** |
| 103 | + (`cx ≈ (W-1)/2`). Every model resizes first, and often changes aspect ratio. Scale and crop |
| 104 | + the intrinsics to match, and check whether the repo wants pixels or normalised values, and |
| 105 | + against which resolution. |
| 106 | + |
| 107 | +Either way, record the source FOV and the model's effective FOV. Our data spans **22°–90°** |
| 108 | +horizontal, so a fixed-FOV model is mismatched for most samples; that is a disclosable caveat, |
| 109 | +not something to silently fix. |
| 110 | + |
| 111 | +## 10. Motion may be eased, not step-constant → D4 |
| 112 | + |
| 113 | +Observed: target velocity approached by exponential smoothing with separate press/release |
| 114 | +time constants (~0.45 s / ~1.0 s). Even with an exactly correct frame count, the first |
| 115 | +fraction of a second of each segment is still accelerating and the previous action is still |
| 116 | +coasting. |
| 117 | + |
| 118 | +Consequence for evaluation: segment boundaries are not sharp. Prefer discarding a fixed |
| 119 | +transition window at the start of each segment **for all models**, so the eased model is not |
| 120 | +penalised relative to step-constant ones. |
| 121 | + |
| 122 | +## 11. "It uses X, so it must not use Y" → G1 |
| 123 | + |
| 124 | +A conclusion of the form "this model is action-conditioned, therefore it has no camera |
| 125 | +geometry" was asserted and turned out to be **wrong**: the repo integrated the action stream |
| 126 | +into camera poses and built a ray-map condition, *in addition* to a separate action module. |
| 127 | + |
| 128 | +Injection mechanisms are not mutually exclusive. One repo used an additive ray-map embedding |
| 129 | +**and** a projection-style camera position encoding inside attention. Grep for all of them |
| 130 | +before characterising a model, and prefer `UNVERIFIED` over a tidy story. |
| 131 | + |
| 132 | +## 12. Example files are ground truth; README prose is a hint |
| 133 | + |
| 134 | +Where a repo ships an example input, load it and print shapes, dtypes and a few values. |
| 135 | +Observed disagreements between README description and shipped file, and one case where the |
| 136 | +example's stated image dimensions were only a reference for intrinsics, not the output |
| 137 | +resolution. |
| 138 | + |
| 139 | +## 13. Package name ≠ import name; installing one thing un-installs another |
| 140 | + |
| 141 | +Environment hazards seen repeatedly: a drop-in replacement package that provides the same |
| 142 | +import name but a different distribution name, so the dependency resolver pulls the original |
| 143 | +back in; a headless variant of a library being silently replaced by the GUI variant as a |
| 144 | +transitive dependency of an unrelated install. |
| 145 | + |
| 146 | +After any install into a model env, re-check the packages your run depends on rather than |
| 147 | +assuming the env is still good. |
| 148 | + |
| 149 | +## 14. Prompts are per (view, domain, image) and not interchangeable → H1 |
| 150 | + |
| 151 | +There are four caption files — `{first_view,third_view}/prompt_{real,style}.txt`, 25 rows each. |
| 152 | +Every `first_view` caption is prefixed `First-person view.` and every `third_view` one |
| 153 | +`Third-person view.`, so the caption itself encodes the camera setup. |
| 154 | + |
| 155 | +Failure modes seen or easily reachable: |
| 156 | + |
| 157 | +- loading one prompt file and reusing it across views → the model is told "third-person" while |
| 158 | + being handed a first-person image; |
| 159 | +- off-by-one row indexing → every video gets its neighbour's caption, which is nearly invisible |
| 160 | + in spot checks and poisons any text-alignment metric; |
| 161 | +- an adapter "improving" or regenerating captions → the text then carries scene information the |
| 162 | + benchmark never supplied, making that domain incomparable. |
| 163 | + |
| 164 | +Use `{view}/prompt_{domain}.txt` row `i` for image `i`, verbatim, and record |
| 165 | +`prompt.source` as `file:row` in the manifest so it is auditable. |
| 166 | + |
| 167 | +Note the zip's own `README.txt` is stale here (it claims only `third_view` has prompts) — see |
| 168 | +trap 12. |
| 169 | + |
| 170 | +## 15. Inputs are not uniform in size or aspect ratio → E5, H3 |
| 171 | + |
| 172 | +Measured: `real` images span 1024×768 … 1920×1200 (AR 1.33–1.79); `style` is uniformly |
| 173 | +1024×1024 (AR 1.00). Target models generate at ~1.7–1.8 AR. |
| 174 | + |
| 175 | +So every sample undergoes a non-trivial resize, and square `style` inputs need a real |
| 176 | +crop/pad decision. An adapter that hardcodes one scale factor, or assumes a fixed input size, |
| 177 | +will be wrong for most of the suite. Decide one policy per model, apply it uniformly, document |
| 178 | +it. |
0 commit comments