Skip to content

README.md

Latest commit

 

History

History
159 lines (126 loc) · 7.45 KB

README.md

File metadata and controls

159 lines (126 loc) · 7.45 KB
Raw

SolarWM Data Engine

The open data pipeline for SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

Project page · Dataset · Model weights · Paper

The engine turns raw video and multi-view scenes into camera-controllable world-model training data. Each clip carries a metric-scale 6-DoF camera trajectory, per-frame intrinsics, a scene-static caption, quality metadata, selection decisions, and provenance aligned to the same frame indices.

This is the code that produced the released SolarWM corpus: 1,425,694 canonical clips from 14 datasets, totaling 25.85 TB across 29,272 shards. 876,593 clips pass the released selection policy — 471,798 in the high tier and 404,795 in xhigh — and the remaining 549,101 are published as well, fully annotated, each with the machine-readable reasons it was turned down.

This standalone repository is also the data-engine submodule of SolarWM. The main project provides training and inference; the SolarWM-Data repository provides release controls and the annotation reconstruction package.

An engine, not a list of clips

Publishing a corpus as a fixed list of accepted clips throws away the expensive part. Camera estimation, captioning and the quality models cost a GPU pass per clip; selection is only metadata computed from what they produced. So this engine annotates everything first and decides afterwards — nothing is discarded for failing a threshold.

That inverts what a change costs. Raising a quality floor, dropping a metric you do not trust, rebalancing the mixture, holding out a different split, or cutting a different window length for a different backbone is an index pass over metadata, not another run across the fleet. The released high and xhigh tiers are one convenient policy over the corpus, not a definition of which clips are valid.

Four layers stay separable, and changing one never rebuilds the one below it:

Layer What it fixes Cost to change
Physical corpus canonical clips, annotations, metrics, verdicts a GPU pass per clip
Logical recipe split membership, tiers, source weights, repeats an index
Model view the fixed-length windows a backbone reads an index
Packed shards WebDataset tars, partitioned by tier a copy

What the corpus contains

Fourteen independently addressable dataset owners, spanning real-world, synthetic and game environments. All 1,425,694 clips were captioned in one pass by Kimi-K2.6 under a single frozen prompt and response contract, so caption style does not vary by source.

Dataset owner All High xhigh Rejected
ABOT 30,966 127 30,715 124
DL3DV-10s 120,924 54,528 60,396 6,000
DL3DV-60s 10,077 3,578 6,065 434
MIND 533 117 402 14
MiraData 140,877 3,683 17,806 119,388
MiraData-Clean 135,224 12,865 5,740 116,619
MultiCamVideo 123,117 89,587 5,369 28,161
OmniWorld 19,632 3,773 13,552 2,307
RealCam 45,697 16,154 16,074 13,469
Sekai-Game 2,550 537 1,410 603
Sekai-Walking 22,990 6,054 12,976 3,960
Sekai-Walking-Clean 109,248 46,831 33,660 28,757
SpatialVID 365,345 100,815 127,180 137,350
SpatialVID-Clean 298,514 133,149 73,450 91,915
Total 1,425,694 471,798 404,795 549,101

Three of those owners are Clean Plate derivatives — 542,986 clips in which people and vehicles are removed while scene layout and camera motion are retained. Dynamic actors add motion the camera does not explain and the user cannot control, so a clean sibling is a different kind of supervision, not a better version of its source: each is its own recipe owner, scored from scratch rather than inheriting its source's metrics.

What one clip looks like

<corpus>/<source>/<clip_id>/
├── video.mp4         # the clip, at the spec's frame rate
├── poses.npy         # (N, 4, 4) float64 — camera-to-world, METRIC scale
├── intrinsics.npy    # (N, 4) float64 — per-frame (fx, fy, cx, cy) in pixels
├── prompt.txt        # scene-static dense caption
├── meta.json         # source, metrics, verdict + reasons, provenance
└── gt_depth.npz      # only where the source ships metric GT depth

Three properties hold or the clip is not usable, and none of them are visible by watching it: frame_count == len(poses) == len(intrinsics) with index i meaning the same instant in each; poses.npy is camera-to-world, so the camera centre is the translation column M[:3,3] directly, not -Rᵀt; and translations are metric, recovered per clip. scripts/validate_clip.py checks the first and is worth running on any source whose extraction or pose code you touch.

Install

pip install -e ".[full]"            # from this directory
solarwm-pipeline spec list          # the output specs, and which one is active
python3 scripts/validate_dryrun.py  # the whole pipeline on synthetic input, no GPU

That much is pure Python and needs no GPU. Producing real annotations additionally needs a CUDA PyTorch environment, the external tools fetched by scripts/setup_*.sh, and dry_run: false in configs/models.yaml — the shipped default emits placeholders that pass every structural check. Copy .env.example, fill it in, and source it; no credential is baked into the code.

Documentation

The detailed pipeline guide is skills/solarwm-data-pipeline/SKILL.md. It covers:

  • adding your own raw source, from probing its layout to wiring it in
  • running the fleet across a worker pool and keeping it busy
  • the output specs, the shipped sources, and how pose modes are chosen
  • calibrating quality thresholds against your data rather than ours
  • verifying that what came out is structurally, geometrically and visually correct
  • assembling recipes, model views and packed shards
  • the invariants that must not change, and what to check when something is wrong

Citation

If you use SolarWM-Data, the data engine, or the released models in your research, please cite our paper.

Paper: https://arxiv.org/abs/2609.02886

@misc{huang2026solarwmopendatascalable,
      title={SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models}, 
      author={Junchao Huang and Guian Fang and Shengju Qian and Xianghao Kong and Zhuoran Zhao and Wei Huang and Yihua Du and Zixin Zhang and Justin Cui and Yuchao Gu and Yukang Chen and Xinting Hu and Tianyu He and Shaoshuai Shi and Zhuotao Tian and Xin Wang and Mike Zheng Shou and Li Jiang},
      year={2026},
      eprint={2609.02886},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.02886}, 
}

Acknowledgements

The camera path combines Pi3X structure, MoGe-2 metric depth, and a VIPE SLAM backend with per-frame intrinsics. We thank those projects and SANA-WM for the work this implementation builds on. See NOTICE for complete attribution and license information.

Licence

Apache-2.0 — see LICENSE. The external models this engine drives, and the datasets it annotates, each keep their own terms; several are non-commercial. See NOTICE.