Learn world models by building them: from the intuition behind latent dynamics to a working simulation, planning, and evaluation system.
Caution
Structured learning path with lecture and project cards.
Concept-first explanations with mermaid diagrams and background callouts for deep-learning readers.
Seven architecture families, three planning mechanisms, side-by-side comparison tables.
Five lectures and six projects that take you from the intuition behind world models to training, evaluating, and causally probing modern world-model systems.
| # | Type | Title | Core Topics |
|---|---|---|---|
| L01 | Lecture | Internal Simulation & Historical Context | Craik's mental models, predictive coding, four eras of world model evolution |
| L02 | Lecture | Observation Encoding & Latent Dynamics | VAE, CNN encoder, ELBO, GRU → MDN-RNN → RSSM |
| L03 | Lecture | Architecture Patterns, Learning Paradigms & Planning | Seven architecture families, CEM-MPC, latent Actor-Critic, TD-MPC |
| L04 | Lecture | Evaluation by World Model | FID, reward correlation, consistency loss, PSNR, horizon drift |
| L05 | Lecture | Frontier Debates | Language vs physical grounding, Bitter Lesson, AGI as a research target |
| P01 | Project | Train a VAE Encoder | Small CNN VAE on 64×64 pixels; ELBO loss curve; latent slider visualization |
| P02 | Project | Build an RSSM Dynamics Model | GRU, MDN-RNN, and RSSM compared; prior vs posterior rollout plots |
| P03 | Project | Train a Dreamer Agent | Full training loop: encoder + RSSM + latent Actor-Critic on a small pixel env |
| P04 | Project | Swap the Dynamics Backbone | Replace RSSM with a small causal Transformer (STORM-style); architecture comparison |
| P05 | Project | World Model Evaluation Dashboard | Per-model metrics side by side: FID, reward correlation, PSNR, latent drift |
| P06 | Project | Counterfactual Action-Conditioned World Model | Interventional and counterfactual rollouts, inverse-dynamics regularization, action-influence metric |
flowchart TD
L01["L01 History and Intuition"] --> L02A
L02A["L02 Part A: VAE Encoder"] --> P01["P01 Train VAE, visualize latent space"]
L02A --> L02B["L02 Part B: GRU to RSSM"]
L02B --> P02["P02 Build RSSM, compare prior vs posterior"]
L02B --> L03A["L03 Part A: Architecture Patterns"]
L03A --> L03B["L03 Part B: Planning mechanisms"]
L03B --> P03["P03 Train Dreamer agent"]
P02 --> P04["P04 Swap RSSM for Transformer backbone"]
L03A --> P04
P03 & P04 --> L04["L04 Evaluation metrics"]
L04 --> P05["P05 Evaluation dashboard"]
P03 & P04 --> P06["P06 Counterfactual action-conditioned WM"]
P05 --> L05["L05 Frontier Debates"]
P06 --> L05
Suggested path: L01, L02, P01, P02, L03, P03, P04, L04, P05, P06, L05
You do not need to finish all theory before starting a project. Build, then come back with questions.
npm install
npm run docs:dev # dev server with hot reload
npm run docs:build # production build
npm run docs:preview # preview built siteTo refresh the README screenshots after a build:
npm run docs:build
npm run screenshots:readmelearn-world-model/
├── docs/ # VitePress documentation site
│ ├── .vitepress/config.mts # nav and sidebar (EN + ZH)
│ ├── en/lectures/ # 5 English lecture pages
│ ├── zh/lectures/ # 5 Chinese lecture pages
│ ├── en/projects/ # 6 English project pages
│ └── zh/projects/ # 6 Chinese project pages
├── external/world-model-tutorial/ # PyTorch source referenced by projects
│ └── references.md # four-era history and architecture survey
├── scripts/ # build utilities (screenshots, PDF)
└── package.json
Scan the QR code to join the WeChat discussion group (微信交流群):
Contributions are welcome. Before submitting a pull request, read CLAUDE.md for the writing style rules that apply to all lecture and project files (no em dashes, no linear mermaid diagrams, no arrow-chain prose, EN/ZH sync, and others). Content that does not follow those rules will be asked to revise before merging.
| Name | Role | Affiliation | GitHub |
|---|---|---|---|
| Zhimin Zhao | Project Lead | Queen's University | @zhimin-z |
| Qi Wang | Project Lead | Chinese Academy of Sciences | @qiwang067 |