docs(tutorials): add Tokenization cheat sheet (#29, bilingual) + runnable tokenization.py (#30)

New tutorial #29 — Tokenization / 分词器 (ZH + EN, ~875 lines each), filling the
foundation the collection had been taking for granted:

- pipeline mental model (chat template / special recognition upstream;
  normalization → pre-tokenization → segmentation model → IDs → post-processing)
- BPE / WordPiece / Unigram with worked examples — training ≠ encoding as the
  spine (frozen-rank encoding, longest-match whole-word [UNK], forward vs
  Viterbi DP with hand-verified 0.22350625 marginal)
- byte-level BPE vs SentencePiece engineering (bytes_to_unicode, Ġ vs ▁,
  nmt_nfkc, byte_fallback ≠ byte-level BPE; Unicode traps; reversibility)
- vocab facts: the five quantities (mergeable/special/len/max-ID+1/embedding
  rows) across BERT/GPT-2/cl100k/o200k/Llama-2/3/Qwen/Gemma, incl. cl100k ID
  holes and size-dependent Qwen padding (151,936 vs 152,064)
- corpus design, vocab-size ledger, multilingual fertility, vocab extension
  (lexical vs control paths, mean-init, regression suite)
- production traps: digits, code whitespace, glitch tokens, prefix
  instability + token healing, special-token injection (two-channel encoding),
  offset mapping, tokenizer versioning; inference-stack alignment table
- tokenizer-free frontier: ByT5 / MegaByte / BLT (+CANINE contrast)
- 25 interview Q&A (L1×8 / L2×10 / L3×7)

Runnable code/tokenization.py (pure stdlib, 8 demos, ~40 asserts): exact GPT-2
bytes_to_unicode (188 identity + 68 shifted pinned), BPE ordered-merge trainer
+ frozen-rank encoder (rank-reversal + non-overlap pinned), WordPiece greedy
dead-end, Unigram Viterbi vs forward, NFKC loss vs byte-fallback round-trip,
prefix-instability counterexample, fertility/BPB with independent recovery,
vocab-extension regression (dense IDs, special atomicity, tied head). Refuses
python -O. Verified locally and on a remote box; §A output verbatim.

Review chain (10 GPT-5.6-sol threads, deepest for any tutorial so far, and the
first with a PRE-DRAFT design review): 81-claim design review before drafting;
math/facts, citations (18/18 drafter-flagged VERIFY markers confirmed), code,
and 25-answer batches; one dispute round where the reviewer RETRACTED its Qwen
vocab claim after counter-evidence; an ultra-effort final acceptance with two
fix-verification rounds → SHIP; render-fidelity round (caught the \|-in-table
renderer footgun) → PASS; EN translation-fidelity 2 rounds → PASS. Corrections
flowed both ways: reviewer caught 6 hard factual errors, orchestrator overturned
one reviewer claim.

READMEs: count 28→29, What's New entry, General-bucket catalog row (ZH+EN).
Gate: tools/verify_reviews.py --mode strict --reproduce → 59 OK / 0 FAIL / PASS.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: wanshuiyin

README.md

@@ -8,7 +8,7 @@
 >
 > 📖 **中文版 (Chinese version)**: [README_CN.md](README_CN.md)
 
-**📚 Jump to a topic** — 28 first-party cheat sheets across 7 categories + 1 community-contributed category:
+**📚 Jump to a topic** — 29 first-party cheat sheets across 7 categories + 1 community-contributed category:
 
 [🧠 General / Foundations](#-general--foundations) · [🎯 Post-Training & Reasoning](#-post-training--reasoning) · [🏛️ LLM Architecture & Systems](#-llm-architecture--systems) · [🌊 Generative Models — Theory & Tokenizers](#-generative-models--theory--tokenizers) · [🎨 Generation Systems (Image / Video / 3D / Diffusion Post-Training)](#-generation-systems--image--video--3d--diffusion-post-training) · [👁️ Multimodal](#-multimodal) · [🤖 Agents](#-agents) · [🦾 Embodied AI / 具身智能](#-embodied-ai--具身智能)
 
@@ -66,6 +66,7 @@ Phone on the subway, iPad at a café, laptop in the library — same HTML link o
 
 ## 📢 What's New
 
+- **2026-07-11** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) 🔤 **#29 Tokenization cheat sheet** — the foundation the whole collection had been taking for granted: the **pipeline mental model** (chat template / special-token recognition upstream · normalization → pre-tokenization → segmentation model → IDs → post-processing) · **BPE / WordPiece / Unigram** (training ≠ encoding — the #1 hand-coding trap) · **byte-level BPE vs byte fallback** (`Ġ` vs `▁`) · **the five vocab quantities** (mergeable · special · len(tokenizer) · max-ID+1 · embedding rows — cl100k has ID holes, Qwen pads by checkpoint size) · **fertility / BPB metrics** · **vocab extension** (lexical vs control paths) · **production traps** (prefix instability / token healing · special-token injection · glitch tokens · offset mapping) · **tokenizer-free** (ByT5 / MegaByte / BLT). **Bilingual** (中文 + EN), with a runnable script ([`code/tokenization.py`](docs/tutorials/code/tokenization.py) — exact GPT-2 `bytes_to_unicode`, frozen-rank BPE encoding, both Unigram DPs (forward 0.22350625 hand-checked), vocab-extension regression; pure stdlib, verified on a real box) and 25 高频题. First tutorial with cross-model review moved UP to the design stage (81 GPT-5.6-sol guardrails before drafting + 5 review batches + 1 dispute round + an ultra final SHIP — corrections flowed both ways: the reviewer caught 6 hard errors, the orchestrator overturned the reviewer's Qwen vocab claim). [`tokenization_tutorial.html`](docs/tutorials/tokenization_tutorial.html).
 - **2026-06-29** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) ⚙️ **#28 Optimizers & LR Schedules cheat sheet** — the optimizer/schedule training-mechanics gap: **SGD·Momentum·Nesterov** · **AdaGrad→RMSProp→Adam** (the adaptive lineage) · **Adam bias correction** (the uncorrected first step is ~3.16× too *large*, not small — the direction trap) · **AdamW decoupled weight decay** (Adam's L2 ≠ weight decay) · **frontier** (Muon / Lion / Shampoo / SOAP / Adafactor / LAMB / Sophia) · **LR schedules** (warmup / cosine / Noam / WSD / one-cycle) · weight decay + LR-batch scaling + no-decay groups · grad clipping + LLM hyperparams (β₂=0.95). **Bilingual** (中文 + EN), with a runnable script ([`code/optimizer_lr_schedule.py`](docs/tutorials/code/optimizer_lr_schedule.py) — SGD/Adam/AdamW from-scratch vs `torch.optim`, the executable **AdamW≠Adam+L2** proof, the bias-correction direction, cosine-warmup, verified on a real box) and 25 高频题. [`optimizer_lr_schedule_tutorial.html`](docs/tutorials/optimizer_lr_schedule_tutorial.html).
 - **2026-06-19** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) 🧱 **#27 Normalization / Residual / Init cheat sheet** — the foundational training-mechanics hole: **BatchNorm / LayerNorm / RMSNorm / GroupNorm** · **Pre-vs-Post-LN** (Xiong's gradient argument — why Post-LN needs warmup) · **DeepNorm / Sandwich / QK-Norm** · **residual connections + scaling** (LayerScale / ReZero / GPT-2's 1/√(2N)) · **Xavier vs Kaiming** (the ReLU factor-2, and why the preserved quantity is the second moment E[y²], not Var) · **μP** (width-invariant HP transfer) · **Fixup / NFNets / DyT** (norm-free) · plus the **covariate-shift debunk** (Santurkar). **Bilingual** (中文 + EN), with a runnable script ([`code/normalization.py`](docs/tutorials/code/normalization.py) — from-scratch LN/RMSNorm vs `torch` + Pre/Post-LN gradient + Kaiming second-moment + GPT-2 residual scaling, verified on a real box) and 25 高频题. The cross-model review caught a real methodological confound in the Pre/Post-LN gradient demo (an output-normalization artifact) and redesigned it to a loss-robust top/bottom weight-grad ratio. [`normalization_init_tutorial.html`](docs/tutorials/normalization_init_tutorial.html).
 - **2026-06-18** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) ⚡ **#26 Linear / Sparse Attention cheat sheet** — the sub-quadratic / efficient-attention hole the collection kept referencing but never derived: **linear attention** (kernel φ + associativity → matrix-state RNN) · **SSM / Mamba** (selective S6) · **Mamba-2 / SSD** (1-semiseparable duality ≡ structured masked linear attention) · **DeltaNet / Gated DeltaNet** (overwrite update) · **chunkwise-parallel** training · **trainable sparse** (NSA three-branch / MoBA / Lightning / DSA) · **hybrid** (Jamba / Hymba / Qwen3-Next / Kimi-Linear / MiniMax-01). **Bilingual** (中文 + EN), with a runnable script ([`code/linear_sparse_attention.py`](docs/tutorials/code/linear_sparse_attention.py) — `chunkwise ≡ recurrent` equivalence + delta-rule + block-sparse, verified on a real box) and 25 高频题, settled through multi-batch Codex GPT-5.5 xhigh citation / math / code / answer / overall + render-fidelity review. [`linear_sparse_attention_tutorial.html`](docs/tutorials/linear_sparse_attention_tutorial.html) ([#22](https://github.com/wanshuiyin/ARIS-in-AI-Offer/pull/22) · [b84e913](https://github.com/wanshuiyin/ARIS-in-AI-Offer/commit/b84e913)).
@@ -113,6 +114,7 @@ Long-form technical blogs — hand-authored, cross-model reviewed; outside the a
 | **Attention Interview Cheat Sheet** | [📄 CN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/attention_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/attention_tutorial_en.html) | [MD](docs/tutorials/attention_tutorial.md) |
 | **Normalization / Residual / Init (BatchNorm / LayerNorm / RMSNorm / Pre-vs-Post-LN / DeepNorm / QK-Norm / Xavier·Kaiming / μP)** | [📄 CN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/normalization_init_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/normalization_init_tutorial_en.html) | [MD](docs/tutorials/normalization_init_tutorial.md) |
 | **Optimizers & LR Schedules (SGD·Momentum / Adam·AdamW / Muon·Lion·Shampoo·SOAP / warmup·cosine·WSD)** | [📄 CN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/optimizer_lr_schedule_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/optimizer_lr_schedule_tutorial_en.html) | [MD](docs/tutorials/optimizer_lr_schedule_tutorial.md) |
+| **Tokenization (BPE / WordPiece / Unigram·SentencePiece / byte-level·byte fallback / vocab·fertility·BPB / tokenizer-free)** | [📄 CN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/tokenization_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/tokenization_tutorial_en.html) | [MD](docs/tutorials/tokenization_tutorial.md) |
 | **KL Divergence in RLHF (k1/k2/k3 · placement gradient bias)** | [📄 CN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/kl_divergence_rlhf_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/kl_divergence_rlhf_tutorial_en.html) | [MD](docs/tutorials/kl_divergence_rlhf_tutorial.md) |
 
 ### 🎯 Post-Training & Reasoning

README_CN.md

@@ -8,7 +8,7 @@
 >
 > 📖 **English version (default)**: [README.md](README.md)
 
-**📚 按方向直达** —— 28 篇 first-party cheat sheet，7 大方向 + 1 个社区贡献方向：
+**📚 按方向直达** —— 29 篇 first-party cheat sheet，7 大方向 + 1 个社区贡献方向：
 
 [🧠 General / 基础](#-general--基础) · [🎯 Post-Training & Reasoning](#-post-training--reasoning) · [🏛️ LLM Architecture & Systems](#-llm-architecture--systems) · [🌊 Generative Models — 理论 & Tokenizers](#-generative-models--理论--tokenizers) · [🎨 Generation Systems（图像 / 视频 / 3D / Diffusion 后训练）](#-generation-systems--图像--视频--3d--diffusion-后训练) · [👁️ Multimodal](#-multimodal) · [🤖 Agents](#-agents) · [🦾 Embodied AI / 具身智能](#-embodied-ai--具身智能)
 
@@ -66,6 +66,7 @@
 
 ## 📢 最新动态
 
+- **2026-07-11** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) 🔤 **第 29 篇 Tokenization / 分词器 cheat sheet 上线** —— 补分词这个全集一直默认已知的地基：**pipeline 心智模型**（chat template/special 识别在上游 · normalization → pre-tokenization → 分词模型 → ID → post-processing）· **BPE / WordPiece / Unigram**（训练 ≠ 编码——最高频手写陷阱）· **byte-level BPE vs byte fallback**（`Ġ` vs `▁`）· **词表五量**（mergeable · special · len(tokenizer) · max-ID+1 · embedding 行——cl100k 有 ID 空洞、Qwen 按尺寸 padding）· **fertility / BPB 口径** · **扩词表**（lexical vs control 两条路）· **生产陷阱**（prefix instability / token healing · special 注入 · glitch token · offset mapping）· **tokenizer-free**（ByT5 / MegaByte / BLT）。**双语**（中文 + EN），含可跑脚本（[`code/tokenization.py`](docs/tutorials/code/tokenization.py) —— 精确复现 GPT-2 `bytes_to_unicode`、BPE 冻结 rank 编码、Unigram 双 DP（forward 0.22350625 手算核对）、扩词表 regression，纯标准库真机验证）和 25 高频题。本篇首次把跨模型审前置到设计阶段（GPT-5.6-sol 起草前给 81 条护栏 + 5 批审 + 1 轮对线 + ultra 终审 SHIP——双向纠错：审稿人抓 6 处硬错，orchestrator 反翻其 Qwen 词表口径一条）。[`tokenization_tutorial.html`](docs/tutorials/tokenization_tutorial.html)。
 - **2026-06-29** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) ⚙️ **第 28 篇 优化器 & LR Schedule cheat sheet 上线** —— 补优化器 / 调度训练机理这个洞：**SGD·Momentum·Nesterov** · **AdaGrad→RMSProp→Adam**（自适应谱系）· **Adam 偏差修正**（不修正首步是 ~3.16× **偏大**不是偏小——方向陷阱）· **AdamW 解耦权重衰减**（Adam 的 L2 ≠ weight decay）· **前沿**（Muon / Lion / Shampoo / SOAP / Adafactor / LAMB / Sophia）· **LR 调度**（warmup / cosine / Noam / WSD / one-cycle）· 权重衰减 + LR-batch 缩放 + no-decay 组 · 梯度裁剪 + LLM 超参（β₂=0.95）。**双语**（中文 + EN），含可跑脚本（[`code/optimizer_lr_schedule.py`](docs/tutorials/code/optimizer_lr_schedule.py) —— SGD/Adam/AdamW from-scratch vs `torch.optim`、可执行的 **AdamW≠Adam+L2** 证明、bias-correction 方向、cosine-warmup，真机验证）和 25 高频题。[`optimizer_lr_schedule_tutorial.html`](docs/tutorials/optimizer_lr_schedule_tutorial.html)。
 - **2026-06-19** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) 🧱 **第 27 篇 归一化 / 残差 / 初始化 cheat sheet 上线** —— 补 foundational 训练机理这个洞：**BatchNorm / LayerNorm / RMSNorm / GroupNorm** · **Pre-vs-Post-LN**（Xiong 梯度论证——Post-LN 为何需 warmup）· **DeepNorm / Sandwich / QK-Norm** · **残差连接 + 缩放**（LayerScale / ReZero / GPT-2 的 1/√(2N)）· **Xavier vs Kaiming**（ReLU 的因子 2，且守恒的量是二阶矩 E[y²] 而非 Var）· **μP**（宽度不变超参迁移）· **Fixup / NFNets / DyT**（归一化-free）· 以及 **covariate-shift 辨析**（Santurkar）。**双语**（中文 + EN），含可跑脚本（[`code/normalization.py`](docs/tutorials/code/normalization.py) —— from-scratch LN/RMSNorm vs `torch` + Pre/Post-LN 梯度 + Kaiming 二阶矩 + GPT-2 残差缩放，真机验证）和 25 高频题。跨模型审抓到 Pre/Post-LN 梯度 demo 的一个真方法学混淆（输出归一化假象），改用对 loss 稳健的顶/底梯度比。[`normalization_init_tutorial.html`](docs/tutorials/normalization_init_tutorial.html)。
 - **2026-06-18** — ![NEW](https://img.shields.io/badge/NEW-red?style=flat-square) ⚡ **第 26 篇 线性 / 稀疏注意力 cheat sheet 上线** —— 补上次二次 / 高效注意力这个全集到处引用却没系统讲的洞：**线性注意力**（kernel φ + 结合律 → 矩阵状态 RNN）· **SSM / Mamba**（selective S6）· **Mamba-2 / SSD**（1-半可分对偶 ≡ 结构化掩码线性注意力）· **DeltaNet / Gated DeltaNet**（改写式更新）· **chunkwise 并行**训练 · **可训练稀疏**（NSA 三分支 / MoBA / Lightning / DSA）· **混合架构**（Jamba / Hymba / Qwen3-Next / Kimi-Linear / MiniMax-01）。**双语**（中文 + EN），含可跑脚本（[`code/linear_sparse_attention.py`](docs/tutorials/code/linear_sparse_attention.py) —— `chunkwise ≡ recurrent` 等价 + delta-rule + block-sparse，真机验证）和 25 高频题，经多批 Codex GPT-5.5 xhigh 引用 / 数学 / 代码 / 答案 / 整体 + render 保真审定稿。[`linear_sparse_attention_tutorial.html`](docs/tutorials/linear_sparse_attention_tutorial.html)（[#22](https://github.com/wanshuiyin/ARIS-in-AI-Offer/pull/22) · [b84e913](https://github.com/wanshuiyin/ARIS-in-AI-Offer/commit/b84e913)）。
@@ -113,6 +114,7 @@
 | **Attention 面试 Cheat Sheet** | [📄 中](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/attention_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/attention_tutorial_en.html) | [MD](docs/tutorials/attention_tutorial.md) |
 | **归一化 / 残差 / 初始化 (BatchNorm / LayerNorm / RMSNorm / Pre-vs-Post-LN / DeepNorm / QK-Norm / Xavier·Kaiming / μP)** | [📄 中](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/normalization_init_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/normalization_init_tutorial_en.html) | [MD](docs/tutorials/normalization_init_tutorial.md) |
 | **优化器 & LR Schedule (SGD·Momentum / Adam·AdamW / Muon·Lion·Shampoo·SOAP / warmup·cosine·WSD)** | [📄 中](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/optimizer_lr_schedule_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/optimizer_lr_schedule_tutorial_en.html) | [MD](docs/tutorials/optimizer_lr_schedule_tutorial.md) |
+| **Tokenization / 分词器 (BPE / WordPiece / Unigram·SentencePiece / byte-level·byte fallback / 词表·fertility·BPB / tokenizer-free)** | [📄 中](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/tokenization_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/tokenization_tutorial_en.html) | [MD](docs/tutorials/tokenization_tutorial.md) |
 | **KL Divergence in RLHF (k1/k2/k3 · placement gradient bias)** | [📄 中](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/kl_divergence_rlhf_tutorial.html) | [📄 EN](https://wanshuiyin.github.io/ARIS-in-AI-Offer/tutorials/kl_divergence_rlhf_tutorial_en.html) | [MD](docs/tutorials/kl_divergence_rlhf_tutorial.md) |
 
 ### 🎯 Post-Training & Reasoning