Rebuild isolated Level 0 as realistic BPE nanoGPT baseline

level_0_baseline/README.md

@@ -1,76 +1,132 @@
-# Level 0 Baseline
+# Realistic isolated Level 0 nanoGPT baseline
 
-A deliberately self-contained nanoGPT baseline. It does not import the repository's existing experiment framework or WW-PGD code.
+This subtree is intentionally independent of the repository's WW-PGD experiment framework. It provides a clean AdamW baseline on natural-language data while retaining deterministic WeightWatcher alpha measurements.
 
-## Scope
+## What Level 0 means here
 
-- one transformer block, one ordinary Q/K/V attention head, width 64, context 256
-- byte-level next-token language modeling on a fixed FineWeb-Edu subset
-- AdamW or Muon with a global warmup/cosine schedule
-- Muon applies only to hidden 2-D matrices; AdamW handles embeddings, tied LM head, LayerNorm parameters, and other non-matrix parameters
-- deterministic seeds for initialization and sampled training windows
-- immutable train, validation, and test splits
-- CSV logging of loss, next-token accuracy, perplexity, validation/test generalization gaps, gradient norm, weight norm, tokens, and elapsed time
-- optional checkpoint-time WeightWatcher layer analysis
-- single-seed and multi-seed notebooks; multi-seed plots use mean ± one standard deviation shaded bands
+The default MacBook preset is designed for an Apple M2 Pro with 16 GB unified memory:
 
-## Install
+- GPT-2 BPE tokenization (`tiktoken:gpt2`), padded model vocabulary 50,304
+- four transformer blocks, four attention heads, width 256
+- context length 256 BPE tokens
+- approximately 16.1 million trainable parameters
+- AdamW with standard matrix/no-matrix weight-decay groups
+- 200-step linear warmup followed by cosine decay
+- peak learning rate `6e-4`, minimum learning rate `6e-5`
+- weight decay `0.1`, betas `(0.9, 0.95)`, gradient clipping `1.0`
+- batch size 8 with four gradient-accumulation steps
+- 5,000 optimizer steps, or 40.96 million processed training tokens
+- fixed train and validation probes with independent RNG streams
+- test evaluation only at the final and validation-selected checkpoints
+- deterministic non-randomized WeightWatcher alpha measurements every 500 steps
+
+This replaces the obsolete one-block, width-64, raw-byte experiment. Old `/tmp/nanogpt-level0/data` byte files are rejected by the new trainer.
+
+## Conda installation
 
 ```bash
-cd level_0_baseline
-python -m venv .venv
-source .venv/bin/activate
-pip install -e '.[data,analysis,test]'
+conda activate ww_prod310
+cd ~/Desktop/work/nanoGPT/nanogpt-experiments/level_0_baseline
+python -m pip install -e '.[data,analysis,test]'
+python -m pip check
 ```
 
 ## Paths
 
-Defaults are under `/tmp/nanogpt-level0`. Override them without editing code:
+All large artifacts remain under `/tmp` by default:
 
 ```bash
-export NANOGPT_LEVEL0_DATA_ROOT=/tmp/my-level0/data
-export NANOGPT_LEVEL0_RESULTS_ROOT=/tmp/my-level0/results
-export NANOGPT_LEVEL0_CACHE_ROOT=/tmp/my-level0/cache
+export NANOGPT_LEVEL0_ROOT=/tmp/nanogpt-level0-bpe
+export NANOGPT_LEVEL0_DATA_ROOT=$NANOGPT_LEVEL0_ROOT/data
+export NANOGPT_LEVEL0_RESULTS_ROOT=$NANOGPT_LEVEL0_ROOT/results
+export NANOGPT_LEVEL0_CACHE_ROOT=$NANOGPT_LEVEL0_ROOT/cache
 ```
 
-## Prepare the real corpus
+## Prepare the pinned FineWeb-Edu corpus
 
-This prepares fixed 50 MB training, 2 MB validation, and 2 MB test byte-token splits from streamed FineWeb-Edu:
+The default preparation writes 20 million training tokens and one million tokens each for validation and test. Splits are fixed, atomic, and document-disjoint at boundaries.
 
 ```bash
-level0-prepare-data --dataset fineweb-edu
+./scripts/prepare_data.sh 2>&1 | tee /tmp/level0-bpe-prepare.log
 ```
 
-To monitor the streamed download and preparation, enable heartbeat logging:
+Equivalent direct command:
 
 ```bash
 level0-prepare-data \
   --dataset fineweb-edu \
+  --output-dir /tmp/nanogpt-level0-bpe/data \
+  --train-tokens 20000000 \
+  --val-tokens 1000000 \
+  --test-tokens 1000000 \
+  --tokenizer gpt2 \
+  --model-vocab-size 50304 \
   --verbose \
   --log-interval-seconds 10
 ```
 
-Verbose output reports documents processed, bytes collected, completion percentage, elapsed time, average throughput, estimated time remaining, and how long the stream has produced no new bytes. The heartbeat continues while the streaming iterator is blocked, making a network or dataset stall visible.
+The progress heartbeat reports dataset resolution, current split, documents, tokens, throughput, ETA, and time since the last new tokens arrived.
+
+## Run AdamW seed 1337
+
+```bash
+./scripts/run_one.sh adamw 1337 mps \
+  2>&1 | tee /tmp/level0-bpe-adamw-seed1337.log
+```
+
+The script resumes automatically when `checkpoint_latest.pt` exists. To deliberately discard a prior run:
 
-## Run one seed
+```bash
+NANOGPT_LEVEL0_OVERWRITE=1 ./scripts/run_one.sh adamw 1337 mps
+```
+
+For a bounded CPU smoke test:
 
 ```bash
-./scripts/run_one.sh adamw 1337
-./scripts/run_one.sh muon 1337
+NANOGPT_LEVEL0_MAX_STEPS=2 \
+NANOGPT_LEVEL0_BATCH_SIZE=2 \
+NANOGPT_LEVEL0_GRAD_ACCUM_STEPS=1 \
+NANOGPT_LEVEL0_EVAL_INTERVAL=1 \
+NANOGPT_LEVEL0_DISABLE_WEIGHTWATCHER=1 \
+NANOGPT_LEVEL0_OVERWRITE=1 \
+./scripts/run_one.sh adamw 1337 cpu
 ```
 
-## Run multiple seeds
+## Output contract
+
+Each run writes:
+
+- `manifest.json`: exact model, optimizer, data identity, fixed-probe hashes, and protocol
+- `metrics.csv`: train/validation loss, perplexity, bits per token, accuracy, gap, LR, gradient norm, weight norm, and throughput
+- `checkpoint_latest.pt`: resumable training state
+- `checkpoint_best.pt`: validation-selected model
+- `checkpoint_final.pt`: final model
+- `final_metrics.json`: final-checkpoint test metrics
+- `selected_checkpoint_metrics.json`: validation-selected checkpoint test metrics
+- `weightwatcher_step_*.csv`: per-matrix alpha, D, xmin, and tail metadata when available
+- `run_complete.json`: transactional completion marker
+- `train.log`: persistent progress log
+
+Test data is not evaluated during training. It is touched only after optimization completes, once for the final checkpoint and once for the validation-selected checkpoint.
+
+## Plot one run
 
 ```bash
-NANOGPT_LEVEL0_SEEDS=1337,2027,4099 ./scripts/run_multiseed.sh
+export NANOGPT_LEVEL0_RESULTS_ROOT=/tmp/nanogpt-level0-bpe/results
+export NANOGPT_LEVEL0_NOTEBOOK_OPTIMIZER=adamw
+export NANOGPT_LEVEL0_NOTEBOOK_SEED=1337
+jupyter lab notebooks/01_single_seed.ipynb
 ```
 
-The notebooks read `NANOGPT_LEVEL0_RESULTS_ROOT`. Select the single-seed run with `NANOGPT_LEVEL0_NOTEBOOK_OPTIMIZER` and `NANOGPT_LEVEL0_NOTEBOOK_SEED`.
+The notebook plots loss, perplexity, exact next-BPE-token accuracy, bits per token, optimization diagnostics, and WeightWatcher alpha trajectories. It also saves PNG files under the run's `plots/` directory.
 
-For a bounded infrastructure smoke test, override the run length and batch size:
+## Multiple seeds
 
 ```bash
-NANOGPT_LEVEL0_MAX_STEPS=2 NANOGPT_LEVEL0_BATCH_SIZE=2 NANOGPT_LEVEL0_EVAL_INTERVAL=1 ./scripts/run_one.sh adamw 1337
+NANOGPT_LEVEL0_SEEDS=1337,2027,4099 \
+NANOGPT_LEVEL0_OPTIMIZERS=adamw \
+NANOGPT_LEVEL0_DEVICE=mps \
+./scripts/run_multiseed.sh
 ```
 
-Next-token error is `1 - next-token accuracy`; the notebooks derive and plot it explicitly.
+Then run `notebooks/02_multiseed.ipynb` for mean and standard-deviation bands.

level_0_baseline/configs/level0.yaml

@@ -1,33 +1,40 @@
 model:
-  vocab_size: 256
+  # MacBook-scale GPT baseline: large enough for meaningful BPE language
+  # modeling and WeightWatcher spectra, while remaining practical on an
+  # M2 Pro with 16 GB unified memory.
+  vocab_size: 50304
   block_size: 256
-  n_layer: 1
-  n_head: 1
-  n_embd: 64
+  n_layer: 4
+  n_head: 4
+  n_embd: 256
   dropout: 0.0
   bias: false
+
 training:
-  batch_size: 16
-  grad_accum_steps: 1
-  max_steps: 2000
-  eval_interval: 50
+  batch_size: 8
+  grad_accum_steps: 4
+  max_steps: 5000
+  eval_interval: 100
   eval_batches: 20
-  checkpoint_interval: 250
+  log_interval: 10
+  checkpoint_interval: 500
   learning_rate: 0.0006
-  muon_learning_rate: 0.02
-  muon_aux_adamw_learning_rate: 0.0006
   min_lr: 0.00006
-  warmup_steps: 100
+  warmup_steps: 200
   weight_decay: 0.1
   beta1: 0.9
   beta2: 0.95
+  epsilon: 1.0e-8
   grad_clip: 1.0
   optimizer: adamw
+  muon_learning_rate: 0.02
+  muon_aux_adamw_learning_rate: 0.0006
   muon_momentum: 0.95
   muon_nesterov: true
   seed: 1337
   compile: false
+
 analysis:
   weightwatcher: true
-  weightwatcher_interval: 250
-  randomize: true
+  weightwatcher_interval: 500
+  randomize: false