Skip to content

Add Level 0 three-by-three epoch diagnostics notebook - #123

Draft
charlesmartin14 wants to merge 1 commit into
mainfrom
agent/level0-repeat-3x3-epoch-diagnostics
Draft

Add Level 0 three-by-three epoch diagnostics notebook#123
charlesmartin14 wants to merge 1 commit into
mainfrom
agent/level0-repeat-3x3-epoch-diagnostics

Conversation

@charlesmartin14

Copy link
Copy Markdown
Member

What changed

Adds level_0_wwpgd/notebooks/04_repeat_3x3_epoch_diagnostics.ipynb to rerun the original isolated Level 0 experiment as a matched three-seed comparison:

  • AdamW baseline: seeds 1337, 2027, 4099
  • AdamW + stock WWPGD: the same three seeds
  • frozen 4-block, 4-head, width-128, context-256 architecture
  • existing 2,000-step training protocol and stock WWPGD event projection

Notebook outputs

The notebook runs the repository's existing paired runner, verifies the frozen configuration before launch, and plots all curves against token-budget-equivalent training epoch (tokens_seen / train_split_tokens). It covers:

  • train, validation, and test cross-entropy
  • train, validation, and test perplexity
  • train/validation token accuracy
  • post-hoc test top-1 and top-5 token accuracy
  • test bits per token
  • validation/test loss and accuracy gaps
  • layerwise WeightWatcher alpha
  • alpha-fit KS D, sigma, fitted-tail support, detX retained count, and ERG gap
  • requested, applied, maximum, and cumulative WWPGD relative Frobenius correction
  • paired final-checkpoint summaries by seed

Test curves are evaluated after training from chronological saved checkpoints using a fixed per-seed test probe. They do not affect optimization or checkpoint selection. Analysis CSVs and PNGs are written under the selected paired result root.

Validation

  • notebook passes nbformat schema validation
  • every Python code cell passes ast.parse
  • branch differs from main by this notebook only

The six full training runs were not executed in this environment because the prepared FineWeb-Edu token data and target accelerator are not present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant