Use incremental binary checkpoint for tokenization resume#1633

Merged

finbarrtimbers merged 36 commits intomainfrom

finbarr/incremental-tokenization

Apr 23, 2026

Commits on Apr 18, 2026

Extract numpy SFT conversion helpers into open_instruct.numpy_dataset_conversion

finbarrtimbers
and
claude
committed
Add CHANGELOG entry for numpy dataset refactor PR #1622 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Add numpy_dataset_conversion tests and byte-for-byte Beaker verification harness Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Reduce verify script to 1 GPU to avoid Jupiter queue backlog Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Use uv run prefix for huggingface-cli and python in verify script Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Use python snapshot_download instead of missing huggingface-cli in verify script Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Pass tokenizer repo ID directly to avoid separate download step Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Add download_hf_repo.py helper and restore local tokenizer path in verify script Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Temporarily point 7b_think_sft_tokenization.sh to olmo-hybrid-fresh dir Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Point tokenization script to open-instruct-dev workspace Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Make tokenization script preemptible Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Use download_hf_repo.py instead of missing huggingface-cli in tokenization script Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed
Fix get_tokenizer_tulu_v2_2 for transformers v5 by using path substring instead of AutoConfig Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
finbarrtimbers
committed

Commits on Apr 20, 2026

Commits on Apr 21, 2026

Commits on Apr 23, 2026