exp87: train 1.5B on combined contacts-v1 + sequence-only (one epoch)#88
Open
timodonnell wants to merge 3 commits into
Open
exp87: train 1.5B on combined contacts-v1 + sequence-only (one epoch)#88timodonnell wants to merge 3 commits into
timodonnell wants to merge 3 commits into
Conversation
Fresh-init 1.5B Llama over a token-proportional mixture of the with-structure contacts-v1 corpus (~4.7B tok) and the ~7x larger sequence-only corpus (~32.65B tok), one epoch. Each batch mixes both kinds (~87% sequence-only); per-document- type eval losses are reported separately on W&B. - Forks exp85's train_common (carries the cache-reader workaround + PyPI marin-source-dist deps). - Reuses exp67's contacts-v1 train/val caches (dry-run confirms 663ba6 / val-92827b, both SUCCESS); tokenizes the sequence-only corpus fresh with the unified 2846-token tokenizer (timodonnell/contacts-v1-unified-tokenizer). - v5p-32, batch 512, LR 7.0e-4 (3.5e-4 x sqrt(4)), seq 8192, ~8,909 steps. - Sequence-only parquet staged to GCS us-east5 from exp64's local output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First launch OOM-killed the CPU-pool nodes (coordinator included) during the sequence-only train tokenize: the corpus is ~7x larger than contacts-v1 and length-banded (early shards hold the longest UniRef50 sequences), so several default 16g map-workers co-located on a node spiked past node RAM. Give the sequence-only TRAIN tokenize step cpu=4/ram=48g workers + coordinator so the scheduler packs fewer per node. Val step (small, length-balanced) succeeded on defaults and is left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First training launch reached step ~186 then a v5p-32 host died (JAX distributed coordination abort; iris fail=1/preempt=0, no executor auto-retry). marin imputes a deterministic run id from the output path, so relaunching this step resumes from the latest PERMANENT checkpoint in base_path — but only steps_per_export checkpoints land there (the rolling checkpoint goes to a ttl-temp path not read on cross-run resume). Drop steps_per_export 1000->250 so progress banks early and a supervised relaunch resumes with minimal loss. Not versioned: same run id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #87: train a 1.5B model for one epoch on the combined
contacts-v1(with-structure) +contacts-v1.sequence_onlycorpus.What
contacts-v1≈ 0.126 /sequence-only≈ 0.874, so each batch is ~87% sequence-only (as the issue expects). Train stream fully shuffled (Feistel).eval/contacts-v1-val/lossandeval/sequence-only-val/loss) via two distinct validation components.<contacts-v1.sequence_only>token (per the issue discussion); trains/exports with the unified 2846-token tokenizertimodonnell/contacts-v1-unified-tokenizer.Data plumbing
contacts-v1-663ba6/contacts-v1-val-92827b(bothSUCCESS), so no re-tokenization of the with-structure corpus.exp64_contacts_v1_sequence_only/documents/), staged there from exp64's local output as part of this work.contacts_v1_train_common.py, carrying its cache-reader workaround (ArrayExemplarTextLmDatasetFormat) and PyPI marin-source-dist dependency pins.Validation
--dry_runlists the 4 tokenize deps with the expected reuse/new split..venv/__pycache__gitignored; only experiment files + regeneratedindex.mdcommitted.Status / follow-ups
-<wandb-runid>suffix + final step intoexport_combined_contacts_v1.py, export to HF (tokenizer co-located), and run the exp82 contact-prediction harness vs exp: train a 1.5B model on contacts-v1 dataset #67/exp: continue the contacts-v1 1.5B run for another epoch (LR re-heat / warm restart) #85.Does not close the issue.
🤖 Generated with Claude Code