Skip to content

Releases: allenai/OLMo-core

Release list

v2.5.0

v2.5.0 Latest
Latest

Choose a tag to compare

View all tags
@tyler-romero tyler-romero released this 03 Apr 17:32
v2.5.0
002e0d7

What's Changed

Read more

Contributors

dirkgr, finbarrtimbers, and 10 other contributors
Assets 2
Loading

v2.4.0

v2.4.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Nov 19:15
v2.4.0
1ed8900

What's new

Added 🎉

  • Added option to skip ranges of steps in the trainer.
  • Send a Slack notification when a Beaker job appears to be stuck.
  • Added ignore_fingerprint_mismatch parameter to NumpyDataLoaderConfig to allow resuming training from a checkpoint with a different dataset mix.
  • Added helpful error messages when OLMo-mix-0625 files are not found, directing users to use OLMo-mix-0925 and the fingerprint override flag.
  • Added olmo_core.generate.chat module to allow interacting with OlmoCore models without conversion to other formats.
  • Added GAPMonitorCallback for monitoring gradients, activations, and parameters (GAP).
  • Added official Olmo 3 7B and 32B pretraining scripts and data mix.
  • Added official Olmo 3 7B and 32B midtraining scripts and data mix.
  • Added official Olmo 3 7B and 32B long-context scripts and data mix.
  • Added a NoOpOptimizer that does nothing, uses no memory, and can be used for debugging.
  • Added official config for Olmo 3 32B.
  • Olmo 3 model card and checkpoint manifests.

Fixed ✅

  • Set missing NCCL_NVLSTREE_MAX_CHUNKSIZE env var that is now needed for running jobs on Augusta cluster.
  • Fixed bug with RemoteFileSystemReader that caused excess memory usage.
  • No longer overrides random's RNG seed when building SourceMixtureDatasetConfig.
  • Fix handling URLs in olmo_core.nn.hf.checkpoint.save_hf_model and in examples/huggingface.
  • Fix potential NaN loss that can occur when using instance masking.
  • Stability improvements developed while training Olmo3 32B.

Changed ⚠️

  • Removed unused field in YaRNRoPEScalingConfig.

Commits

1ed8900 (chore) prepare for release v2.4.0
2c179c2 (chore) prepare for release v2.4.0 (#467)
7e0431f Fix link to 7B midtrain script (#469)
843fe3d Olmo3 model cards, checkpoint manifest, and readme (#468)
cbdc2f1 Olmo3 32B cleanup and checkin (#460)
20548a0 Official Olmo3 32B long-context script (#465)
14b15cc Official Olmo3 32B midtrain script(s) and mix(es) (#466)
a25a514 Official Olmo3 32B pretrain config and data mix (#464)
6b73ba0 Official Olmo3-7B long context script (#458)
bdc61e4 Official Olmo3-7B midtraining scripts (#445)
55804bf 32B official config (#454)
a86131d Slight refactor of Yarn Scaling Config (#456)
68c7409 Handle target URLs properly in HF conversion (#453)
2504cc2 Instance mask correction to avoid nan loss (#452)
137274e Add callback to monitor grads, activations, params (#446)
0959a54 Improve mem usage of RemoteFileSystemReader (#451)
600d2fe Official Olmo3-7B pretraining scripts (#443)
accc310 make launch timeout configurable from CLI
aa0e629 Avoid overriding RNG seed when building SourceMixtureDatasetConfig (#449)
98ba2e4 NoOp optimizer (#444)
03e6836 OlmoCore native chat interface (#439)
7a0bbd7 unset 2 NCCL env vars per Google's recommendation
aacb6eb only send local Slack notifications when callback is enabled (#441)
bfc8d7a Min python version to 3.10 (#442)
96d43d4 Set missing NCCL_NVLSTREE_MAX_CHUNKSIZE env var (#440)
5ad6db5 hot fix for listing gcs dirs
2186957 Allow manual bypass of fingerprint mismatch when switching datasets (#435)
043505d hot fix to step regex
dd7e747 Send a Slack notification when a Beaker job appears to be stuck (#431)
e27a9b4 Add WSDS (Warmup-Stable-Decay-Simplified) Scheduler (#419)
c92320f Use a dataclass for 'Trainer.steps_to_skip' (#430)
9669268 clean up checkpointing code to minimize distributed communication (#428)
87d64b9 fix changelog
269bf02 Add option to skip ranges of steps in the trainer (#425)

Loading

v2.3.0

v2.3.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 17 Oct 16:22
v2.3.0
5b32459

What's new

Fixed ✅

  • Fixed parsing username+password git remote URLs in launch.beaker module.
  • Fixed bug with default setup steps in launch.beaker.BeakerLaunchConfig when a branch can't be resolved.
  • Cluster names in Beaker have changed.
  • Fixed mixture rounding error with SourceMixtureDataset, which was previously causing samples to be repeated at the end of training.
  • Don't DDOS Beaker from big jobs.
  • A configuration error is now raised if you pass in a URL for the trainer or dataset's working directory.
    Previously the URL would just get mangled into a local path, leading to unexpected behavior.
  • Fixed an issue where the ConsoleLoggerCallback would attempt to log before the first step.
  • Only call teardown_distributed_environment() when training ends cleanly to avoid a hang for the duration of the distributed backend's timeout when there's an error from one rank.
  • Fixed tensor parallelism issue with torch 2.8.
  • More fixes for Beaker cluster names.
  • Callback.post_train() will still be called even if the run is canceled before the dry-run batch.
  • GarbageCollectorCallback will restore gc settings even when Trainer.fit() exits on an error.
  • Make move_to_device blocking for MPS device to fix possible incorrect transfer of data from CPU to MPS.
  • Fixed bug where glob_directory() would fail to match certain glob patterns.
  • Added one more type of error to retry on when the Google Storage API throws it.
  • Perform a garbage collection after checkpointing to avoid running out of CPU memory.
  • Avoidable overflow error when using NumpyPackedFSLDataset.
  • Fixed issue with NumpyFSLDatasetMixture + SourceMixtureDataset where not all instances would have the same sequence length.
  • Attention backend will no longer default to flash in non-CUDA environments.

Changed ⚠️

  • The dir option to Trainer.maybe_load_checkpoint() is now optional and defaults to the save_folder.
  • Set fused_linear_cross_entropy_loss accum_dtype to fp32 in LMHead.
  • Increased NCCL_FASTRAK_PLUGIN_ACCEPT_TIMEOUT_MS from 10 minutes to 30 minutes.
  • SlackNotifierCallback will now notify on checkpoint saved and post epoch events.
  • BeakerLaunchConfig.launch() will now send Slack notifications by default when follow=True if the env var SLACK_WEBHOOK_URL is set.
  • src/examples/llama/ has been renamed to src/examples/llm/.
  • Refactored eval task groups into task_groups.py
  • The use_flash argument to the Attention classes is deprecated. Use backend="flash_2" instead.
  • Refactored NumpyDatasetConfig by splitting it into a separate config per underlying dataset class.
  • Refactored internal/experiment module to facilitate modifying datasets or supplying a fully custom ExperimentConfig.
  • Simplified SourceMixtureDatasetConfig by removing redundant sequence_length and dtype fields.
  • The model_id argument to convert_state_from_hf is deprecated. Conversion information is deduced from the model type.
  • Refactored the example conversion scripts to/from HF, including decreasing false failures in validation.
  • Small refactor to source_mixture.py to make it easier to define data mixes in yaml.
  • Reorganized/cleaned up internal training scripts.

Added 🎉

  • Added CLI script src/scripts/unshard.py for converting distributed checkpoints to regular PyTorch or safetensors format.
  • Added a custom block that does LayerNorm scaling.
  • Added OLMo-mix-0625-150Bsample data mix.
  • Added alias support to DataMix enum.
  • Added the HalfCos learning rate scheduler.
  • Added CONTRIBUTING.md guidelines.
  • Added a lightweight, gantry-like Beaker launch CLI: python -m olmo_core.launch.beaker.
  • Added Beaker images with torch 2.8. There is olmo-core-tch280cu128-2025-09-18 and olmo-core-tch280cu129-2025-09-18 for CUDA 12.8 and 12.9, respectively.
  • Added TransformerEngine to Docker images and a TransformerEngine attention backend.
  • Added Callback.close() method, which is always called when exiting Trainer.fit().
  • Added flash-attention 3 to Docker images, added flash_3 attention backend.
  • Added support for sliding window attention to the Torch attention backend. Performance is not optimized, so other backends should be preferred.
  • Added RoPEScalingConfig.to_hf_config() for each RoPE scaling method to support automatic conversion to HuggingFace format.
  • Guide to dataset mixing in docs/source/guides/data_mixing.rst.
  • Added support for converting FlexOlmo models (with both dropless and default MoEs) between OLMo Core and HF formats.
  • Added olmo3_7B model config.
  • Added additional internal configuration tools.
  • Added a new named data mix that we used for the 32B run
  • Added internal OLMo3 7B midtraining and long-context configs.
  • Added ability to convert OLMo3 models to/from HF format with support for rope scaling configs.
  • Added a script that can pull out a single training batch from a training job

Commits

5b32459 (chore) prepare for release v2.3.0
3f21b77 Reorganize internal scripts for Olmo3 (#423)
67dd1d6 Use exec to start torchrun (#422)
206d25e Cookbook migration part 2 - long context config (#404)
eabb869 Raise timeout error if training job doesn't start in 5 mins (#421)
cc69286 Script to dump training tokens (#418)
b5ba7be Script for unsharding (#420)
464d01e olmo3 conversion w/ support for rope scaling (#415)
3afa7dd Add Rope scaling configs to rope module's exports (#414)
3ef0c05 Cookbook migration part 1 - midtraining config (#403)
cdb7922 Add Dolma 3 sample, and a way to alias data mixes (#412)
77adc0b pull Slack webhook URL from secret if available (#413)
c8e32ab NumpyFSLDatasetMixture + SourceMixtureDataset fix (#411)
6ce62cc Fix OverflowError in pack_documents (#409)
e229c17 Cookbook migration part 0 - more setup (#397)
8eef6f3 Data mix for the 32B (#405)
9ba6154 Retry more on GS failures (#406)
73d11f8 Do GC after checkpointing (#407)
cdfd201 Typo
f477a8d Fix glob_directory bug (#402)
bc2a3f1 Support conversion of dropless MoE to FlexOlmo (#401)
7fc49de Support rope scaling configs for hf conversion (#394)
0f38dc6 HF Conversion Refactor (#390)
a53f825 source mixture dataset simplification and documentation (#399)
69bd9d2 MPS bug fixes (#395)
601d336 refactor internal experiment configuration assembler (#386)
874be5d Add flash_3 attention backend (#377)
3aac611 Remove old beaker refs (#393)
1aeb369 Add sliding window support for torch attention backend (#388)
7bebea3 Fully migrate to new cluster names, fix internal experiment launching on Augusta (#391)
1d4b67f Add flash-attn-3 install to dockerfile (#392)
04d5d5d Add Callback.close() method, other minor callback improvements (#389)
dcf3a0e Add attention backend abstraction and integrate transformer engine's attention (#384)
0356c1b Np Dataset Config Refactor (#381)
afb5827 Improve distributed error handling (#380)
45d2031 Consolidate CLI code in public scripts (#379)
cbcc735 Port task groups from cookbook to OlmoCore (#378)
71f0023 bump torch and other dependencies in our Docker build (#360)
b3c3b4e Add an all-in-one guide for researchers (#374)
65d411b CONTRIBUTING.md (#376)
e469561 Treat cordoned beaker hosts as occupied (#375)
75d4d3a Avoid DDOS-ing Beaker for real (#372)
258a7a8 fix
b068044 Set accum_grad to fp32 for fused_linear_cross_entropy_loss, bump liger-kernel version (#370)
b550f9b Add the ability to send Slack notifications from launch.beaker (#371)
801de4b Makes it possible to override the common config builder (#369)
59a2b2f Adds a new, highly specific LR scheduler (#368)
d4ad23f fix rounding error with mixing datasets (#316)
46f7211 [Feat] Add LNS training example (#320)
3d9e9cd redo base dir calculation for new cluster names (#364)
b852ec0 Up NCCL_FASTRAK_PLUGIN_ACCEPT_TIMEOUT_MS to 30 minutes (#365)
72b7c1b Moved google-cloud-compute dependency from dev to beaker group. (#363)
c00d715 catch issues with dolma metadata files earlier (#361)
fa5a5dc Refine hostname constraints for beaker experiments on Google clusters (#355)
386b0a8 Fix parsing username+password git remote URLs (#356)
6726ffc make release process more robust

Loading

v2.2.0

v2.2.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 26 Aug 16:44
v2.2.0
de89fbe

What's new

Added 🎉

  • Added option to set LR scheduler based on tokens instead of steps (e.g. --train_module.scheduler.units=tokens).
  • Added a "packed" numpy FSL variant that packs documents into sequences using the best-fit-decreasing bin packing algorithm following the work from Fewer Truncates Improve Language Modeling.
  • Added module olmo_core.testing.
  • Added a "interleaved" numpy FSL variant that interleaves several documents into sequences following the work from LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models.
  • Added sliding window attention as a feature
  • Added BatchSizeSchedulerCallback for setting a batch size schedule over the course of a training run.
  • Added optional TrainModule method, .pre_train(), which runs right after Callback.pre_train().
  • The BeakerCallback will save the config and Python requirements to the results dataset.
  • Added from_file method to Config class.
  • Added in-loop evals for OLMES basic skills eval
  • Added in-loop fast MCQA for in-loop evals and translated MBPP tasks
  • Added in-loop few-shot HumanEval BPB
  • Added fast and full in-loop recommendations, where fast is a roughly 2-3x faster subset of full
  • Added support for converting to HF models in lower precisions.
  • Added support for headwise QK norm.
  • Add BOS token in in-loop evals, when specified by the tokenizer (ai2-olmo-eval==0.8.4)
  • Add support for BOS token matching EOS token for intra-document masking in FSL numpy datasets.
  • Added option to allow profiler to record on multiple ranks.
  • Added support for accessing Google on non-Google clusters via auth with service account keys.
  • Added support for revisions in convert_checkpoint_from_hf.py and the load_hf_model method of olmo_core.nn.hf.checkpoint.
  • foreach support in SkipStepAdamW.
  • Added budget mode for activation checkpointing configuration.
  • Added io.remove_file() and io.glob_directory functions.
  • Added ABF, PI, and YaRN rope scaling strategies.
  • Added a script to compare two WandB runs
  • Added namespace option to nn.buffer_cache.BufferCache.
  • Added the option to configure head_stride for context parallelism with ring-flash-attn.
  • Added the option to group multiple npy source files together for packing with the packed FSL dataset by setting source_group_size to an integer greater than 1.
  • Added load_optim_state: Optional[bool] option to Trainer.load_checkpoint().
  • Added GenerationModule for OLMo-core native autoregressive generation with support for kv caching.

Changed ⚠️

  • Output of LMHead when labels is passed as input is now a 4-tuple instead of a 3-tuple, with (logits, loss, ce_loss, z_loss), where loss is the combined loss (ce_loss + z_loss).
  • The ConfigSaver callback will automatically set the config to save for other callbacks (WandBCallback, CometCallback, and BeakerCallback as of now).
  • Fixed bug causing slow evals in BPB/RC in-loop evals due to fast MC
  • Changed default precision of converted HF models in src/examples/huggingface/convert_checkpoint_to_hf.py to bfloat16.
  • Changed default cluster to saturn in src/examples/llama/train_launch.py.
  • Made some beaker secrets optional for internal experiments.
  • Changed SlidingWindowAttentionConfig to improve clarity.
  • Changed the default Beaker budget

Fixed ✅

  • Modify TokenizerConfig.from_hf() to fallback to tokenizer_config.json if config.json is not found.
  • Fixed loading checkpoints with missing keys from transformer train modules using torch 2.7.
  • Made MoE load balancing loss more robust.
  • Fixed a bug with ReorderedNormTransformerBlock when using fine-grained FSDP wrapping and activation checkpointing together.
  • Fixed an issue preventing tensor parallelism from working with LMHead when using the "fused_linear" loss implementation.
  • Fixed a bug with LMHead when using "fused_linear" loss implementation where the ce_loss output included the z_loss added to it.
  • Fixed training on single GPU when using a SkipStepOptimizer.
  • Fixed the initialization of the CosWithWarmupAndLinearDecay learning rate scheduler
  • Ensured eval tasks are sorted to maintain the same order across ranks (the cookbook was configuring these in an unsorted way).
  • W&B callback uses working directory instead of save folder for local cache.
  • Reset speed monitor callback after changing batch size.
  • Fixed parallelism compatiblity between cp + tp and cp + pp and added test to catch regressions.
  • Ensure sharded parameters are initialized differently on separate ranks.
  • Fixed fingerprinting for FSL datasets
  • Fixed bug where step state in SkipStepAdamW was not incremented, biasing the optimizer steps. Added option to restore the bug for backwards compatibility.
  • Removed sklearn from upstream dependency ai2-olmo-eval.
  • Made removing ephemeral checkpoints more robust.
  • Made running bookkeeping operations more robust.
  • Ensure RoPE modules with different settings use a unique sub-cache for their buffers.
  • Fixed bug with context parallelism where every transformer block would use the same RoPE buffers even if their RoPE was configured differently.
  • Fixed MFU computation to work with FSDP, corrected some device specs.
  • Optimization: avoid redundant calls to model.train() in TransformerTrainModule.
  • NumpyDatasetConfig.expand_glob now works with remote directories.
  • Fixed Attention block sharding when TP and head-wise QK norm are both applied.

Commits

de89fbe (chore) prepare for release v2.2.0
54d3af0 run gpu tests with gantry (#357)
c82d13c GenerationModule with support for KV Caching (#324)
effdef3 Add option to Trainer.load_checkpoint() to ignore optim state (#351)
2cd5b82 Fix TP when headwise QK norm is applied (#353)
429054a Fix empty config.json output in convert_checkpoint_from_hf (#354)
e97f58d Fix skip step optimizer with TP (#352)
7f4b45f Option to group npy sources together for packing (#349)
9fb6366 Add io.glob_directory function (#348)
67854f9 Avoid redundant calls to model.train() (#345)
7e14a68 Fix CP bug with RoPE buffers (#341)
699971d Support configuring head_stride for ring-flash-attn (#344)
b0019cc Pull updates from olmo3 branches (#334)
026b882 Fix MFU Calculation (#343)
4162151 Add 'namespaces' to BufferCache to avoid collisions (#340)
085755c Port the WandB comparison tool from the old trainer (#338)
9e42a20 The Beaker default budget has changed (#337)
e395b82 Ensure LR scheduler's unit are tokens when using BZ scheduler (#335)
08340f9 fix off-by-one issue with SWA
abc12e5 Make async bookkeeping more robust (#333)
6d2f334 Add TrainModule.pre_train(), fix pain point with BZ scheduler (#332)
8fb85bc make Scheduler a subclass of Config (#331)
e1bac95 More Rope Scaling Implementations (PI, Yarn) (#330)
97a6f4e Make removing ephemeral checkpoints more robust (#329)
ebe0b19 loosen numpy requirement (#328)
5b924a6 Add missing return in init (#327)
34beda5 Bump ai2-olmo-eval==0.8.5 (#326)
f107e9c fix typo in release process
bddf65a fix initialization (again) (#319)
992a79e Memory budget strategy for activation checkpointing (#297)
0dda3ec DDP parameter dtype casting for 16-bit precision and flash attention support (#314)
26998de Improve clarity of SWA config (#301)
91630ea Add config option for to enable/disable the step-increment bugfix. (#317)
51e2049 "Better" sorting of Augusta ranks at runtime (#313)
fe50d9e Only check for beaker secrets in non-distributed settings (#302)
c41962a SkipStepAdamW foreach implementation; bug fix for step state in SkipStepAdamW (#309)
a04e93f ensure sharded parameters initialized different on different ranks (#307)
f7c394d Fix fingerprints for various FSLDatasets (#303)
5f93eed hot fix
20d833b Add support for revisions in conversion from HF (#304)
b0f1d9e use qualname instead of name
b744d9f Make async bookkeeping more robust (#305)
796c2dc Add support for accessing Google cloud on non-Google clusters (#299)
2bdb93c Allow profiler to record on multiple ranks (#298)
00f9b00 Fix for parallelism compatibilities in build_world_mesh (#293)
ef2845f Bump pytorch, ring-flash-attn, and liger-kernel versions (#295)
4cffd08 Add support for BOS token matching EOS token in documents (#291)
c5777a3 Make ruff line-length match black line-length (#290)
f2cc497 Reset speed monitor after changing batch size (#289)
9816e44 ai2-olmo-eval==0.8.4 (#288)
a42ce36 Add head QK norm support for Attention (#287)
e7d01b1 use work_dir instead of save_folder for W&B cache
2cd5a31 Update default cluster for src/examples/llama/train_launch.py (#286)
2b73817 Support lower precisions for conversion to HF (#277)
aa96c3a Bump ai2-olmo-eval==0.8.3 (RC/BPB speed fix) (#285)
9fac3c5 HF conversion improvements (#284)
db2b8a4 catch other types of error when importing liger-kernel (#283)
fbeaa97 Add "fast" in-loop task set (#282)
776778e Fast in-loop MCQA (#281)
c779ca5 Bump ai2-olmo-eval==0.7.2 (in-loop Basic Skills) (#279)
fd44f03 Update images to stable 2.7.0, use CUDA 12.8 by default (#280)
1acde9d Fix Numpy data loader indices (again) (#276)
b18e58a Ensure eval tasks are sorted for consistent order (#275)
e185944 Save metadata to Beaker results dir (#274)
490f03a make indices filename robust to change in batch size (#273)
80cf40e Add BatchSizeSchedulerCallback (#272)
cd3d995 Sliding Window Attention (#271)
53c28a4 use REST API to follow jobs again
ef28f2a update pins
1662d0d don't auto upgrade beaker-py for now
5cc16a2 Add Interleaved Numpy Dataset (#263)
1cb5add Scheduler init (#267)
2caadea Move test utilities to new submodule olmo_core.testing (#266)
5d4c7ec Fix singe-GPU training with SkipStepOptimizer (#265)
9a19a71 More fixes for LMHead with TP (#264)
5bd9006 minor improvements to log streaming
1d4bd1d Add a numpy FSL dataset variant ...

Read more
Loading

v2.1.0

v2.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 14 Apr 18:08
v2.1.0
b8070fb

What's new

Added 🎉

  • Added 50B Dolmino 11/24 mix.
  • Added support for auxiliary-loss-free MoE load-balancing, similar to DeepSeek-v3. You can activate this by setting bias_gamma to a non-zero float in your MoERouter config.
  • Added support for sequence-level MoE load balancing loss.
  • Compatibility with B200s.
  • Added support for warmup_fraction as an alternative to warmup_steps in all schedulers, allowing warmup to be specified as a fraction of total training steps.
  • A better config for the 1B model, ported from the old OLMo trainer.
  • Added auto_resume option to CometCallback for resume an existing run.
  • (BETA) Added methods load_hf_model and save_hf_model for saving supported OLMo Core models to HF transformers format.
    Also added lower-level methods for converting state between the formats.
  • Added the ability to run the evaluator callback on .pre_train() by setting eval_on_startup=True, and to cancel the run after the first time evals run by setting cancel_after_first_eval=True.
  • Added support for label mask files with numpy FSL datasets.
  • Added a git configuration to BeakerLaunchConfig.

Changed ⚠️

  • TransformerTrainModuleConfig can now be used to build a TransformerPipelineTrainModule by adding a pp_config spec. This makes the TransformerPipelineTrainModuleConfig redundant, but it will be kept around for backwards compatibility until the next major release.
  • Several state dict methods in TrainModule now take an optim option, which can disable the use of optimizer state.
  • Updated Float8Config for latest version of torchao.
  • Undo a fix applied to olmo_core.data.numpy_dataset.NumpyFSLDatasetMixture that was generating a mismatch between the shape of instances in the dataset and the shape of instances in the data loader.
  • Made the 1B and 7B scripts more similar to each other.
  • Changed underlying logic and top-level arguments of convert_checkpoint_from_hf.py and convert_checkpoint_to_hf.py.
  • Beaker experiments launched with the BeakerLaunchConfig will now log with ANSI colors enabled.

Fixed ✅

  • Fixed calculation of total steps based on epochs at the end of a training job.
  • Fixed a bug where the trainer might try to save a duplicate final checkpoint if the run that already completed was restarted.
  • When submitting a Beaker job from a branch that's tracking a GitHub fork, OLMo-core now instructs Beaker to pull from the fork instead of from the main repo.
  • Made Beaker image resolution more robust.
  • Having t_max overrides in the default model configs is confusing and error prone, so we removed them.
  • Beaker launcher will only clone a single branch at runtime when possible, which can be much faster.

Commits

b8070fb (chore) prepare for release v2.1.0
7bc8aa2 remove erroneous license in test file
db91b7f Add a git config to BeakerLaunchConfig (#251)
36b791a [HF Converter] Expect model and optim state in model_and_optim subdirectory (#253)
1f2f6f9 Log with ANSI colors in Beaker (#252)
d0ab790 No more t_max (#247)
5653c92 rename * (unscaled) metrics to * unscaled
60a19c3 clone single branch when possible (#250)
e9a34e8 More MoE updates (#246)
c149b73 Update images for torch 2.7.0 (#249)
6d2bb0a Added 50B Dolmino-1124 mix (#248)
53e67ce Add option to cancel run after first evals (#244)
b493d50 fix in-loop normalization with v2 (#243)
a07ef78 Add a self-contained template train script (#242)
746408e Port the 1B from old OLMo (#234)
4ec0866 Add support for label masks with numpy datasets (#241)
ecb14e0 only resume if name matches (#240)
fc84edc Add option to auto resume Comet experiments (#239)
f5d85a9 OLMo Core to HF conversion refactor (#226)
23c6cb1 clean up logging output from source mixture tests
d502b7e Mapping new ladder to old ladder (#146)
a135883 fix calculation of max steps based on epoch at the end (#236)
2f66fd9 Added warmup_fraction to all schedulers (#235)
be06aa0 B200 compatibility (#232)
0973d4d make beaker image resolution more robust (#233)
78be552 Pick the correct remote (#230)
590138d Temp disables custom read_chunk_from_array in SourceMixture (#231)
082e0b1 Fix bug when restarting a completed run (#229)
6c626f2 Update float8 API for latest torchao (#228)
8919dff Some MoE changes/additions to support auxiliary-loss-free load-balancing (#227)
26e9476 Allow train modules to not load/save optimizer state (#225)
8c20a64 run cuda gc at the end of training
a907892 Merge transformer train module configs (#224)
b47e01c Added 32B stage2 checkpoints .csv (#220)

Loading

v2.0.1

v2.0.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Mar 20:56
v2.0.1
27b1ae8

What's new

Added 🎉

  • Added information about the official 32B training run.
  • Added automatic support for LL128 when running on Augusta.

Fixed ✅

  • The official config for the 32B had unrealistic batch size settings.
  • Ignore group_overrides for frozen parameters instead of throwing an error.

Removed 👋

Commits

27b1ae8 (chore) prepare for release v2.0.1
79ebc7f Add hybrid MoE transformer architecture (#223)
bce2b5b authenticate with Docker Hub to avoid rate limits
b1e0bbd Remove fused CE loss, reorganize MoE kernels/ops (#221)
56e06ee Ignore group_overrides for frozen params (#219)
9d80e8d Update logo for README header. (#218)
974e555 fix some typos, consistent naming
45fe007 Updated documentation (#217)
51aedcf More working config (#216)
47b2ad5 add release PR comments back in

Loading

v2.0.0

v2.0.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 13 Mar 01:42
v2.0.0
dfa8f2b

What's new

This major release introduces a few breaking changes. We've provided more information here: OLMo-core v2 design and upgrade guide.

Added 🎉

  • Added TrainModule abstraction with TransformerTrainModule implementation, which encapsulates both a model and optimizer.
  • Added namespace argument to Trainer.record_metric().
  • Added support for context parallelism.
  • Added support for expert parallelism with MoE models.
  • Added in-loop evals for Minerva, GSM, HumanEval, MBPP (ai2-olmo-eval==0.7.0)
  • Added CosWithWarmupAndLinearDecay learning rate scheduler
  • Added WSD learning rate scheduler

Changed ⚠️

  • The Trainer now takes a TrainModule instead of a model and optimizer, and several configuration options have been moved to TransformerTrainModule, including rank_microbatch_size, fused_loss, compile_loss, z_loss_multiplier, and autocast_precision.
  • Several TransformerModelConfig options have been to TransformerTrainModule / TransformerTrainModuleConfig, including dp_config, tp_config, float8_config, and compile.

Removed 👋

  • Removed the following callbacks: MoEHandlerCallback, SchedulerCallback, MatrixNormalizerCallback, GradClipperCallback, and Float8HandlerCallback.
    The functionality from all of those callbacks has been moved to the TransformerTrainModule class.
  • Removed the callback methods .pre_eval_batch() and .post_eval_batch().

Fixed ✅

  • Fixed the model ladder code when training on mps or cpu device

Commits

dfa8f2b (chore) prepare for release v2.0.0
95fb084 add work-around for pytorch/ao#1871 (#205)
3ce0c58 32B Documentation (#210)
41f8ddc Add a public "official" version of our 32B train script (#214)
7e58d12 Update data paths in example to public URLs (#213)
4327bb9 upload data to r2 and updated their paths (#208)
0e6ea23 Assorted improvements (#207)
9ceb1e4 Add CUDA 12.6 images (#209)
eda3afb guard against wrapping MoE modules for AC (#206)
6e5b16f Bump ai2-olmo-eval==0.7.0 (in-loop Minerva, GSM, HumanEval, MBPP) (#204)
eccdc00 Make it easier for external users to run train scripts (#203)
da33f5b fix entrypoint steps
947a293 clean up changelog
725adf3 V2 (#202)

Loading

v1.9.0

v1.9.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 10 Mar 20:41
v1.9.0
41a7dbd

What's new

Fixed ✅

  • Ensure certain optimizer param group fields are not overridden by the values in a checkpoint.

Added 🎉

  • Added instance_filter_config field to NumpyDatasetConfig.
  • Added conversion script for OLMo 2 checkpoints to Huggingface format.
  • Added BeakerCallback.
  • Added logging for in-loop eval throughput

Fixed ✅

  • Ensure certain optimizer param group fields are not overridden by the values in a checkpoint.
  • Fixed issue where non-zero ranks would report partially-reduced values for training metrics.

Commits

41a7dbd (chore) prepare for release v1.9.0
d7301e6 32B scripts (#201)
d55562c Log in-loop eval throughput (#200)
260dafd Add support for BF16 optim state in SkipStepAdamW (#148)
e522437 fix inferring sequence length
0bef5aa allow dynamic batch sizes (#170)
fa11a40 Port over instance filtering from old codebase (#157)
8ef038a update formatting of bucket distribution
c9ca78a Add a BeakerCallback (#177)
e1cd8f6 use effective sequence length
32cb0fa Conversion script for OLMo 2 models trained with OLMo core to HuggingFace (#158)
feb57eb all-reduce train metrics (#166)
2b43d59 reset initial LR to configured value after loading (#163)
2902a9c Improve Config.from_dict (#156)
b4cee6d ignore class name field when config from dict
c1d1a53 update DTensor imports to use public module (#153)
4594231 activate virtual env before running script

Loading

v1.8.0

v1.8.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 30 Jan 01:12
v1.8.0
7899e7c

What's new

Added 🎉

  • Added support for tensor parallelism. See the TransformerConfig class for usage.
  • Added more downstream tasks from the model ladder.
  • Added io.copy_dir() function.
  • Added new LR schedulers: LinearWithWarmup, InvSqrtWithWarmup, ConstantWithWarmup, SequentialScheduler.
  • Added option to pre-download checkpoint files from remote storage before trying to load a checkpoint.
  • Added a callback for sending Slack notifications.
  • Makes the MPS device work on Apple Silicon
  • Added SkipStepAdamW optimizer.
  • The trainer can load model-only checkpoints now.
  • Added the option to throttle checkpoint uploads to one rank from each node at a time.
  • Added support for logging rich Table objects as text in source mixture datasets.
  • Added unshard_strategy parameter to unshard_checkpoint() function in olmo_core.distributed.checkpoint.
  • Added function load_keys() to olmo_core.distributed.checkpoint.

Changed ⚠️

  • Changed storage of shared shard state in sharded checkpoints from smallest shard to lowest rank (normally 0).
  • Changed how the trainer handles loading a checkpoint when load_path is provided. Now load_path is only used if no checkpoint is found in the save_folder.

Fixed ✅

  • Added missing weights_only=False argument to fix loading train checkpoints with newer versions of PyTorch.
  • Fixed bug where GCS upload does not retry on transient failures.
  • Fixed bug where source mixture datasets were truncating source files instead of randomly sampling.
  • Fixed bug in source mixture datsets where sampling from small npy files raised an mmap exception due to 0 instances in the sampled index.

Commits

7899e7c (chore) prepare for release v1.8.0
907b9c5 Send Slack notification on releases (#151)
1ef7851 fix get_mock_batch() when training on MPS again
29a468d Fix mixture dataset class (#147)
98ccb67 remove ganymede cluster
205fe90 remove deleted cluster
7ec9114 always make mock batch on CPU
7122b1d save max steps to trainer state (#143)
9a78829 Log elapsed time per eval (#149)
075a36a Make training on the MPS device work (#131)
b4a195b Add more options to the unshard_checkpoint function to help scale (#145)
16885ab fix merge list with prefix
7b755c9 minor logging improvement
212108f Add option to throttle checkpoint uploads to one rank from each node at a time (#142)
7633461 pull fixes from 32B branch (#139)
48abe8c checkpoint hot fix (#140)
0c096e2 Handle model-only checkpoints with the trainer
9818232 move release scripts to subfolder (#137)
05ab673 update cluster list (#136)
7ccf726 add pr comments on release
0ff19d7 update citation
7519e0a Change the way load_path is handled (#132)
03a597a limit the number of exception lines posted to Slack
c634066 include link to Beaker job with Slack noties
3505660 Make context manager set original state correctly (#126)
9e0992b Add a callback for sending Slack notifications (#125)
6d60464 fix
ee27348 Sync eval changes in OLMo/ladder-1xC to here (#122)
0789479 Add option to pre-download checkpoint to load (#123)
1380f0e add copy_dir() io function
5cc704f Add learning rate schedulers (#119)
de5be27 don't check for beaker-py upgrades
b0103f0 Fix loading train state for newer versions of torch
5de774f updates
8474ee8 update docker image tags
d3f6f01 Update PyTorch and other deps in Docker images, change naming scheme of images (#120)
10c4978 Publish Docker images to GHCR (#118)
d6981b3 Add support for tensor parallelism and add OLMo2-26B model config / train script (#117)
aa4d188 Update table formatting

Loading

v1.7.0

v1.7.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Nov 22:31
v1.7.0
62d2c9e

What's new

Added 🎉

  • Added key_mapping argument to olmo_core.distributed.checkpoint.load_model_and_optim_state()
    for loading checkpoints with different key names.
  • Added load_key_mapping field to the trainer, same idea as the new key_mapping argument above.
  • Added an implementation of nGPT called NormalizedTransformer.
  • Added an example showing how to convert a HuggingFace Llama 3.2 checkpoint into the right format for OLMo-core.
  • Added an API for scaling RoPE embeddings.
  • Added a ModelLadder API.

Changed ⚠️

  • The w_out and norm top-level children of the Transformer model are now wrapped together in an lm_head module. Training scripts will have backwards compatibility with older checkpoints due to the load_key_mapping explained above.

Fixed ✅

  • (Optimization) Mark model input sizes as dynamic for torch.compile() to avoid recompile during evals or variable-sequence / batch size training. This doesn't seem to hurt throughput.
  • Made HTTPS and GCS IO functions more robust.
  • Fixed a bug where we were always getting dolma2 tokenized validation data when generating config with DataMix.v3_small_ppl_validation.

Commits

62d2c9e (chore) prepare for release v1.7.0
cb77039 mark model ladder as a beta feature
08c8073 Adapt conversion script to work with OLMo2 models (#116)
8e716b5 Add model ladder building blocks (#114)
1647f78 Add some more tests for nGPT (#113)
37e0e88 improve docs
d68d47a Make nn configs more flexible (#112)
0bcc840 RoPE scaling, document how to convert HuggingFace checkpoints (#111)
7655a3b Add template variable to ppl validation file manifest (#110)
ca44cf4 Implement nGPT (#108)
c47df7c make IO functions more robust (#109)
4f2c8ef Update README.md
57b38ad Mark model input as dynamically sized (#105)
776e235 remove duplicate script

Loading