v0.3.0

This is a big one, as I haven't released since March. Whoops! Also my last release as maintainer, as today is my last day at Ai2. I'll miss you all.

Added

• Add minimax-m3 to PRICE_PER_MILLION_TOKENS in open_instruct/judge_utils.py for cost tracking on the latest MiniMax flagship model.

Changed

• Add minimal support for DPO-training the Olmo Hybrid 7B (GDN linear attention) model with open_instruct/dpo.py (OLMo-core): bump OLMo-core to a rev with the olmo3_hybrid_7B preset and bidirectional HF weight conversion, bump flash-linear-attention to 0.5.0 and add tilelang (correct GDN gradients on Hopper), add selected_modules activation checkpointing (forwarding determinism_check="none" through OLMo-core's activation-checkpointing config so torch.compile and the opaque fla kernels coexist), extend ModelDims with GDN FLOPs/params accounting, and add scripts/train/olmo-hybrid/7b_instruct_dpo_sweep_olmo_core.sh (#1715).
• Record a _metrics_keepalive metric on every rank every GRPO+OLMo-core step to keep _metrics non-empty, preventing OLMo-core's empty-skip in _log_metrics from desyncing the bookkeeping process group and deadlocking gloo for 30 minutes at save-time flushes (#1708).
• Expand type-checking coverage by replacing # ty: ignore directives with typed casts and fixing related type issues (#1688).
• Add TV divergence rho filtering for GRPO (#1681).
• Export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_OPEN_INSTRUCT=0.0.0+debug in scripts/train/debug/grpo.sh and grpo_fast.sh (local Ray debug scripts that disable torch compile) so setuptools-scm can resolve the package version (#1696).
• Simplify GRPO clip fraction handling by returning the final policy loss and clip fraction directly from compute_grpo_loss (#1679).
• Bring grpo.py (OLMo-core GRPO) to feature parity with grpo_fast.py: add EvalCallback, setup_eval actor RPC, unconditional vLLM-sync callback, ConstantWithWarmup scheduler support, and StepTimingCallback end-to-end step timing (#1672).
• Remove references to deleted ppo_vllm_thread_ray_gtrl.py script: delete broken launch scripts (scripts/train/debug/ppo.sh, scripts/train/rlvr/tulu_rlvr.sh, scripts/train/tulu3/ppo_8b.sh) and add historical-reference notes to docs/tulu3.md and docs/archived_dev_scripts/olmoe_0125.sh pointing to the deletion commit. Also drop the dead update_command_args.py references: delete scripts/train/benchmark.sh and its section in docs/get_started/ai2_internal_setup.md, and update the README RLVR quickstart to launch grpo_fast.py via scripts/train/build_image_and_launch.sh.
• Bump vllm to >=0.19.1 (and refresh uv.lock, including compressed-tensors v0.14.0.1 → v0.15.0.1).
• Move maybe_evaluate from grpo_fast.py to grpo_utils.py and drop the duplicate PolicyTrainerRayProcess.calculate_token_counts method, routing both trainer paths through the shared grpo_utils.calculate_token_counts (#1669).
• Rename time/trainer_idle_waiting_for_inference to time/trainer_waiting_for_data and time/generation_idle_waiting_for_trainer to time/generation_waiting_for_trainer, and emit per-Group generation timing (time/group_generation_{mean,max,min} plus batch/per_group_generation_times histogram) so latency vs. throughput in the inference pipeline is legible from wandb (#1690).
• Add parameterized combine_dataset tests in open_instruct/test_utils.py against local jsonl fixtures (no network), covering varied fractional/sample-count weight combinations and split-count mismatch (would have caught the bug fixed in #1674). Extract the interleaved-list→dict parsing into a shared utils.parse_dataset_mixer_list helper (with its own parameterized unit tests) and tighten combine_dataset / get_datasets to accept dict-only dataset_mixer; the one external list-form caller (rejection_sampling/generation.py) now converts at the call site.
• Make mason.py --output_dir / --checkpoint_state_dir overrides idempotent via replace_or_append_flag, add open_instruct/grpo.py to OPEN_INSTRUCT_COMMANDS / OPEN_INSTRUCT_RESUMABLES, and wire OLMo-core checkpoint save/resume into grpo.py (CheckpointerCallback + DataPreparationActorCheckpointCallback + LoadStrategy.if_available) so resumable Beaker jobs actually resume (#1666).
• Make --budget optional in mason.py (falls back to the workspace's default budget) and drop the explicit --budget flag from launch scripts where it already matched the workspace default (#1673).
• Restore 🤡 to resample warnings and use self.training_step in DataPreparationActor.run (#1663).
• Add a unified use_rho_correction interface (clamp + mask, per-token or sequence-level) for the train/infer engine mismatch in GRPO loss; replaces truncated_importance_sampling_ratio_cap and the IcePop flags (#1650).
• Resample on filtered batches in DataPreparationActor instead of emitting empty CollatedBatchData, unifying the grpo.py and grpo_fast.py consumer paths and removing the now-dead empty-batch checks in grpo_fast.py (#1660).
• Update Beaker budget from ai2/oe-omai to ai2/oe-other across launch scripts and beaker configs.
• Update Beaker budget from ai2/oe-adapt to ai2/oe-omai across launch scripts and beaker configs to fix experiment launch failures from the retired budget (#1662).
• Log every filtered prompt in accumulate_inference_batches at INFO level with the zero/solved/nonzero breakdown, and add batch/filtered_prompts_pct to wandb so policy collapse / convergence is visible without spelunking debug logs (#1657).
• Aggregate prompt/response lengths across all DP ranks (deduplicating SP groups) when computing GRPO step token counts and utilization metrics, instead of using only rank 0 (#1659).
• Split accumulate_inference_batches into process_single_result and combine_processed_results for clarity (#1614).
• Match reference SFT run: olmo_core_finetune.py parity with pure olmo-core; default CP strategy switched to ulysses and ring-flash-attn dependency removed (#1620).
• Address review feedback on #1620: derive vocab size from the run's tokenizer (no longer hardcoded to dolma2), validate complete numpy artifacts before reusing the SFT cache, fold seed/max_seq_length into the cache directory, fix HF-vs-olmo-core checkpoint detection for relative local paths, and log which checkpoint format was detected (#1620).
• Stream SFT tokens/labels/boundaries directly to _*.partial.bin files and derive per-dataset stats at the end from disk, dropping the explicit _checkpoint.json file. --resume now works by truncating the partial files to a consistent sample boundary (#1631).
• Revert reapply of packaging fix from #1634 (#1637).
• Drop unused data_types import and inline batch["batch"].to(device) in GRPOTrainModule (#1635).
• Use incremental binary checkpoint for SFT tokenization resume, eliminating O(N²) re-serialization (#1633).
• Extract numpy SFT conversion helpers into open_instruct.numpy_dataset_conversion (#1622).
• Simplified model step tracking logic (#1616).
• Pass attention_mask=None in GRPO forward_for_logprobs calls — HF constructs the correct 3D intra-document mask from position_ids internally (#1617).
• Migrate GRPO trainer→vLLM weight sync to vLLM 0.16.0's native weight transfer API (NCCLWeightTransferEngine), replacing custom NCCL process-group and broadcast code (#1515).
• Extend pre-commit hook to also ban nonlocal keyword (#1613).
• Set checkpoint_state_freq default in data_loader.py, not mason.py (#1600).
• Inline data prep actor naming in StreamingDataLoader and GRPO, removing redundant helpers and parameter plumbing (#1326).
• Use local fixture for AceCode test instead of downloading from HuggingFace (#1593).
• Now, to disable max_grad_norm clipping, set None, not -1 (#1591).
• Inline GRPO utility functions and rename ExperimentConfig to GRPOExperimentConfig (#1578).
• Extract shared OLMo-core config classes and helpers into olmo_core_utils.py; refactor DPO to use shared configs (#1576).
• Decouple mix_data.py from finetune.py by replacing FlatArguments import with a lightweight MixDataArguments dataclass (#1573).
• Extracted shared find_free_port utility function (#1607).

Deprecated

...

v0.2.0

Notable changelog items:

1. Fixed a major but where ZeRO-2 was discarding gradients.
2. We now support training with a RLEnvironment abstraction, including code execution sandboxing.
3. The Olmo-core DPO is ready to go, with support for TP, packing, and torch.compile. This should be much faster than the previous HF implementation.
4. We have a model merging implementation that runs on Beaker (for internal Ai2 users).
5. Now, GRPO doesn't let the actors run excessively ahead of the learner during training if the actors are much faster than the learners.

Fixed

• Fix ZeRO-2 discarding gradients during manual gradient accumulation by using set_gradient_accumulation_boundary() (#1498).

Added

• Clean up OLMo 3.X tokenizer docs: clarify think SFT tokenization workaround, add dev/release tokenizer matrix, create allenai/olmo-3-tokenizer-instruct-release (#1487).
• Add Docker sandbox backend and GenericSandboxEnv environment for code execution during RL training. DockerBackend with command timeout, configurable memory limits, put_archive/get_archive file I/O, and remove=True auto-cleanup. GenericSandboxEnv provides execute_bash (stateful bash with env/cwd persistence) and str_replace_editor (view/create/str_replace/insert with correct line numbering). Configurable penalty, image, and memory via GenericSandboxEnvConfig. Includes 1-GPU debug script (#1490).
• TextRLEnvironment base class for text-based RL environments: model output is passed as a plain string instead of parsed tool calls, with response formatted using the parser's role_template. Includes WordleTextEnv example, role-aware format_tool_outputs in all parsers, shadow tool call dispatch in process_request, and 1-GPU debug script (#1489).
• Wire RL environments into vLLM generation loop and preprocessing: unified tool/env system with single TOOL_REGISTRY, pooled actors via shared EnvironmentPool Ray actor (async acquire/release, auto-sized to rollout concurrency), RolloutState tracks all per-rollout state, PassthroughVerifier + RewardAggregator for per-turn rewards (verifier score folded into last turn before aggregation), BaseEnvConfig in environments/base.py, --max_steps unified, --pool_size configurable, auto-discovery of tools from datasets, 1-GPU debug scripts for counter/guess_number envs (#1479).
• RL environment abstraction: RLEnvironment base class with Tool as a subclass, unifying tools and environments under a single step(EnvCall) -> StepResult interface. Removes Executable/EnvOutput/_execute/safe_execute indirection. Moves tools under open_instruct/environments/tools/. Includes example environments (CounterEnv, GuessNumberEnv) (#1478).
• Enable packing with torch.compile for DPO training, fix cu_seq_lens offset bug for padded chosen/rejected sequences, add tokens_per_second_per_gpu metric (#1466).
• Model merging scripts for Beaker: mergekit_merge.sh (mergekit) and direct_merge.sh (direct safetensors averaging for hybrid models) (#1459).
• Production DPO script for OLMo3-7B hybrid (#1449).
• Gradient accumulation/microbatching support for OLMo-core DPO training (#1447).
• Evolving rubrics support with RubricVerifier and utility functions for GRPO training (#1460).
• New perf metrics in PerfCallback: total_tokens, data_loading_seconds, data_loading_pct, wall_clock_per_step, step_overhead_pct (#1457).
• Warning when eval prompts are queuing up (new eval round starts before the previous one completes) (#1461).
• OLMo 3 tokenizer settings documentation covering chat template decisions for Instruct and Think models (#1455).
• torch.compile support for OLMo-core DPO training (#1445).
• Adds a GRPOTrainModule as part of the Olmo-core migration (#1412)
• FSDP shard_degree and num_replicas configuration for OLMo-core DPO training (#1446).
• Budget mode gradient checkpointing support for OLMo-core DPO training (#1444).
• PerfCallback for MFU metrics in OLMo-core DPO training (#1442).
• NVIDIA H200 GPU support in GPU_SPECS (#1441).
• Documentation and runtime warning for dataset_mixer_list format (float=proportion, int=count) (#1434).

Changed

• Bound async data preparation to stay within async_steps of training, preventing training data getting too far out of sync with trainer. (#1496).
• Refactor Legacy and DRTulu tool parsers to use OpenAI-format tool_definitions instead of Ray tool_actors. Removes import ray from parsers.py, fixes DRTulu parser which was broken after the pool refactor, and fixes --tool_parser_type typo in dr_tulu debug script (#1491).
• Replaces lambda collators with a "single_example_collator" (#1472).
• Clarified activation_memory_budget guidance in DPO utils with a practical default (0.5) and memory/speed tradeoff notes (#1460).
• Let TransformerTrainModule handle FSDP parallelism instead of manual application in DPO (#1458).
• Refactored DPOTrainModule to inherit from TransformerTrainModule (#1456)
• Increased vLLM health check timeout from 30s to 600s (10 minutes) (#1452).
• Updated vllm version to 0.14.1 (#1433).
• Changed default wandb x-axis from episode to training_step for grpo_fast (#1437).
• Made a bunch of changes to dpo.py so it matches dpo_tune_cache.py perfectly (#1451).

Fixed

• Updated previous fix of weight sync thread to only be active when inflight_updates=False, removing an issue with weight sync updates stalling (#1499).
• Fixed weight sync thread hang when inflight_updates=False: wait for all vLLM engine.update_weight RPCs to complete before unpausing actors, preventing health_check_fn from blocking indefinitely (#1480).
• Fixed nodes_needed calculation in grpo_fast kv_cache_max_concurrency warning using math.ceil() instead of floor division to avoid undercounting required inference nodes (#1474).
• Fixed eval_on_step_0 never triggering in grpo_fast because it was gated behind the training_step % local_eval_every == 0 modulo check; also guard local_eval_every <= 0 to prevent accidental every-step eval or ZeroDivisionError (#1485).
• Fixed TypeError in pack_padded_sequences when attention_mask is a float tensor, and vectorized the packing to avoid per-sequence host-device synchronizations (#1486).
• Fixed silent prompt/ground-truth mismatch in RLVR caused by redundant dataset shuffle desyncing the "index" column from positional indices, leading to wrong rewards and wrong exclude_index exclusions (#1484).
• Fixed test single_example_collator returning raw int for index, causing TypeError in _iter_batches (#1477).
• Fixed SFT integration test failing due to missing --try_launch_beaker_eval_jobs false flag (#1470).
• Fixed checkpoint cleanup race condition on shared filesystems by using ignore_errors=True and restricting cleanup to global rank 0 (#1468).
• Fixed checkpoint resume failing on Beaker retries by removing non-deterministic timestamp from exp_name (#1468).
• Fixed MFU calculation to count LM head FLOPs per token (#1457).
• Fixed training hang when inflight_updates is disabled by waiting for weight sync to complete before health check (#1454).
• Fixed evaluation responses being lost on timeout in grpo_fast by requeuing partial results (#1439).
• Beaker Experiment Launch now passes (#1424 (review)).

v0.1.0

We have made the following changes to open-instruct:

Added

• Added OLMo-core based DPO training script (#1391).
• Added SLURM scripts for OLMo SFT training with checkpoint resume support and configurable shuffle seed. #1368
• Added retry logic with exponential backoff to make_api_request for tool API calls (retries on timeouts, connection errors, 429, and 5xx). Also added configurable max_concurrency parameter to tool configs for controlling Ray actor concurrency per-tool. #1388
• Added support for generic MCP tools during training, with some limitations (no changing tools, no tool discovery during training). For details: #1384
• Added the ability to set active tools on a per-sample basis. See the PR for more details: #1382
• Added a new changelog Github Action that makes sure you contribute to the changelog! #1276
• Now, we type check open_instruct/dataset_transformation.py (#1390).
• Added a linter rule that imports go at the top of the file (#1394).
• Refactors GRPO config into a grpo_utils.py file in preparation for Olmo-core implementation (#1396_.
• Now, we save the generated rollouts to disk during RL when the --save_traces flag is passed (#1406).
• Pulls out weight sync code from GRPO into a more generic function (#1411 (review))

Changed

• Updated library versions via uv lock --upgrade (#1400).
• Now, large_test_script.sh exercises the tp > 1 code path (#1413).

Fixed

• Added automatic type coercion for tool arguments via safe_execute() - prevents crashes when models send wrong types (e.g., bool instead of string) (#1418).
• Fixed argparse conflict error for --save_traces by removing duplicate field definitions from StreamingDataLoaderConfig (#1416).
• Increased MetricsTracker max_metrics from 64 to 512 to fix ValueError: Exceeded maximum number of metrics when training with many tools or verifier functions (#1415).
• Fixed JSON serialization error in LocalDatasetTransformationCache.save_config when caching datasets locally (#1402).
• Now, we can support PRs from external contributors while still maintaining security for internal tokens (#1408).
• Improved error handling for tool calls with missing/invalid arguments - now returns a clear error message instead of crashing (#1404).
• Fixed GenerationConfig validation error when saving OLMo-3 models - config is now set after unwrapping the model, and OLMo-3 is detected from both chat_template_name and model name (#1404).
• Fixed the benchmark so that it runs (#1401).

Removed

• Removed open_instruct/ppo.py and related PPO training scripts (#1395).
• Removed scripts/train/debug/tool_grpo_fast.sh; use scripts/train/debug/tools/olmo_3_parser_multigpu.sh for tool use experiments (#1404).