Skip to content

Commit aa29e21

Browse files
Nathan Lambertclaude
andauthored
Add model merging scripts for Beaker (#1459)
* Add model merging scripts for Beaker Two approaches for merging HuggingFace models on Beaker: - mergekit_merge.sh: Uses mergekit for standard architectures - direct_merge.sh: Direct safetensors averaging for all architectures, including hybrid models that mergekit doesn't support Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add changelog entry for merge scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Validate weights length and zero-sum in direct_merge.py Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Print merge config in Beaker logs, add test run links to README Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix direct_merge.py: update docstring path, remove unused import, use +=, allow tokenizer overwrite Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix direct_merge.sh: rename /tmp/linear_merge.py to /tmp/direct_merge.py Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Address PR review: refactor merge_models, add tests, document dual approach - Move scripts/merge/direct_merge.py to open_instruct/merge_models.py - Apply all code review fixes: Path types, logger, math.isclose, model_weights rename, helper functions, module-level parser - Add 14 unit tests (synthetic + pythia-14m integration) - Update shell scripts to reference new module path - Fix naming inconsistencies (linear_merge -> direct_merge) - Document why both mergekit and direct merge are needed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move launch_merges.sh example into README Address PR feedback: launch_merges.sh is an example script, not something that should be checked in. Moved its content into a "Batch launching" section in the README. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 25ac864 commit aa29e21

6 files changed

Lines changed: 646 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ All notable changes to this project will be documented in this file.
1313
- Wire RL environments into vLLM generation loop and preprocessing: unified tool/env system with single `TOOL_REGISTRY`, pooled actors via shared `EnvironmentPool` Ray actor (async acquire/release, auto-sized to rollout concurrency), `RolloutState` tracks all per-rollout state, `PassthroughVerifier` + `RewardAggregator` for per-turn rewards (verifier score folded into last turn before aggregation), `BaseEnvConfig` in `environments/base.py`, `--max_steps` unified, `--pool_size` configurable, auto-discovery of tools from datasets, 1-GPU debug scripts for counter/guess_number envs (https://github.com/allenai/open-instruct/pull/1479).
1414
- RL environment abstraction: `RLEnvironment` base class with `Tool` as a subclass, unifying tools and environments under a single `step(EnvCall) -> StepResult` interface. Removes `Executable`/`EnvOutput`/`_execute`/`safe_execute` indirection. Moves tools under `open_instruct/environments/tools/`. Includes example environments (`CounterEnv`, `GuessNumberEnv`) (https://github.com/allenai/open-instruct/pull/1478).
1515
- Enable packing with torch.compile for DPO training, fix cu_seq_lens offset bug for padded chosen/rejected sequences, add tokens_per_second_per_gpu metric (https://github.com/allenai/open-instruct/pull/1466).
16+
- Model merging scripts for Beaker: `mergekit_merge.sh` (mergekit) and `direct_merge.sh` (direct safetensors averaging for hybrid models) (https://github.com/allenai/open-instruct/pull/1459).
1617
- Production DPO script for OLMo3-7B hybrid (https://github.com/allenai/open-instruct/pull/1449).
1718
- Gradient accumulation/microbatching support for OLMo-core DPO training (https://github.com/allenai/open-instruct/pull/1447).
1819
- Evolving rubrics support with RubricVerifier and utility functions for GRPO training (https://github.com/allenai/open-instruct/pull/1460).

open_instruct/merge_models.py

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
"""
2+
Simple linear model merge by averaging safetensors weights.
3+
4+
This bypasses mergekit's architecture detection, which is useful for
5+
new model architectures that mergekit doesn't support yet.
6+
7+
Usage:
8+
python -m open_instruct.merge_models \
9+
--models /path/model1 /path/model2 /path/model3 \
10+
--output_dir /path/to/output \
11+
--model_weights 1.0 1.0 1.0
12+
"""
13+
14+
import argparse
15+
import math
16+
import shutil
17+
from pathlib import Path
18+
19+
import torch
20+
from safetensors import safe_open
21+
from safetensors.torch import save_file
22+
from tqdm import tqdm
23+
24+
from open_instruct import logger_utils
25+
26+
logger = logger_utils.setup_logger(__name__)
27+
28+
parser = argparse.ArgumentParser(description="Linear merge of HuggingFace models")
29+
parser.add_argument("--models", nargs="+", required=True, help="Paths to models to merge")
30+
parser.add_argument(
31+
"--model_weights", nargs="+", type=float, default=None, help="Weights for each model (default: equal weights)"
32+
)
33+
parser.add_argument("--output_dir", required=True, help="Output directory for merged model")
34+
parser.add_argument(
35+
"--dtype", default="bfloat16", choices=["bfloat16", "float16", "float32"], help="Output dtype (default: bfloat16)"
36+
)
37+
38+
39+
def get_safetensor_files(model_path: Path) -> list[str]:
40+
"""Get sorted list of safetensor filenames in a model directory."""
41+
return sorted(f.name for f in model_path.glob("*.safetensors"))
42+
43+
44+
def _load_tensors(model_paths: list[Path], safetensor_file: str) -> list[dict[str, torch.Tensor]]:
45+
"""Load tensors from all models for a single safetensor file."""
46+
model_tensors = []
47+
for model_path in model_paths:
48+
sf_path = model_path / safetensor_file
49+
with safe_open(str(sf_path), framework="pt", device="cpu") as f:
50+
tensors = {key: f.get_tensor(key) for key in f.keys()} # noqa: SIM118 (safe_open is not iterable)
51+
model_tensors.append(tensors)
52+
return model_tensors
53+
54+
55+
def _merge_tensors(
56+
model_tensors: list[dict[str, torch.Tensor]], model_weights: list[float], torch_dtype: torch.dtype
57+
) -> dict[str, torch.Tensor]:
58+
"""Merge tensors from multiple models using weighted averaging."""
59+
tensor_names = list(model_tensors[0].keys())
60+
merged_tensors = {}
61+
for name in tensor_names:
62+
merged = torch.zeros_like(model_tensors[0][name], dtype=torch_dtype)
63+
for i, tensors in enumerate(model_tensors):
64+
merged += tensors[name].to(torch_dtype) * model_weights[i]
65+
merged_tensors[name] = merged
66+
return merged_tensors
67+
68+
69+
def merge_models(
70+
model_paths: list[Path],
71+
output_dir: Path,
72+
model_weights: list[float] | None = None,
73+
dtype: torch.dtype = torch.bfloat16,
74+
) -> None:
75+
"""Merge models by weighted averaging of their safetensors weights."""
76+
model_weights = model_weights or [1.0] * len(model_paths)
77+
78+
if len(model_weights) != len(model_paths):
79+
raise ValueError(f"Number of weights ({len(model_weights)}) must match number of models ({len(model_paths)})")
80+
81+
total_weight = sum(model_weights)
82+
if math.isclose(total_weight, 0.0):
83+
raise ValueError("Weights must not sum to zero")
84+
model_weights = [w / total_weight for w in model_weights]
85+
86+
logger.info(f"Merging {len(model_paths)} models with weights: {model_weights}")
87+
88+
safetensor_files = get_safetensor_files(model_paths[0])
89+
logger.info(f"Found {len(safetensor_files)} safetensor files")
90+
91+
# Verify all models have identical safetensor file sets
92+
for model_path in model_paths[1:]:
93+
other_files = get_safetensor_files(model_path)
94+
if other_files != safetensor_files:
95+
raise ValueError(f"Model {model_path} has different safetensor files: {other_files} vs {safetensor_files}")
96+
97+
output_dir.mkdir(parents=True, exist_ok=True)
98+
99+
for safetensor_file in tqdm(safetensor_files, desc="Processing files"):
100+
model_tensors = _load_tensors(model_paths, safetensor_file)
101+
merged_tensors = _merge_tensors(model_tensors, model_weights, dtype)
102+
save_file(merged_tensors, str(output_dir / safetensor_file))
103+
104+
# Copy config files from first model
105+
config_files = ["config.json", "generation_config.json", "model.safetensors.index.json"]
106+
for f in config_files:
107+
src = model_paths[0] / f
108+
if src.exists():
109+
shutil.copy(src, output_dir / f)
110+
logger.info(f"Copied {f}")
111+
112+
# Copy tokenizer files from first model
113+
tokenizer_files = [
114+
"tokenizer.json",
115+
"tokenizer.model",
116+
"tokenizer_config.json",
117+
"special_tokens_map.json",
118+
"added_tokens.json",
119+
"vocab.json",
120+
"merges.txt",
121+
"chat_template.jinja",
122+
]
123+
for f in tokenizer_files:
124+
src = model_paths[0] / f
125+
if src.exists():
126+
shutil.copy(src, output_dir / f)
127+
logger.info(f"Copied {f}")
128+
129+
logger.info(f"Merge complete! Output at: {output_dir}")
130+
131+
132+
def main() -> None:
133+
args = parser.parse_args()
134+
merge_models(
135+
model_paths=[Path(p) for p in args.models],
136+
output_dir=Path(args.output_dir),
137+
model_weights=args.model_weights,
138+
dtype=getattr(torch, args.dtype),
139+
)
140+
141+
142+
if __name__ == "__main__":
143+
main()

scripts/merge/README.md

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Model Merging Guide
2+
3+
Merge HuggingFace models on Beaker using either [mergekit](https://github.com/arcee-ai/mergekit) or direct safetensors averaging.
4+
5+
## Files
6+
7+
| File | Description |
8+
|------|-------------|
9+
| `mergekit_merge.sh` | Beaker launcher using mergekit (for supported architectures) |
10+
| `direct_merge.sh` | Beaker launcher using direct safetensors averaging (for all architectures) |
11+
12+
## Why Two Approaches?
13+
14+
We maintain both scripts because they serve complementary roles:
15+
16+
- **`mergekit_merge.sh`** wraps [mergekit](https://github.com/arcee-ai/mergekit), which supports advanced merge methods (`linear`, `slerp`, `ties`, `dare_ties`). However, mergekit relies on architecture-specific layer detection, so it only works with architectures it already knows about (e.g., Llama, OLMo3).
17+
- **`direct_merge.sh`** uses `open_instruct/merge_models.py` to do architecture-agnostic weighted averaging of safetensors files. It works with *any* architecture — including hybrid models like `Olmo3_5HybridForCausalLM` — but only supports linear (weighted average) merging.
18+
19+
In short: use **mergekit** when you need a complex merge method on a supported architecture, and **direct merge** when you need to merge models whose architecture mergekit doesn't support yet.
20+
21+
## Usage
22+
23+
Both scripts take the same arguments:
24+
25+
```bash
26+
./scripts/merge/<script>.sh <output_dir> <model1> <model2> [model3] ...
27+
```
28+
29+
Models can be local paths or HuggingFace model IDs (mergekit only).
30+
31+
### Mergekit
32+
33+
```bash
34+
./scripts/merge/mergekit_merge.sh \
35+
/weka/oe-adapt-default/nathanl/merged/my-merge \
36+
/weka/path/to/model1 \
37+
/weka/path/to/model2
38+
```
39+
40+
### Direct Merge
41+
42+
```bash
43+
./scripts/merge/direct_merge.sh \
44+
/weka/oe-adapt-default/nathanl/merged/my-merge \
45+
/weka/path/to/model1 \
46+
/weka/path/to/model2
47+
```
48+
49+
## How It Works
50+
51+
Both scripts use base64-encoding to safely pass configs/scripts through the shell layers (bash -> mason.py -> Beaker), avoiding YAML escaping issues. On Beaker, the encoded string is decoded and executed.
52+
53+
- `mergekit_merge.sh`: Base64-encodes a YAML config, installs mergekit on Beaker, and runs `mergekit-yaml`
54+
- `direct_merge.sh`: Base64-encodes `open_instruct/merge_models.py`, sends it to Beaker, and runs it directly
55+
56+
Both scripts copy tokenizer files and `chat_template.jinja` from the first model.
57+
58+
### Custom Transformers
59+
60+
For models that require a custom transformers build (e.g., new OLMo architectures not yet in released transformers), `mergekit_merge.sh` has commented-out variables for installing from a PR:
61+
62+
```bash
63+
# In mergekit_merge.sh - uncomment and set PR number as needed
64+
# CUSTOM_INSTALL="uv pip install git+https://github.com/huggingface/transformers.git@refs/pull/XXXXX/head"
65+
# INSTALL_CMD="${CUSTOM_INSTALL} && uv pip install mergekit"
66+
```
67+
68+
## Examples
69+
70+
### Mergekit with HuggingFace models
71+
72+
[Test run](https://beaker.org/ex/01KGK2AWYGFY5E6R0M3F94NN49)
73+
74+
```bash
75+
./scripts/merge/mergekit_merge.sh \
76+
/weka/oe-adapt-default/nathanl/merged/test-mergekit \
77+
allenai/Olmo-3-7B-Think-DPO \
78+
allenai/Olmo-3-7B-Think-SFT
79+
```
80+
81+
### Mergekit with local weka checkpoints
82+
83+
[Test run](https://beaker.org/orgs/ai2/workspaces/olmo-instruct/work/01KGK1QXQ7XQHMC0NPKK5HG52A)
84+
85+
```bash
86+
./scripts/merge/mergekit_merge.sh \
87+
/weka/oe-adapt-default/nathanl/merged/test-mergekit-olmo3 \
88+
/weka/oe-adapt-default/saumyam/checkpoints/olmo2-7B-sft/rl-sft/olmo3-32b-SFT-1e-4/step10500-hf \
89+
/weka/oe-adapt-default/saumyam/checkpoints/olmo2-7B-sft/rl-sft/olmo3-32b-SFT-1e-4/step9000-hf
90+
```
91+
92+
### Direct merge for hybrid models
93+
94+
[Test run](https://beaker.org/orgs/ai2/workspaces/olmo-instruct/work/01KGK1BBTG0Q4MFHMK6TDJNH9X)
95+
96+
```bash
97+
# 2-model merge
98+
./scripts/merge/direct_merge.sh \
99+
/weka/oe-adapt-default/nathanl/merged/sft-2model-linear \
100+
/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR2.5e-5/step46412-hf \
101+
/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR4.5e-5_seed42/step46412-hf
102+
103+
# 3-model merge
104+
./scripts/merge/direct_merge.sh \
105+
/weka/oe-adapt-default/nathanl/merged/sft-3model-linear \
106+
/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR2.5e-5/step46412-hf \
107+
/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR4.5e-5_seed42/step46412-hf \
108+
/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR1e-5/step46412-hf
109+
```
110+
111+
### Batch launching multiple merges
112+
113+
```bash
114+
#!/bin/bash
115+
set -euo pipefail
116+
117+
BEAKER_USER=$(beaker account whoami --format json | jq -r '.[0].name')
118+
119+
MODEL_1="/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR2.5e-5/step46412-hf"
120+
MODEL_2="/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR4.5e-5_seed42/step46412-hf"
121+
MODEL_3="/weka/oe-adapt-default/nathanl/checkpoints/TEST_HYBRIC_SFT_LARGER_LR1e-5/step46412-hf"
122+
123+
# 2-model merge
124+
./scripts/merge/direct_merge.sh \
125+
"/weka/oe-adapt-default/${BEAKER_USER}/merged/sft-2model-linear" \
126+
"$MODEL_1" "$MODEL_2"
127+
128+
# 3-model merge
129+
./scripts/merge/direct_merge.sh \
130+
"/weka/oe-adapt-default/${BEAKER_USER}/merged/sft-3model-linear" \
131+
"$MODEL_1" "$MODEL_2" "$MODEL_3"
132+
```
133+
134+
## Notes
135+
136+
- **MergeKit** doesn't support hybrid models (e.g., `Olmo3_5HybridForCausalLM`) because it expects uniform layer structure. Use `direct_merge.sh` for those.
137+
- Weights are normalized by default, so `1.0 + 1.0 + 1.0` gives equal (33.3%) weight to each model
138+
- For different ratios, use different weights (e.g., `2.0 + 1.0` = 66.7% / 33.3%)

scripts/merge/direct_merge.sh

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/bin/bash
2+
#
3+
# Launch a linear model merge job on Beaker (bypasses mergekit)
4+
#
5+
# This script directly averages safetensors weights, which works for
6+
# new architectures that mergekit doesn't support yet (e.g., Olmo3_5Hybrid).
7+
#
8+
# Usage: ./scripts/merge/direct_merge.sh <output_dir> <model1> <model2> [model3] ...
9+
#
10+
# Example:
11+
# ./scripts/merge/direct_merge.sh /weka/oe-adapt-default/nathanl/merged/my-merge \
12+
# /weka/path/to/model1 \
13+
# /weka/path/to/model2
14+
#
15+
set -euo pipefail
16+
17+
if [ $# -lt 3 ]; then
18+
echo "Usage: $0 <output_dir> <model1> <model2> [model3] ..."
19+
echo ""
20+
echo "Example:"
21+
echo " $0 /weka/oe-adapt-default/user/merged/my-merge /weka/path/model1 /weka/path/model2"
22+
exit 1
23+
fi
24+
25+
IMAGE="nathanl/open_instruct_auto"
26+
OUTPUT_DIR="$1"
27+
shift
28+
MODELS=("$@")
29+
NUM_MODELS=${#MODELS[@]}
30+
31+
# Build the models argument string
32+
MODELS_ARG=""
33+
for model in "${MODELS[@]}"; do
34+
MODELS_ARG="$MODELS_ARG $model"
35+
done
36+
37+
echo "=============================================="
38+
echo "Launching ${NUM_MODELS}-model linear merge"
39+
echo "Output: $OUTPUT_DIR"
40+
echo "Models:"
41+
for model in "${MODELS[@]}"; do
42+
echo " - $model"
43+
done
44+
echo "=============================================="
45+
46+
# Base64 encode the Python module to avoid escaping issues
47+
SCRIPT_B64=$(base64 < open_instruct/merge_models.py | tr -d '\n')
48+
49+
uv run python mason.py \
50+
--cluster ai2/jupiter \
51+
--budget ai2/oe-adapt \
52+
--workspace ai2/olmo-instruct \
53+
--image "$IMAGE" \
54+
--pure_docker_mode \
55+
--no-host-networking \
56+
--gpus 0 \
57+
--priority normal \
58+
--description "Linear merge: ${NUM_MODELS}-model" \
59+
-- bash -c \"cd /stage \&\& echo $SCRIPT_B64 \| base64 -d \> /tmp/merge_models.py \&\& uv run python /tmp/merge_models.py --models $MODELS_ARG --output_dir $OUTPUT_DIR --dtype bfloat16\"

0 commit comments

Comments
 (0)