This repository contains additional code and data for the paper "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension".
All OlmPool models were pretrained and context extended with OLMo-core. The original checkpoints are available in OLMo-core format on Google cloud. For convenience, we also convert checkpoints to Hugging Face format; you can access those checkpoints from the allenai/olmpool collection.
Note that these models are early in pretraining with little-to-no instruction-format data, and thus are very poor at most tasks. The final checkpoint for each model is a 7-8B model that has been trained to 150B tokens (140B in pretraining and 10B in context extension).
The training configuration for each model's pretraining run is available in src/configs. Names here are identical to the names on the Hugging Face Hub for each model.
To retrain any of these models (or to load these models in OLMo-core), firrst replace the config.py in OLMo-core with the provided configs/config.py to add the new model classes.
Scripts to replicate the analysis by running generation through OLMo-core will be released shortly. All evaluations were run through OLMo-core; for exact replications of evals, run generation with the OLMo-core checkpoints.
We're happy to chat! Please open an issue.
@misc{bertsch2026cracks,
title={Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension},
author={Amanda Bertsch and Luca Soldaini and Matthew R. Gormley and Graham Neubig and Hanna Hajishirzi and Kyle Lo and Dirk Groeneveld},
year={2026},
}