Before this repo, we have been working in a few places (an early name for this project was LlamaFold):
This repo has early experiments in training LlamaFold models using huggingface transformers. It also has our data generation pipeline. https://github.com/timodonnell/LlamaFold-experiments/tree/main
Later, we switched to marin and worked in this branch and experiment dir: https://github.com/marin-community/marin/tree/protein-training-1b/experiments/protein
The AFDB-to-documents pipeline (BigQuery selection, mmCIF download to parquet, Foldseek-based clusters, and the actual document generators for all three document formats — including the original contacts_and_distances_v1.py we are now porting into MarinFold) lives in:
https://github.com/timodonnell/contactdoc
Now, with the creation of this repo, we are renaming the project to MarinFold and working out of here. We want all data curation, evals, and model training to happen in the MarinFold repo.
There are two other important repos to know about:
https://github.com/marin-community/marin-experiments https://github.com/marin-community/marin
We want to follow the marin-experiments repo in that we will have independent little experiments that depend on marin. We also need to be aware of and watching the marin repo as it has the underlying implementations of many components we will use - iris, levanter, zephyr, etc. In particular we want to reuse the "experiment" infrastructure that Marin has set up, where we use issues to track experiments that are checked in as code.
So far we have been training entirely on AlphaFold Database (AFDB).
The “text” data (pre-tokenization) we are feeding these models is on huggingface protein-docs. Note there are different “subsets” (which the huggingface preview doesn’t seem to render correctly) corresponding to different document layouts that I’ve tried over time. The different layouts are documented in the README.
The underlying AFDB data we’ve curated is also on huggingface afdb-1.6M.
I was bad about saving the tokenizers for my early experiments, but the tokenizer for the “contacts-and-distances-v1” document type (currently the latest document type) is here. I plan to keep adding to this as we make new document types.
Checkpoints
For models that seem particularly good or interesting (both pre-Marin and after switching to Marin models), I’ve been uploading them to huggingface here.
Model names should include the wandb run name so we can connect the two.
With the rename to MarinFold we should now put interesting checkpoints on this bucket.
Weights and biases
After switching to Marin, runs are going here: https://wandb.ai/timodonnell/marin
Before Marin, I was sending runs to different projects for each experiment (names correspond to experiment dirs in the LlamaFold-experiments github repo): exp4, exp5, exp6
After the rename to MarinFold, let's put runs at https://wandb.ai/open-athena/MarinFold (will need to create the project first time it is used).
GCS
Large MarinFold experiment artifacts that don't fit in git (raw distograms, prediction batches, intermediate parquets — typically produced by iris jobs on TRC) live under
gs://marin-<region>/protein-structure/MarinFold/<experiment-name>/...
in the marin-<region> bucket matching the job's compute zone. For
example, our current v5p TPU train/eval jobs often use
marin-us-east5, while a CPU data-gen job pinned to us-central1-a
should use marin-us-central1; the region follows the pinned compute,
not the other way around. <experiment-name> should be descriptive
enough to be recognizable to the marin team (e.g.
exp26/protein-contacts-1_5b-distance-masked-70f8f5-step-49999-foldbench-monomers/).
See AGENTS.md "GCS bucket" and "Cross-region data transfers" for the
full convention. Small CSVs and plots that feed READMEs still live in
the experiment dir's data/ and plots/ — GCS is for the big stuff.