Skip to content

Commit f434a27

Browse files
jnweijnwei
authored
Merge pull request #439 from aqlaboratory/setup-improvements
Adds Documentation and minor quality of life fixes
2 parents 3eef7ca + d8117ce commit f434a27

47 files changed

Lines changed: 41897 additions & 1139 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/docker-image.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,7 @@ jobs:
1111
runs-on: ubuntu-latest
1212
steps:
1313
- uses: actions/checkout@v4
14+
- name: Cleanup # https://github.com/actions/virtual-environments/issues/2840
15+
run: sudo rm -rf /usr/share/dotnet && sudo rm -rf /opt/ghc && sudo rm -rf "/usr/local/share/boost" && sudo rm -rf "$AGENT_TOOLSDIRECTORY"
1416
- name: Build the Docker image
15-
run: docker build . --file Dockerfile --tag openfold:$(date +%s)
17+
run: docker build . --file Dockerfile --tag openfold:$(date +%s)

‎.readthedocs.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
version: 2
2+
3+
# Set the OS, Python version and other tools you might need
4+
build:
5+
os: ubuntu-22.04
6+
tools:
7+
python: "mambaforge-4.10"
8+
9+
# Build documentation in the "docs/" directory with Sphinx
10+
sphinx:
11+
configuration: docs/source/conf.py
12+
13+
conda:
14+
environment: docs/environment.yml

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,7 @@
187187
same "printed page" as the copyright notice for easier
188188
identification within third-party archives.
189189

190-
Copyright [yyyy] [name of copyright owner]
190+
Copyright 2024 AlQuraishi Laboratory
191191

192192
Licensed under the Apache License, Version 2.0 (the "License");
193193
you may not use this file except in compliance with the License.

‎README.md‎

Lines changed: 3 additions & 562 deletions
Large diffs are not rendered by default.

‎docs/Makefile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Minimal makefile for Sphinx documentation
2+
#
3+
4+
# You can set these variables from the command line, and also
5+
# from the environment for the first two.
6+
SPHINXOPTS ?=
7+
SPHINXBUILD ?= sphinx-build
8+
SOURCEDIR = source
9+
BUILDDIR = build
10+
11+
# Put it first so that "make" without argument is like "make help".
12+
help:
13+
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
14+
15+
.PHONY: help Makefile
16+
17+
# Catch-all target: route all unknown targets to Sphinx using the new
18+
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
19+
%: Makefile
20+
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

‎docs/environment.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
name: openfold-docs
2+
channels:
3+
- conda-forge
4+
dependencies:
5+
- sphinx=7
6+
- myst-parser=3
7+
- furo

‎docs/imgs/of_banner.png‎

1.84 MB
Loading

‎docs/source/Aux_seq_files.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Auxiliary Sequence Files for OpenFold Training
2+
3+
The training dataset of OpenFold is very large. The `pdb` directory alone contains 185,000 mmcifs; each chain for has multiple sequence alignment files and mmcif files.
4+
5+
OpenFold introduces a few new file structures for faster access to alignments and mmcif data.
6+
7+
This documentation will explain the benefits of having the condensed file structure, and explain the contents of each of the files.
8+
9+
## Default alignment file structure
10+
11+
One way to store mmcifs and alignments files would be to have a directory for each mmcif chain.
12+
13+
For example, consider two protein as a case study
14+
```
15+
- OpenProteinSet
16+
└── mmcifs
17+
├── 3lrm.cif
18+
└── 6kwc.cif
19+
...
20+
```
21+
22+
In the `alignments` directory, [PDB:6KWC](https://www.rcsb.org/structure/6KWC) is a monomer with one chain, and thus would have one alignment direcotry. [PDB:3LRM](https://www.rcsb.org/structure/3lrm), a homotetramer, would have one alignment directory for each of its four chains.
23+
```
24+
- OpenProteinSet
25+
└── alignments
26+
└── 3lrm_A
27+
├── bfd_uniclust_hits.a3m
28+
├── mgnify_hits.a3m
29+
├── pdb70_hits.hhr
30+
└── uniref90_hits.a3m
31+
└── 3lrm_B
32+
├── bfd_uniclust_hits.a3m
33+
├── mgnify_hits.a3m
34+
├── pdb70_hits.hhr
35+
└── uniref90_hits.a3m
36+
└── 3lrm_C
37+
├── bfd_uniclust_hits.a3m
38+
├── mgnify_hits.a3m
39+
├── pdb70_hits.hhr
40+
└── uniref90_hits.a3m
41+
└── 3lrm_D
42+
├── bfd_uniclust_hits.a3m
43+
├── mgnify_hits.a3m
44+
├── pdb70_hits.hhr
45+
└── uniref90_hits.a3m
46+
└── 6kwc_A
47+
├── bfd_uniclust_hits.a3m
48+
├── mgnify_hits.a3m
49+
├── pdb70_hits.hhr
50+
└── uniref90_hits.a3m
51+
...
52+
```
53+
54+
In practice, the IO overhead of having one directory per protein chain makes accessing the alignments slow.
55+
56+
## OpenFold DB file structure
57+
58+
Here we describe a new filesystem that can be used by OpenFold for more efficient access of alignment file and index file contents
59+
60+
All together, the file directory would look like:
61+
```
62+
- OpenProteinSet
63+
├── duplicate_pdb_chains.txt
64+
└── pdb
65+
├── mmcif_cache.json
66+
└── mmcifs
67+
├── 3lrm.cif
68+
└── 6kwc.cif
69+
└── alignment_db
70+
├── alignment_db_0.db
71+
├── alignment_db_1.db
72+
...
73+
├── alignment_db_9.db
74+
└── alignment_db.index
75+
```
76+
77+
We will describe each of the file types here.
78+
79+
### Alignments db files and index files
80+
81+
To speed up access of MSAs, OpenFold has an alternate alignments storage procedure. Instead of storing dedicated files for each single alignment, we consolidate large sets of alignments to single files referred to as _alignments_db's_. This can reduce I/O overhead and in practice we recommend using around 10 `alignments_db_x.db` files to store the total training set of OpenFold. During training, OpenFold can access each alignment using byte index pointers that are stored in a separate index file (`alignments_db.index`). The alignments for the `3LRM` and `6KWC` examples would be recorded in the index file as follows:
82+
83+
```alignments_db.index
84+
{
85+
...
86+
"3lrm_A": {
87+
"db": "alignment_db_0.db",
88+
"files": [
89+
["bfd_uniclust_hits.a3m", 212896478938, 1680200],
90+
["mgnify_hits.a3m", 212893696883, 2782055],
91+
["pdb70_hits.hhr", 212898159138, 614978],
92+
["uniref90_hits.a3m", 212898774116, 6165789]
93+
]
94+
},
95+
"6kwc_A": {
96+
"db": "alignment_db_1.db",
97+
"files": [
98+
["bfd_uniclust_hits.a3m", 415618723280, 380289],
99+
["mgnify_hits.a3m", 415618556077, 167203],
100+
["pdb70_hits.hhr", 415619103569, 148672],
101+
["uniref90_hits.a3m", 415617547852, 1008225]
102+
]
103+
}
104+
...
105+
}
106+
```
107+
108+
For each entry, the corresponding `alignment_db` file and the byte start location and number of bytes to read the respective alignments are given. For example, the alignment information in `bfd_uniclust_hits.a3m` for chain `3lrm_A` can be found in the database file `alignment_db_0.db`, starting at byte location `212896478938` and reading in the next `1680200` bytes.
109+
110+
### Chain cache files and mmCIF cache files
111+
112+
Information from the mmcif files can be parsed in advance to create a `chain_cache.json` or a `mmcif_cache.json`. For OpenFold, the `chain_cache.json` is used to sample chains for training, and the `mmcif_cache.json` is used to prefilter templates.
113+
114+
Here's what the chain_cache.json entry looks like for our examples:
115+
116+
```chain_cache.json
117+
{
118+
...
119+
"3lrm_A": {
120+
"release_date": "2010-06-30",
121+
"seq": "MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
122+
"resolution": 2.7,
123+
"cluster_size": 6
124+
},
125+
"3lrm_B": {
126+
"release_date": "2010-06-30",
127+
"seq": "MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
128+
"resolution": 2.7,
129+
"cluster_size": 6
130+
},
131+
"3lrm_C": {
132+
"release_date": "2010-06-30",
133+
"seq": "MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
134+
"resolution": 2.7,
135+
"cluster_size": 6
136+
},
137+
"3lrm_D": {
138+
"release_date": "2010-06-30",
139+
"seq": "MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
140+
"resolution": 2.7,
141+
"cluster_size": 6
142+
},
143+
"6kwc_A": {
144+
"release_date": "2021-01-27",
145+
"seq": "GSTIQPGTGYNNGYFYSYWNDGHGGVTYTNGPGGQFSVNWSNSGEFVGGKGWQPGTKNKVINFSGSYNPNGNSYLSVYGWSRNPLIEYYIVENFGTYNPSTGATKLGEVTSDGSVYDIYRTQRVNQPSIIGTATFYQYWSVRRNHRSSGSVNTANHFNAWAQQGLTLGTMDYQIVAVQGYFSSGSASITVS",
146+
"resolution": 1.297,
147+
"cluster_size": 195
148+
},
149+
...
150+
}
151+
```
152+
153+
The mmcif_cache.json file would contain similar information, but condensed by mmcif id, e.g.
154+
155+
```mmcif_cache.json
156+
{
157+
"3lrm": {
158+
"release_date": "2010-06-30",
159+
"chain_ids": [
160+
"A",
161+
"B",
162+
"C",
163+
"D"
164+
],
165+
"seqs": [
166+
"MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
167+
"MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
168+
"MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK",
169+
"MFAFYFLTACISLKGVFGVSPSYNGLGLTPQMGWDNWNTFACDVSEQLLLDTADRISDLGLKDMGYKYIILDDCWSSGRDSDGFLVADEQKFPNGMGHVADHLHNNSFLFGMYSSAGEYTCAGYPGSLGREEEDAQFFANNRVDYLKYANCYNKGQFGTPEISYHRYKAMSDALNKTGRPVFYSLCNWGQDLTFYWGSGIANSWRMSGDVTAEFTRPDSRCPCDGDEYDCKYAGFHCSIMNILNKAAPMGQNAGVGGWNDLDNLEVGVGNLTDDEEKAHFSMWAMVKSPLIIGANVNNLKASSYSIYSQASVIAINQDSNGIPATRVWRYYVSDTDEYGQGEIQMWSGPLDNGDQVVALLNGGSVSRPMNTTLEEIFFDSNLGSKKLTSTWDIYDLWANRVDNSTASAILGRNKTATGILYNATEQSYKDGLSKNDTRLFGQKIGSLSPNAILNTTVPAHGIAFYRLRPSSDYKDDDDK"
170+
],
171+
"no_chains": 4,
172+
"resolution": 2.7
173+
},
174+
"6kwc": {
175+
"release_date": "2021-01-27",
176+
"chain_ids": [
177+
"A"
178+
],
179+
"seqs": [
180+
"GSTIQPGTGYNNGYFYSYWNDGHGGVTYTNGPGGQFSVNWSNSGEFVGGKGWQPGTKNKVINFSGSYNPNGNSYLSVYGWSRNPLIEYYIVENFGTYNPSTGATKLGEVTSDGSVYDIYRTQRVNQPSIIGTATFYQYWSVRRNHRSSGSVNTANHFNAWAQQGLTLGTMDYQIVAVQGYFSSGSASITVS"
181+
],
182+
"no_chains": 1,
183+
"resolution": 1.297
184+
},
185+
...
186+
}
187+
```
188+
189+
190+
### Duplicate pdb chain files
191+
192+
Duplicate chains occur across pdb entries. Some of these chains are the homomeric units of a multimer, others are subunits that are shared across different protein.
193+
194+
To reduce storage overhead of creating / storing identical data for duplicate entries, we have a duplicate chain file. Each line stores the all chains that are identical. Our `6kwc` and `3lrm` examples would be stored as follows.
195+
196+
```duplicate_pdb_chains.txt
197+
...
198+
6kwc_A
199+
3lrm_A 3lrm_B 3lrm_C 3lrm_D
200+
...
201+
```
202+
203+

‎docs/source/FAQ.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# FAQ
2+
3+
Frequently asked questions or encountered issues when running OpenFold.
4+
5+
## Setup
6+
7+
- When running unit tests (e.g. [`./scripts/run_unit_tests.sh`](https://github.com/aqlaboratory/openfold/blob/main/scripts/run_unit_tests.sh)), I see an error such as
8+
```
9+
ImportError: version GLIBCXX_3.4.30 not found
10+
```
11+
12+
> Solution: Make sure that the `$LD_LIBRARY_PATH` environment has been set to include the conda path, e.g. `export $LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH`
13+
14+
- I see a CUDA mismatch error, eg.
15+
```
16+
The detected CUDA version (11.8) mismatches the version that was used to compile
17+
PyTorch (12.1). Please make sure to use the same CUDA versions.
18+
```
19+
20+
> Solution: Ensure that your system's CUDA driver and toolkit match your intended OpenFold installation (CUDA 11 by default). You can check the CUDA driver version with a command such as `nvidia-smi`
21+
22+
- I get some error involving `fatal error: cuda_runtime.h: No such file or directory` and or `ninja: build stopped: subcommand failed.`.
23+
24+
> Solution: Something went wrong with setting up some of the custom kernels. Try running `install_third_party_dependencies.sh` again or try `python3 setup.py install` from inside the OpenFold folder. Make sure to prepend the conda environment as described above before running this.
25+
26+
## Training
27+
28+
- My model training is hanging on the data loading step:
29+
> Solution: While each system is different, a few general suggestions:
30+
- Check your `$KMP_AFFINITY` environment setting and see if it is suitable for your system.
31+
- Adjust the number of data workers used to prepare data with the `--num_workers` setting. Increasing the number could help with dataset processing speed. However, to many workers could cause an OOM issue.
32+
33+
- When I reload my pretrained model weights or checkpoints, I get `RuntimeError: Error(s) in loading state_dict for OpenFoldWrapper: Unexpected key(s) in state_dict:`
34+
> Solution: This suggests that your checkpoint / model weights are in OpenFold v1 format with outdated model layer names. Convert your weights/checkpoints following [this guide](convert_of_v1_weights.md).

0 commit comments

Comments
 (0)