How many CPU hours did it cost to generate the pretraining data?

[ekiefl]

I ran the following experiment:

• Domain: GB1 (56AA)
• Variants: 1000
• Cores: 12 (all being used)
• Environment: local MacOS

The runtime was 0.87 hours.

That's a rate of 0.01 CPU hours / variant (or 0.6 minutes / variant), which I would consider best-case-scenario, since 56AA is quite a small protein.

You guys generated 20M variants for 148 proteins = 3B total variants. At that rate, we're looking at least 30M CPU hours... It seems impractical, right? Or does this line up with your runtimes?

A few considerations:

• I'm using my own re-implementation of this workflow, since I found it hard this codebase hard to plug into. Happy to share my code if you're interested.
• One of the biggest differences is that I'm running Rosetta through their linux/amd64 Docker image. But since I'm on an M3 mac chip, maybe the emulation is slow?

[agitter]

A couple relevant snippets from the preprint:

For each local pretraining dataset, we generated ∼20M protein sequence variants with a maximum of 5 amino acid substitutions.

and

For each of the remaining 148 structures (Table S2), we generated ∼200K variants with a maximum of 5 amino acid substitutions, for a total of ∼30M variants.

The METL-Local datasets used 20M variants per protein, which is more than is typically needed (Figure S13). However, the METL-Global dataset only used 200K per protein.

Our response to reviewers letter contained some timing information:

Rosetta simulation times scale with protein sequence length. Average runtimes per variant ranged from ~37-50 seconds for smaller proteins (56-102 residues, e.g., GB1, GRB2, Pab1, Ube4b) to 135-215 seconds for larger proteins (237-403 residues, e.g., GFP, PTEN, TEM-1). Based on these timings and the reduced pretraining data requirement of 512K examples, the total compute time ranges from 800 to 1,274 compute-days depending on protein size. Since this workload is highly parallelizable, wall-clock time can be substantially reduced through distributed computing. Our runs on OSG routinely used upwards of 5,000 CPU cores. Our new Jupyter notebook provides this access to OSG to eligible readers.

Because we generated the Rosetta datasets in OSG, it was somewhat challenging to track exact cumulative runtimes. The runtimes for all of the small batches of Rosetta jobs are logged in a database, but we do not have the complete CPU hours tallied.

[ekiefl]

Hey @agitter, sorry for the late response, it seems my message never sent.

This is quite a monumental dataset and surprisingly expensive to run at the scale you guys did.

In the end, I ended up re-writing this repo, which allows for these features:

• Rosetta Docker containerization
• Snakemake workflow for single node computation
• Nextflow workflow for single node computation + scaling out to thousands of AWS Batch nodes

It's probably not exactly an apples-to-apples comparison, given inevitable differences in the codebases, but I'll be able to provide some rough CPU hour costs soon. I'm also planning on open sourcing the repo as soon as it passes our public repo standards.

If you're interested, I'd like to fill you in on what we're planning and how we're potentially building upon METL. Please let me know if you would be interested in getting involved in any capacity!

[agitter]

@ekiefl very cool! It would be great to talk more so we can see if there are opportunities to work together on shared resources.

[ekiefl]

Wonderful, I sent you an email!