The goal here is to be able to generate our training data on TRC using zephyr.

We only need to do this for the most recent document type, contacts-and-distances-v1. Previously I generated this data on a single node (a google cloud high memory node) and then uploaded it to huggingface here. The code I used to generate it is here.

I am porting the code to generate this document type over to MarinFold (#4 ) where it will live in document-structures/. The implementation there will have just the business logic of converting a structure (mmcif format) into a document (text), but it won't handle any of the zephyr stuff.

My thought here is to make data/ into a small library with the functionality needed to run any document structure's generation step on zephyr and store the results in gcs and/or huggingface. Then we would have a little experiment (initially in experiments/ and then graduating into data/ once done) that exercises the library to generate the contacts-and-distances-v1 dataset on zepyr.

This is blocked currently on me finishing the port of the implementation in document-structures/ but I will update this issue once that is done, which should be soon.