Setting up the stage for temporal workflow integration

[ka-sarthak]

AutoXRD is well-suited for temporal workflows, since it can be long-running. Training can be lengthy, depending on the number of epochs and the size of the training data. Inference can also be long, depending on how many XRD patterns are being processed together.

To this end, some restructuring is needed, primarily restructuring current training and analysis modules into functions that can be later used to define activities. These functions can be set up as:

• Data preprocessing for training data - CIFs are the input for training and need to be processed into spectrums while also adding data augmentation. These functions will be based on the spectrum_generation, solid_solns, and tabulate_cif modules of the XRD-AutoAnalyser package.
• Data batching for training
• Running the training loop and saving the models
• Data preprocessing for inference data - XRD data entries from NOMAD are the input for the inference and need to be processed into spectra that can be used for analysis. These functions extract the data from the entry, interpolate the pattern to match the input size for the model, and convert it into a PDF if using a PDF model.
• Data postprocessing after inference - The predicted phases are processed together with the original pattern for postprocessing. These functions will be based on the SpectrumAnalyser.enumerate_routes method of the XRD-AutoAnalyser package.
• Data batching for running inference

The current functions for training and analysing using NOMAD entries will need to be adapted and made part of schema_packages.jupyter

[ka-sarthak]

• An activity to load models from a different upload

[ka-sarthak]

To get started, we should implement an interface for running the workflow. This will involve two new schemas: one for training and one for inference. The temporal workflow can simply include the current training and inference code from the Jupyter notebooks. The granularization of the workflow into more detailed activities can be taken up later. Now implemented #27