In classic Appion, our end-users (the microscopists) frequently encounter conceptual issues with run directories and what a run should be used for. Specifically, the following constraints apply to (or have applied to) a run directory:
- By convention, each run directory operates on only one microscope preset (e.g.,
enn, esn). Occasionally, one of the microscopists will process multiple presets in the same run directory and this has broken code that I'd added to classic Appion that assumed only one preset (e.g., a done marker file that only applied to one preset).
- By convention, each run directory uses only one alignment label (e.g.,
-a, -b).
- Each run directory is supposed to represent a different set of parameters passed in (e.g., bin size, etc).
What if, instead of there being multiple run directories (one for each preset, alignment label, set of input flags), there were only one run directory per session? This would reduce confusion and assumptions about how data is organized on disk, and also give us the ability to make some optimizations.
Namely, I was thinking that we could hash the inputs passed into the motioncor2 command line interface and use the resulting hash code as a machine-friendly label to prevent accidentally recomputing the same results twice. For instance, if someone were to start two Appion instances and pass each of them the same parameters except for the label, Appion could simply check a cache directory to see if motioncor2 had already run and then make a hard/soft link.
The main caveat here is that multiple Appion instances might want to run at the same time, and there would need to be a way to allow them to coordinate / not step on each other if they're sharing the same working directory.
In classic Appion, our end-users (the microscopists) frequently encounter conceptual issues with run directories and what a run should be used for. Specifically, the following constraints apply to (or have applied to) a run directory:
enn,esn). Occasionally, one of the microscopists will process multiple presets in the same run directory and this has broken code that I'd added to classic Appion that assumed only one preset (e.g., a done marker file that only applied to one preset).-a,-b).What if, instead of there being multiple run directories (one for each preset, alignment label, set of input flags), there were only one run directory per session? This would reduce confusion and assumptions about how data is organized on disk, and also give us the ability to make some optimizations.
Namely, I was thinking that we could hash the inputs passed into the
motioncor2command line interface and use the resulting hash code as a machine-friendly label to prevent accidentally recomputing the same results twice. For instance, if someone were to start two Appion instances and pass each of them the same parameters except for the label, Appion could simply check a cache directory to see ifmotioncor2had already run and then make a hard/soft link.The main caveat here is that multiple Appion instances might want to run at the same time, and there would need to be a way to allow them to coordinate / not step on each other if they're sharing the same working directory.