Running spIsoNet for 1.5 mil particle dataset

[dbuckley-git]

Hi,

I'm excited to be using your procedure! I have been working to optimize my SLURM resources to run 3D autorefine using spIsoNet on my dataset of ~1.5 million particles. We have a 2-day limit for submitting jobs, and it looks like it finished iteration 12 in that time. I was astonished to see the estimated time for iteration 13 to be 34 hours! Maybe I am missing the point in running spIsoNet -- do I need to submit all 1.5 mil particles? This dataset went to ~2.1 A in CryoSPARC, and has some minor orientation bias that I think would be nicely cleaned-up by spIsoNet.

I submitted the job with 220Gb memory over 2 A100 GPUs (I believe our cluster is limited to 2 GPUs), and here are the MPI details:

=== RELION MPI setup ===

• Number of MPI processes = 5
• Number of threads per MPI process = 12
• Total number of threads therefore = 60
• Leader (0) runs on host = g004.mgmt.hellbender
• Follower 4 runs on host = g004.mgmt.hellbender
  =================
• Follower 1 runs on host = g004.mgmt.hellbender
• Follower 2 runs on host = g004.mgmt.hellbender
• Follower 3 runs on host = g004.mgmt.hellbender

Here's some sample output for the last iteration it was working on:

• External reconstruction finished successfully, reading result back in ...
  18.42/18.42 min ............................................................(,_,">
  Auto-refine: Iteration= 13
  Auto-refine: Resolution= 2.91347 (no gain for 0 iter)
  Auto-refine: Changes in angles= 6.79735 degrees; and in offsets= 0.564594 Angstroms (no gain for 0 iter)
  Estimating accuracies in the orientational assignment ...
  3/ 3 sec ............................................................(,,">
  Auto-refine: Estimated accuracy angles= 0.635 degrees; offsets= 0.194345 Angstroms
  Auto-refine: Angular step= 1.875 degrees; local searches= false
  Auto-refine: Offset search range= 1.74704 Angstroms; offset step= 0.21838 Angstroms
  CurrentResolution= 2.91347 Angstroms, which requires orientationSampling of at least 1.90476 degrees for a particle of diameter 175 Angstroms
  Oversampling= 0 NrHiddenVariableSamplingPoints= 14450688
  OrientationalSampling= 3.75 NrOrientations= 294912
  TranslationalSampling= 0.436759 NrTranslations= 49
  =============================
  Oversampling= 1 NrHiddenVariableSamplingPoints= 462422016
  OrientationalSampling= 1.875 NrOrientations= 2359296
  TranslationalSampling= 0.21838 NrTranslations= 196
  =============================
  Expectation iteration 13
  25.14/34.92 hrs ...........................................~~(,,">slurmstepd: error: *** JOB 5205596 ON g004 CANCELLED AT 2024-12-14T16:45:24 DUE TO TIME LIMIT ***

Thanks,
David

[procyontao]

Hi David,

I sometime observe that for some Relion iterations the expectation time is much longer than it should be. Right now, I do not have an idea why this happen. But certainly something done by spIsoNet is not compatible with Relion, I will try to figure this out.

In the current situation, you may want to continue the Relion auto refine with local searches or find some people that know Relion better.

[dbuckley-git]

Hello Dr. Liu, thank you for your response. It turns out, I have access to more GPUs than I thought. From some testing, this is what I've seen for the expected time for Iteration 13 for my dataset:
2 GPUs --> 35 hours (A100)
4 GPUs --> 16 hours (A100)
8 GPUs --> 5 hours (H100)

I also noticed that if I indicate --gpu "0,1,2,3", then the threads are mapped in a shared manner across the GPUs, for example:

=== RELION MPI setup ===

• Number of MPI processes = 5
• Number of threads per MPI process = 12
• Total number of threads therefore = 60
• Leader (0) runs on host = g013.mgmt.hellbender
• Follower 1 runs on host = g013.mgmt.hellbender
• Follower 2 runs on host = g013.mgmt.hellbender
• Follower 3 runs on host = g013.mgmt.hellbender
• Follower 4 runs on host = g013.mgmt.hellbender
  =================
  uniqueHost g013.mgmt.hellbender has 4 ranks.
  Follower 1 will distribute threads over devices 0 1 2 3
  Thread 0 on follower 1 mapped to device 0
  Thread 1 on follower 1 mapped to device 1
  Thread 2 on follower 1 mapped to device 2
  Thread 3 on follower 1 mapped to device 3
  Thread 4 on follower 1 mapped to device 0
  Thread 5 on follower 1 mapped to device 1
  Thread 6 on follower 1 mapped to device 2
  Thread 7 on follower 1 mapped to device 3
  Thread 8 on follower 1 mapped to device 0
  Thread 9 on follower 1 mapped to device 1
  Thread 10 on follower 1 mapped to device 2
  Thread 11 on follower 1 mapped to device 3
  Follower 2 will distribute threads over devices 0 1 2 3
  Thread 0 on follower 2 mapped to device 0
  Thread 1 on follower 2 mapped to device 1
  Thread 2 on follower 2 mapped to device 2
  Thread 3 on follower 2 mapped to device 3
  Thread 4 on follower 2 mapped to device 0
  Thread 5 on follower 2 mapped to device 1
  Thread 6 on follower 2 mapped to device 2
  Thread 7 on follower 2 mapped to device 3
  Thread 8 on follower 2 mapped to device 0
  Thread 9 on follower 2 mapped to device 1
  Thread 10 on follower 2 mapped to device 2
  Thread 11 on follower 2 mapped to device 3
  Follower 3 will distribute threads over devices 0 1 2 3
  Thread 0 on follower 3 mapped to device 0
  Thread 1 on follower 3 mapped to device 1
  Thread 2 on follower 3 mapped to device 2
  Thread 3 on follower 3 mapped to device 3
  Thread 4 on follower 3 mapped to device 0
  Thread 5 on follower 3 mapped to device 1
  Thread 6 on follower 3 mapped to device 2
  Thread 7 on follower 3 mapped to device 3
  Thread 8 on follower 3 mapped to device 0
  Thread 9 on follower 3 mapped to device 1
  Thread 10 on follower 3 mapped to device 2
  Thread 11 on follower 3 mapped to device 3
  Follower 4 will distribute threads over devices 0 1 2 3
  Thread 0 on follower 4 mapped to device 0
  Thread 1 on follower 4 mapped to device 1
  Thread 2 on follower 4 mapped to device 2
  Thread 3 on follower 4 mapped to device 3
  Thread 4 on follower 4 mapped to device 0
  Thread 5 on follower 4 mapped to device 1
  Thread 6 on follower 4 mapped to device 2
  Thread 7 on follower 4 mapped to device 3
  Thread 8 on follower 4 mapped to device 0
  Thread 9 on follower 4 mapped to device 1
  Thread 10 on follower 4 mapped to device 2
  Thread 11 on follower 4 mapped to device 3
  Device 0 on g013.mgmt.hellbender is split between 4 followers
  Device 1 on g013.mgmt.hellbender is split between 4 followers
  Device 2 on g013.mgmt.hellbender is split between 4 followers
  Device 3 on g013.mgmt.hellbender is split between 4 followers
  The following warnings were encountered upon command-line parsing:
  WARNING: Option --keep_lowres is not a valid RELION argument
  Running CPU instructions in double precision.

In contrast, when I just use --gpu "" (as is indicated in the example Relion command in the tutorial; I used the comma syntax because it is shown is the 'spisonet.py reconstruct' command) the threads are uniquely mapped, and I don't get the 'Device is split' message:

=== RELION MPI setup ===

• Number of MPI processes = 5
• Number of threads per MPI process = 12
• Total number of threads therefore = 60
• Leader (0) runs on host = g002.mgmt.hellbender
  =================
  uniqueHost g002.mgmt.hellbender has 4 ranks.
  Follower 1 will distribute threads over devices 0
  Thread 0 on follower 1 mapped to device 0
  Thread 1 on follower 1 mapped to device 0
  Thread 2 on follower 1 mapped to device 0
  Thread 3 on follower 1 mapped to device 0
  Thread 4 on follower 1 mapped to device 0
  Thread 5 on follower 1 mapped to device 0
  Thread 6 on follower 1 mapped to device 0
  Thread 7 on follower 1 mapped to device 0
  Thread 8 on follower 1 mapped to device 0
  Thread 9 on follower 1 mapped to device 0
  Thread 10 on follower 1 mapped to device 0
  Thread 11 on follower 1 mapped to device 0
  Follower 2 will distribute threads over devices 1
  Thread 0 on follower 2 mapped to device 1
  Thread 1 on follower 2 mapped to device 1
  Thread 2 on follower 2 mapped to device 1
  Thread 3 on follower 2 mapped to device 1
  Thread 4 on follower 2 mapped to device 1
  Thread 5 on follower 2 mapped to device 1
  Thread 6 on follower 2 mapped to device 1
  Thread 7 on follower 2 mapped to device 1
  Thread 8 on follower 2 mapped to device 1
  Thread 9 on follower 2 mapped to device 1
  Thread 10 on follower 2 mapped to device 1
  Thread 11 on follower 2 mapped to device 1
  Follower 3 will distribute threads over devices 2
  Thread 0 on follower 3 mapped to device 2
  Thread 1 on follower 3 mapped to device 2
  Thread 2 on follower 3 mapped to device 2
  Thread 3 on follower 3 mapped to device 2
  Thread 4 on follower 3 mapped to device 2
  Thread 5 on follower 3 mapped to device 2
  Thread 6 on follower 3 mapped to device 2
  Thread 7 on follower 3 mapped to device 2
  Thread 8 on follower 3 mapped to device 2
  Thread 9 on follower 3 mapped to device 2
  Thread 10 on follower 3 mapped to device 2
  Thread 11 on follower 3 mapped to device 2
  Follower 4 will distribute threads over devices 3
  Thread 0 on follower 4 mapped to device 3
  Thread 1 on follower 4 mapped to device 3
  Thread 2 on follower 4 mapped to device 3
  Thread 3 on follower 4 mapped to device 3
  Thread 4 on follower 4 mapped to device 3
  Thread 5 on follower 4 mapped to device 3
  Thread 6 on follower 4 mapped to device 3
  Thread 7 on follower 4 mapped to device 3
  Thread 8 on follower 4 mapped to device 3
  Thread 9 on follower 4 mapped to device 3
  Thread 10 on follower 4 mapped to device 3
  Thread 11 on follower 4 mapped to device 3
  Running CPU instructions in double precision.

I noticed that Relion 3D auto-refine will start from the first iteration regardless of the mapping, but using --gpu "" was critically important for running a 'continue' job from Iteration 12. If I used --gpu "0,1,2,3", it always would run out of memory at the start. Hopefully this info is helpful for someone else! By the way, I am using this Relion version: relion/4.0.1_gcc_9.5.0_cuda.

When you run the 'continue' job, you get a number of those 'Option -- xxxxxx is not a valid RELION argument', but I noticed that spIsoNet is still called and runs. Am I right to assume all those parameters are retained when running a 'continue' job?

Thank you very much,
David