Adiabatic simulations not progressing

[ecritell]

Hello,

we are trying to run adiabatic simulations using CGYRO, here is an example input file:

AE_FLAG = 1
N_ENERGY = 8
E_MAX = 8
N_XI = 32
N_THETA = 32
N_RADIAL = 64
N_FIELD = 1
DELTA_T = 0.01
DELTA_T_METHOD = 1
AMP = 0.01
PRINT_STEP = 200
MAX_TIME = 1000.0
FREQ_TOL = 0.01
KY = 0.030000000000000002
EQUILIBRIUM_MODEL = 2
RMIN = 0.6367692524514235
RMAJ = 1.9625016348394586
KAPPA = 2.5600490233845186
S_KAPPA = 0.015159986683600855
DELTA = 0.28311883245633646
S_DELTA = 0.2801453728892607
SHIFT = -0.3991883206232521
Q = 4.205681423
S = 1.108070285
BETA_STAR_SCALE = 1.0
COLLISION_MODEL = 4
Z_EFF = 1.0
Z_EFF_METHOD = 1
N_SPECIES = 1
DENS_AE = 1.0
TEMP_AE = 1.0
MASS_AE = 0.000272308510742333
DLNNDR_AE = 0.947190717
DLNTDR_AE = 0.359540884
Z_1 = 1
DENS_1 = 1.0
TEMP_1 = 1.0328837640672761
MASS_1 = 1.0
DLNNDR_1 = 0.947190717
DLNTDR_1 = 0.680878947
NONLINEAR_FLAG = 1
BOX_SIZE = 6.96221
N_TOROIDAL = 64
TOROIDALS_PER_PROC = 16
THETA_PLOT = 32
PX0 = 0.0
FIELD_PRINT_FLAG = 1
MOMENT_PRINT_FLAG = 1
ZMAG = 0.0
DZMAG = 0.0
IPCCW = -1.0
BTCCW = -1.0
MACH = 0.0
GAMMA_P = 0.0
GAMMA_E = 0.0

However the job seems to progress very slowly and only completes around 80 steps (ie the latest restart file we have is bin.cgyro.restart 80.00) out of the expected 1000 in about 12 hours. We were unable to spot errors or anything unusual in the logs that would explain the cause of the simulation not progressing, as an example this is the out.cgyro.info output file

INFO: (CGYRO) Velocity order 1
INFO: (CGYRO) MPI rank alignment 2
INFO: (CGYRO) Multiple toroidal harmonics
INFO: (CGYRO) Ion direction: omega < 0
INFO: (CGYRO) ExB shear: OFF
INFO: (CGYRO) RESOLUTION WARNING -- n_radial not a multiple of box_size.
INFO: (CGYRO) Time integrator: Cash-Karp 6:5(4) [adaptive]
INFO: (CGYRO) Using adiabatic electrons
INFO: (CGYRO) Profile model: local input (input.cgyro)
INFO: (CGYRO) Equilibrium: Miller Extended Harmonic (MXH)
INFO: (CGYRO) Electrostatic fluctuations (Phi)
INFO: (CGYRO) Rotation terms: O(mach) only (traditional GYRO)
INFO: (CGYRO) Collision model: Sugama
INFO: (CGYRO) Collision terms: L D Rm Re kp ions field
INFO: (CGYRO)                  x x x  x  x   x     x

 n_theta | n_species | n_energy | n_xi
    32         1          8        32

 nc_loc | nv_loc | nsplit | n_jtheta | n_MPI | n_OMP
   256       32      256          8       32    1

           n    Delta      Max     L/rho    n_fft
 kx*rho:  64    0.035    1.079    180.49       96  2(5)3(1)
 ky*rho:  64    0.030    1.890    209.44      192  2(6)3(1)

  D-theta:  6    D-radial:  6    D-alpha:  6   [dissipation order]
 up_theta: 1.0  up_radial: 1.0  up_alpha: 0.0  [dissipation strength]
 C(theta): 0.0

  r/a  0.63677   R/a  1.96250   q  4.20568   zmag  0.00000    kappa  2.56005
               shift -0.39919   s  1.10807  dzmag  0.00000  s_kappa  0.01516

  c1  0.00000   s_c1  0.00000   delta  0.28312  s_delta  0.28015
  c2  0.00000   s_c2  0.00000    zeta  0.00000   s_zeta  0.00000

 gamma_e:  0.000E+00  gamma_p:  0.000E+00    mach:  0.000E+00  [rho/a]:  4.542E-03
   betae:  0.000E+00   beta_*:  0.000E+00  lamb_*:  0.000E+00  [z_eff]:  1.000E+00

 i  z  n/n_norm   T/T_norm   m/m_norm     a/Ln       a/Lt       nu
 1  1  1.000E+00  1.033E+00  1.000E+00  9.472E-01  6.809E-01  1.572E-03

INFO: (CGYRO) GPU-aware code triggered.
INFO: (CGYRO) NL using FFT batching of   256,  128 and   128

Interestingly we have a corresponding simulation for kinetic electrons, the main differences in the input files being no AE_FLAG, N_FIELD = 3, and N_SPECIES = 2, where we then consequently defines MASS, TEMP, DENS... for species 1 and 2, and add BETAE. These simulations run perfectly fine and complete all the 1000 steps in a few hours.

Any help or suggestion much appreciated, thanks in advance!

[bellie]

Can you please tell us the githash for the version of CGYRO you are using, how many nodes you are running on, and a few lines of out.cgyro.timing?

FYI: BOX_SIZE must be an integer. [not related to your issue but it does cause the file to crash on some systems.]

[ecritell]

Hi Emily, thanks for your reply!

The git hash is 0b453b8, on top we added some very minimal changes to the file dumps (we are unable to push those due to lacking write permissions to the repo), but really those should not affect any code logic.
We are running on 4 nodes on ISAMBARD, and below are some lines of out.cgyro.timing:

Setup time input str_init nl_init coll_init io_init 0.002 0.456 1.044 10.856 1.033 Run time str str_mem str_comm nl nl_mem nl_comm field field_com shear coll coll_mem coll_comm io TOTAL 0.693 0.012 0.567 1.159 0.145 0.571 0.621 0.506 0.000 0.160 0.027 0.533 0.164 5.230 0.475 0.012 0.551 1.159 0.144 0.234 0.629 0.481 0.000 0.167 0.007 0.526 0.180 4.638 0.519 0.012 1.106 1.183 0.164 0.907 0.697 0.701 0.000 0.175 0.007 0.543 0.161 6.250 0.475 0.012 0.547 1.156 0.145 0.233 0.638 0.588 0.000 0.164 0.007 0.523 0.441 5.002 0.476 0.012 0.548 1.161 0.144 0.222 0.642 0.500 0.000 0.161 0.007 0.524 0.170 4.640 0.477 0.012 0.549 1.159 0.143 0.240 0.642 0.497 0.000 0.162 0.007 0.525 0.175 4.661 0.476 0.012 0.547 1.160 0.143 0.226 0.642 0.488 0.000 0.162 0.007 0.528 0.169 4.634 0.476 0.012 0.532 1.159 0.144 0.228 0.637 0.498 0.000 0.165 0.007 0.527 0.432 4.891 0.477 0.012 0.545 1.160 0.143 0.213 0.639 0.498 0.000 0.165 0.007 0.529 0.160 4.620 0.477 0.012 0.550 1.164 0.144 0.187 0.641 0.514 0.000 0.165 0.007 0.525 0.506 4.965 0.476 0.012 0.548 1.157 0.143 0.206 0.642 0.510 0.000 0.162 0.007 0.529 0.157 4.624 0.477 0.012 0.521 1.157 0.143 0.208 0.646 0.532 0.000 0.157 0.007 0.525 0.151 4.610 0.474 0.012 0.543 1.158 0.143 0.224 0.642 0.509 0.000 0.158 0.007 0.527 0.165 4.635 0.473 0.012 0.550 1.157 0.143 0.230 0.636 0.502 0.000 0.161 0.007 0.526 0.157 4.627 0.472 0.012 0.566 1.160 0.143 0.207 0.637 0.510 0.000 0.159 0.007 0.526 1.212 5.683 0.475 0.012 0.552 1.158 0.143 0.227 0.641 0.488 0.000 0.162 0.007 0.528 0.152 4.619 0.473 0.012 0.564 1.160 0.143 0.271 0.637 0.486 0.000 0.158 0.007 0.536 0.154 4.674 0.472 0.011 0.544 1.154 0.144 0.241 0.632 0.483 0.000 0.164 0.007 0.532 0.569 5.026 0.470 0.011 0.549 1.160 0.143 0.225 0.637 0.496 0.000 0.159 0.007 0.529 0.158 4.617 0.472 0.011 0.545 1.160 0.143 0.211 0.634 0.513 0.000 0.162 0.007 0.529 0.270 4.730 0.472 0.011 0.551 1.158 0.142 0.222 0.635 0.508 0.000 0.161 0.007 0.524 0.156 4.622 0.472 0.011 0.539 1.159 0.144 0.211 0.634 0.515 0.000 0.164 0.007 0.526 0.161 4.617 0.474 0.011 0.546 1.158 0.143 0.224 0.642 0.502 0.000 0.155 0.007 0.523 0.186 4.644 0.471 0.011 0.555 1.164 0.143 0.216 0.631 0.497 0.000 0.166 0.007 0.528 0.157 4.618

Hope this helps!
Cheers, Edoardo

[jcandy]

Hi Edoardo. The resolution is a bit strange for your case. Can you please send us the input.cgyro file for kinetic electrons? We'll fix the resolution and generate the equivalent adiabatic electron case.

[bellie]

I agree with Jeff that the resolutions are strange -- in particular kx_max is very low and likely much large n_radial is required. For this specific case, I am getting about the same timings - 4 sec per print step and nothing unusual is happening. I ran it to t a/cs150 and the transport is 0. What are the equivalent timings for the kinetic electron case -- assuming same node count?

[ecritell]

Thanks for your replies!

Here's our input.cgyro for the kinetic electron case of this simulation:

N_ENERGY = 8
E_MAX = 8
N_XI = 32
N_THETA = 32
N_RADIAL = 64
N_FIELD = 3
DELTA_T = 0.01
DELTA_T_METHOD = 1
AMP = 0.01
PRINT_STEP = 200
MAX_TIME = 1000.0
FREQ_TOL = 0.01
KY = 0.030000000000000002
EQUILIBRIUM_MODEL = 2
RMIN = 0.6367692524514235
RMAJ = 1.9625016348394586
KAPPA = 2.5600490233845186
S_KAPPA = 0.015159986683600855
DELTA = 0.28311883245633646
S_DELTA = 0.2801453728892607
SHIFT = -0.3991883206232521
BETAE_UNIT = 0.01448995
Q = 4.205681423
S = 1.108070285
BETA_STAR_SCALE = 1.0
COLLISION_MODEL = 4
NU_EE = 0.032687963711238464
Z_EFF = 1.0
Z_EFF_METHOD = 1
N_SPECIES = 2
Z_1 = -1
DENS_1 = 1.0
TEMP_1 = 1.0
MASS_1 = 0.000272308510742333
DLNNDR_1 = 0.947190717
DLNTDR_1 = 0.359540884
Z_2 = 1
DENS_2 = 1.0
TEMP_2 = 1.0328837640672761
MASS_2 = 1.0
DLNNDR_2 = 0.947190717
DLNTDR_2 = 0.680878947
NONLINEAR_FLAG = 1
BOX_SIZE = 6.96221
N_TOROIDAL = 64
TOROIDALS_PER_PROC = 16
THETA_PLOT = 32
PX0 = 0.0
FIELD_PRINT_FLAG = 1
MOMENT_PRINT_FLAG = 1
ZMAG = 0.0
DZMAG = 0.0
IPCCW = -1.0
BTCCW = -1.0
MACH = 0.0
GAMMA_P = 0.0
GAMMA_E = 0.0

Here is the timing file output of this one on the same number of nodes:

Setup time input str_init nl_init coll_init io_init 0.001 0.506 0.901 68.698 3.617 Run time str str_mem str_comm nl nl_mem nl_comm field field_com shear coll coll_mem coll_comm io TOTAL 1.745 0.025 3.053 4.050 0.408 3.392 1.139 3.002 0.000 0.631 0.053 0.809 0.594 19.038 1.723 0.027 3.289 4.396 0.441 3.155 1.170 3.390 0.000 0.641 0.017 0.802 0.597 19.791 1.485 0.024 3.055 3.864 0.385 2.618 1.117 2.798 0.000 0.650 0.017 0.796 0.678 17.617 1.281 0.022 2.585 3.355 0.334 2.249 1.105 2.541 0.000 0.626 0.017 0.814 1.782 16.823 1.243 0.021 2.488 3.267 0.329 2.192 1.078 2.573 0.000 0.647 0.018 0.808 0.601 15.378 1.135 0.020 2.286 2.951 0.293 1.965 1.080 2.160 0.000 0.621 0.017 0.813 0.659 14.106 1.007 0.019 2.090 2.635 0.262 1.763 1.066 1.942 0.000 0.612 0.018 0.811 0.602 12.923 0.870 0.018 1.887 2.306 0.229 1.514 1.035 1.697 0.000 0.622 0.017 0.811 0.597 11.689 0.779 0.017 1.711 2.069 0.206 1.355 1.026 1.526 0.000 0.612 0.018 0.826 1.114 11.342 0.772 0.017 1.730 2.074 0.203 1.366 1.018 1.523 0.000 0.617 0.018 0.810 0.761 10.989 0.778 0.016 1.744 2.069 0.207 1.346 0.993 1.527 0.000 0.637 0.018 0.805 0.593 10.817 0.776 0.017 1.729 2.071 0.206 1.356 1.004 1.525 0.000 0.628 0.017 0.808 0.672 10.894 0.781 0.017 1.727 2.065 0.206 1.353 1.002 1.527 0.000 0.628 0.018 0.814 0.599 10.819 0.779 0.017 1.741 2.070 0.205 1.353 0.996 1.523 0.000 0.634 0.018 0.804 0.602 10.825 0.777 0.016 1.727 2.069 0.205 1.362 1.004 1.526 0.000 0.627 0.017 0.809 0.591 10.812 0.782 0.017 1.724 2.069 0.205 1.354 1.004 1.528 0.000 0.626 0.017 0.814 0.583 10.806 0.790 0.017 1.737 2.066 0.205 1.344 0.998 1.521 0.000 0.632 0.017 0.813 0.588 10.807

Cheers,
Edoardo

[ecritell]

Hi @bellie @jcandy , just wondering if you had any update on this ?

Thanks,
Edoardo

[jcandy]

Yes, I have some results I'll send later today. Sorry for the delay.

[jcandy]

Below I attach an example kinetic electron simulation and an adiabatic electron equivalent. Noticing that your cases were linearly stable, I changed parameters somewhat to create linearly unstable (but nonlinearly subcritical) conditions.

input.cgyro.ae.txt
input.cgyro.ke.txt

[jcandy]

The resolution for your cases was probably not optimal. I suggest you start over from the resolutions used in the attached inputs. Generally, you will need N_RADIAL >> N_TOROIDAL. A classic resolution is N_RADIAL=128 and N_TOROIDAL=16. I would also avoid use of TOROIDALS_PER_PROC unless necessary. N_XI=18 is also probably more than enough. Let me know if this helps.

[ecritell]

Thanks @jcandy , will try and let you know!

Edoardo