Shouldn't we remove this since we are defining

--gpu-bind list:0.0:0.1:1.0:1.1:2.0:2.1:3.0:3.1:4.0:4.1:5.0:5.1

@abagusetty what's the equivalent of OpenMPI's OMPI_COMM_WORLD_LOCAL_RANK on Aurora-mpich?
Kokkos raises

`Warning: unable to detect local MPI rank. Falling back to the first GPU available for execution. Raised by Kokkos::initialize()`

if $MPI_LOCALRANKID env-var is undefined. $PALS_LOCAL_RANKID appears to be empty also.

Shouldn't we remove this since we are defining

--gpu-bind list:0.0:0.1:1.0:1.1:2.0:2.1:3.0:3.1:4.0:4.1:5.0:5.1

I am assuming you are asking for the last line. Yes, for <arg name="gpu_maps">$ENV{GPU_TILE_COMPACT}</arg> this can be removed. Since gpu-bind takes care of binding tiles with MPI-processes and wouldnt need a script.

@amametjanov Aurora MPICH doesnt have an equivalent. But PALS should work the same: PALS_LOCAL_RANKID. It depends on where you would be using this var. Notsure why is this empty. I believe this warnings were fixed a while back.

As long as PALS_LOCAL_RANKID is embedded after mpiexec launch command, PALS should be defined. For instance: mpiexec ... $PALS_LOCAL_RANK $EXE...

Ok thanks. Straight removal of gpu_tile_compact.sh script was raising those warnings. I'll push a commit to appropriately export correct $MPI_LOCALRANKID.

Is there a reason to prefer the gpu-bind argument over the script?

-gpu-bind is a more of a official mpich option that allows topology aware bindings internally over the script. The script was a temp. WA just since we didnt have a GPU-binding mechanism for Aurora in the earlier days

Passing mpiexec ... --genv MPI_LOCALRANKID=${PALS_LOCAL_RANKID} ... isn't helping: empty MPI_LOCALRANKID.
I added a Kokkos mod adding PALS_LOCAL_RANKID to the list of recognized env-vars at E3SM-Project/EKAT#372 . When that PR makes it into E3SM master, I can remove the call to gpu_tile_compact.sh in a separate PR. If that's okay, then maybe this PR can go without that mod.

Occasionally, job submissions' stdout returns more output beyond a job's id: e.g.

ERROR: Couldn't match jobid_pattern '^(\d+)' within submit output:
 'auth: error returned: 15007
auth: Failed to receive auth token
No Permission.
qstat: cannot connect to server aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov (errno=15007)
5435693.aurora-pbs-0001.hostmgmt.cm.aurora.alcf.anl.gov'

This regex update is to extract job-id from such longer strings to let CIME continue with its job stages.