Help debugging ut_read_xml() failed error

[mankoff]

Hello,

I've created a Docker image. It works for me in Docker. It does not work on our HPC system. I installed the latest Singularity in a VM to test, and it does work there. This suggests this is a problem with the HPC system, but I'm having a hard time figuring out why. I hope someone here can help. I'm fairly new to docker and very new to Singularity. Given the issue described below, can anyone suggest steps to help debug, or anything I can pass on to the HPC admins to help them debug this?

I've seen a bunch of issues (moved to discussion) explaining how Singularity is different from Docker. I do not yet see how that could be the cause of this issue.

This re-creates the issue for me.

singularity pull docker://mankoff/pism:latest # From https://github.com/pism/containers/tree/main/basic
singularity exec \
            --bind $(pwd)/:/gpfsm \
            docker://mankoff/pism:latest \
            bash -c "cd; mpiexec -n 4 pism -test G -y 200"

No error on Ubuntu 22.04 clean VM set up with singularity 4.3.0, but error on our HPC with the same version:

INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
PISM ERROR: ut_read_xml() failed
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[57959,1],1]
  Exit code:    1
--------------------------------------------------------------------------
INFO:    Cleaning up image...

[mankoff]

Exploring more I notice that env in Docker reports almost nothing. Only about 5 environment variables are set. env in the Singularity container reports ~100 settings. It looks like my external environment is in Singularity, which makes sense as I've read (and see) that my home folder is auto-mounted, etc.

Messing around with LD_LIBRARY_PATH does not seem to change anything though...

[dtrudg]

You might want to take a look in the docs here...
https://docs.sylabs.io/guides/4.3/user-guide/singularity_and_docker.html#differences-and-limitations-vs-docker

And here...
https://docs.sylabs.io/guides/4.3/user-guide/singularity_and_docker.html#docker-like-compat-flag

We have quite a lot of explanation of how Singularity's defaults differ from Docker... and how to use flags to influence this (such as --compat ... or individual options that infers).

[mankoff]

Thank you. --compat does fix it.