possible to add --nv option when running tensorflow GPU container directly?

[LinuxPersonEC]

One of the cool things I see is that one can run commands from inside a container without having to run the singularity (or apptainer) command. For Tensorflow with GPU support, to get to the GPUs I have to run the full cmd:

singulairty exec --nv /apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu-sha256.sif python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
[..]
This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:4', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:5', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:6', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:7', device_type='GPU')]

Ideally, I'd like to run any of the tensorflow-* cmds directly:

 tensorflow-
tensorflow-container          tensorflow-inspect-deffile    tensorflow-run
tensorflow-exec               tensorflow-inspect-runscript  tensorflow-shell

However the GPU's are not seen by TF this way. Running python then

import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

results in:

>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
2025-01-14 13:47:50.718096: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (34)
2025-01-14 13:47:50.718152: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:137] retrieving CUDA diagnostic information for host: ax08.rc.zi.columbia.edu
2025-01-14 13:47:50.718159: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:144] hostname: ax08.rc.zi.columbia.edu
2025-01-14 13:47:50.718252: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:168] libcuda reported version is: NOT_FOUND: was unable to find libcuda.so DSO loaded into this program
2025-01-14 13:47:50.718287: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:172] kernel reported version is: 550.120.0

[vsoch]

You'd want to add it to singularity options in the container.yaml you install from, if I remember correctly.

[LinuxPersonEC]

What I'd do is grab the command and test different other options (e.g., fakeroot or similar). I haven't tested this specific case so I'm not sure what to suggest.

Would that be by editing the module file?

[vsoch]

I would just manually run the command to test it out, you don't need a module to do it for you. I think this debugging is beyond shpc.

[vsoch]

The module file can just show you what the command might look like to hit the right container, entrypoint, etc.

[LinuxPersonEC]

I would just manually run the command to test it out, you don't need a module to do it for you. I think this debugging is beyond shpc.

Do you mean to run the full apptainer cmd, e.g., apptainer run --nv /apps/Miniforge/lib/python3.12/site-packages/containers/tensorflow/tensorflow/latest-gpu/tensorflow-tensorflow-latest-gpu.sif python? I'ver tried with --fakeroot and --userns to no avail. I would think any SHPC users would want access to GPUs.

[vsoch]

Yes that is what you would start with to debug. SHPC is just generating modules to make those commands - debugging why your GPU is not working with that apptainer command is out of scope for what we will help with here. If you find a solution that works and it's a flag, that can be added to the singularity options. If it's something more system level that we can expose as an option, that could be a new feature. But the issue / bug you are reporting is not related to shpc.

[LinuxPersonEC]

Here are differences with the --debug option:

First I also set always use nv = yes
VERBOSE [U=1822857372,P=1254486]SetGPUConfig() 'always use nv = yes' found in apptainer.conf

differences as root:

DEBUG   [U=0,P=1257265]    handleRemoteConf()            Ensuring file permission of 0600 on /root/.apptainer/remote.yaml
DEBUG   [U=0,P=1257265]    SetContainerEnv()             Forwarding SUDO_GID environment variable
DEBUG   [U=0,P=1257265]    InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=1257265]    SetuidMountAllowed()          Kernel squashfs mount allowed because running as root
DEBUG   [U=0,P=1257279]    InitImageDrivers()            Skipping installing fuseapps image driver because running as root
DEBUG   [U=0,P=1257279]    prepareRootCaps()             Root full capabilities

DEBUG   [U=0,P=1257286]    func1()                       starter was not relocated from /usr/libexec
DEBUG   [U=0,P=1257279]    PrepareConfig()               image driver is 
VERBOSE [U=0,P=1257265]    wait_child()                  stage 1 exited with status 0
DEBUG   [U=0,P=1257265]    cleanup_fd()                  Close file descriptor 4
DEBUG   [U=0,P=1257265]    cleanup_fd()                  Close file descriptor 5
DEBUG   [U=0,P=1257265]    cleanup_fd()                  Close file descriptor 6
DEBUG   [U=0,P=1257265]    mountImage()                  Mounting loop device /dev/loop0 to /var/lib/apptainer/mnt/session/rootfs of type squashfs
DEBUG   [U=0,P=1257265]    createCwdDir()                Using /root as current working directory

Differences as non root

DEBUG   [U=1822857372,P=1254486]setCgroups()                  Recording rootless XDG_RUNTIME_DIR / DBUS_SESSION_BUS_ADDRESS
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding XDG_SESSION_TYPE environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding CONDA_PREFIX environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding XDG_SESSION_CLASS environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding _CE_CONDA environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding LMOD_FAMILY_CONDA_VERSION environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding XDG_SESSION_ID environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding LMOD_FAMILY_CONDA environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding CONDA_PYTHON_EXE environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding XDG_RUNTIME_DIR environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding DBUS_SESSION_BUS_ADDRESS environment variable
DEBUG   [U=1822857372,P=1254486]SetContainerEnv()             Forwarding SSH_TTY environment variable
DEBUG   [U=1822857372,P=1254486]SetuidMountAllowed()          Kernel squashfs mount not allowed because ECL not activated
DEBUG   [U=1822857372,P=1254486]findOnPath()                  Found "squashfuse_ll" at "/usr/libexec/apptainer/bin/squashfuse_ll"
DEBUG   [U=1822857372,P=1254486]findOnPath()                  Found "fuse2fs" at "/usr/libexec/apptainer/bin/fuse2fs"
DEBUG   [U=1822857372,P=1254486]findOnPath()                  Found "fuse-overlayfs" at "/usr/libexec/apptainer/bin/fuse-overlayfs"
DEBUG   [U=1822857372,P=1254486]findOnPath()                  Found "gocryptfs" at "/usr/libexec/apptainer/bin/gocryptfs"
DEBUG   [U=1822857372,P=1254486]InitImageDrivers()            squashfuse_ll supports -o uid
DEBUG   [U=1822857372,P=1254486]InitImageDrivers()            Setting ImageDriver to fuseapps
VERBOSE [U=0,P=1254486]    is_suid()                     Check if we are running as setuid: 1
VERBOSE [U=0,P=1254486]    priv_drop()                   Drop root privileges
VERBOSE [U=1822857372,P=1254486]  priv_escalate()               Get root privileges
VERBOSE [U=0,P=1254486]    priv_escalate()               Change filesystem uid to 1234
VERBOSE [U=0,P=1254486]    init()                        Spawn master process
VERBOSE [U=0,P=1254508]    create_namespace()            Create mount namespace
VERBOSE [U=0,P=1254486]    enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=1254486]    enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=1254508]    create_namespace()            Create mount namespace
DEBUG   [U=0,P=1254486]    set_master_privileges()       Set master privileges
DEBUG   [U=0,P=1254486]    apply_privileges()            Effective capabilities:   0x00000000000000c0
DEBUG   [U=0,P=1254486]    apply_privileges()            Permitted capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=1254486]    apply_privileges()            Bounding capabilities:    0x000001ffffffffff
DEBUG   [U=0,P=1254486]    apply_privileges()            Inheritable capabilities: 0x000001ffffffffff
DEBUG   [U=0,P=1254486]    apply_privileges()            Ambient capabilities:     0x0000000000000000
DEBUG   [U=0,P=1254486]    apply_privileges()            Set user ID to 1234
DEBUG   [U=0,P=1254509]    set_rpc_privileges()          Set RPC privileges
DEBUG   [U=0,P=1254509]    apply_privileges()            Effective capabilities:   0x0000000000200000
DEBUG   [U=0,P=1254509]    apply_privileges()            Permitted capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=1254509]    apply_privileges()            Bounding capabilities:    0x0000000008204000
DEBUG   [U=0,P=1254509]    apply_privileges()            Inheritable capabilities: 0x0000000000000000
DEBUG   [U=0,P=1254509]    apply_privileges()            Ambient capabilities:     0x0000000000000000
DEBUG   [U=0,P=1254509]    apply_privileges()            Set user ID to 1234
DEBUG   [U=1822857372,P=1254509]  set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1822857372,P=1254509]  init()                        Spawn RPC server
DEBUG   [U=1822857372,P=1254486]func1()                       executablePath is /usr/libexec/apptainer/bin/starter-suid
DEBUG   [U=1822857372,P=1254486]func1()                       Base is starter-suid which never relocates
DEBUG   [U=1822857372,P=1254486]func1()                       Install prefix is /usr
DEBUG   [U=1822857372,P=1254501]PrepareConfig()               image driver is fuseapps
VERBOSE [U=1822857372,P=1254486]  wait_child()                  stage 1 exited with status 0
DEBUG   [U=1822857372,P=1254486]  cleanup_fd()                  Close file descriptor 3
DEBUG   [U=1822857372,P=1254486]  cleanup_fd()                  Close file descriptor 4
DEBUG   [U=1822857372,P=1254486]  cleanup_fd()                  Close file descriptor 5
DEBUG   [U=1822857372,P=1254486]  cleanup_fd()                  Close file descriptor 6
DEBUG   [U=1822857372,P=1254486]  cleanup_fd()                  Close file descriptor 7

DEBUG   [U=1822857372,P=1254486]func2()                       Starting image driver fuseapps
DEBUG   [U=1822857372,P=1254486]mountImageDriver()            Do FUSE mount for image driver with fd=6,rootmode=40000,user_id=1234,group_id=4567,allow_other /var/lib/apptainer/mnt/session/rootfs
DEBUG   [U=1822857372,P=1254486]mountImageDriver()            Do image mount with source /proc/self/fd/10 and target /dev/fd/3
DEBUG   [U=1822857372,P=1254486]Mount()                       Executing /usr/libexec/apptainer/bin/squashfuse_ll -f -o allow_other,ro,uid=1234,gid=4567,offset=40960 /proc/self/fd/3 /dev/fd/4

Could fuseapps be a part of the problem?

[vsoch]

I think you probably want to open an issue here: https://github.com/apptainer/apptainer/issues

[LinuxPersonEC]

I think you probably want to open an issue here: https://github.com/apptainer/apptainer/issues

Thanks if there's anything you think I can add to the bug report LMK. Perhaps it is something simple as a flag.

[LinuxPersonEC]

So on RHEL 8 with Anaconda Python 3.11.8 and shpc 0.1.30, running:
ml use /apps/modulefiles/tensorflow/tensorflow/latest-gpu

then the module file shows as just module.

Running module load module get this error:

Lmod has detected the following error:  Unable to load module because of error when evaluating modulefile:
     /apps/modulefiles/tensorflow/tensorflow/latest-gpu/module.lua: [string "-- Lmod Module..."]:97: attempt to concatenate a boolean value
     Please check the modulefile and especially if there is a the line number specified in the above message 
While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    module           /apps/modulefiles/tensorflow/tensorflow/latest-gpu/module.lua

Line 97:
97 whatis("Version : " .. myModuleVersion())

[vsoch]

What if we take it at face value and assume (for some reason) this module version is a string, and then:

whatis("Version     : " .. tostring(myModuleVersion()))

Can you try that and report back?

[LinuxPersonEC]

@muffato @vsoch any thoughts on the RHEL 8 issue I mentioned?