change location of PTX contents possible?

[HansN87]

Hi,

I am trying to run JAX (0.5.2) using a singularity container on a GPU compute node of a compute cluster.
JAX was installed using pip within a virtual env: pip install -U "jax[cuda12]"

The GPUs are older models: GeForce GTX 1080
(py3_jax_env_cuda12.1_singularity) Apptainer> nvidia-smi -L GPU 0: NVIDIA GeForce GTX 1080

The GPU is visible to JAX:

>>> import jax
>>>
>>> jax.devices()
[CudaDevice(id=0)]
>>> import jax.numpy as jnp

However, simple operations lead to Permission issues:

jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

[...]

File ".../venv/py3_jax_env_cuda12.1_singularity-5mUYqkWx/lib/python3.10/site-packages/jax/_src/dispatch.py", line 90, in apply_primitive
    outs = fun(*args)
jaxlib.xla_extension.XlaRuntimeError: PERMISSION_DENIED: /scratch/tempfile-gtx-7; Read-only file system
        Unable to write PTX contents to: /scratch/tempfile-gtx-7-1bddc80aa4cd3eb0-12663-62fc9411161af

Is there any way to change the default output/temp directory for these PTX contents via XLA flags/env variables or something similar?

I tried using XLA_FLAGS=--xla_dump_to=some_folder_I_have_permission_to_write_to, but the problem remains.

Any pointers would be appreciated! Thank you.