Tensorflow2-gpu cannot be employed as the backend

[ZhouMengdi1994]

Dear developers,
I really need your help. I installed the DeepXDE 1.9.0, TensorFlow-gpu 2.10.0 and TensorFlow Probability 0.18.0. When the tensorflow.compat.v1 is used as the backend, everything is fine. But when I modify the "config.json" as "{"backend": "tensorflow"} and run the example “ode_system.py”, the following errors will show up. I don't know how to handle this issue, so I sincerely seek your help.

Python 3.8.16 | packaged by conda-forge | (default, Feb 1 2023, 15:53:35) [MSC v.1929 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 8.12.2 -- An enhanced Interactive Python.

runfile('D:/code/tensorflow学习/DeepXDE案例.py', wdir='D:/code/tensorflow学习')
Using backend: tensorflow
Other supported backends: tensorflow.compat.v1, pytorch, jax, paddle.
paddle supports more examples now and is recommended.
Enable just-in-time compilation with XLA.

Compiling model...
'compile' took 0.000341 s

Training model...

D:\environment\AIfluid\lib\site-packages\keras\initializers\initializers_v2.py:120: UserWarning: The initializer GlorotUniform is unseeded and being called multiple times, which will return identical values each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initalizer instance more than once.
warnings.warn(
Step Train loss Test loss Test metric
0 [3.17e-01, 2.06e-01, 5.56e-01, 1.06e+00] [3.24e-01, 2.11e-01, 5.56e-01, 1.06e+00] [1.25e+00]
Traceback (most recent call last):

File D:\environment\AIfluid\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)

File d:\code\tensorflow学习\deepxde案例.py:68
losshistory, train_state = model.train(iterations=10000)

File D:\environment\AIfluid\lib\site-packages\deepxde\utils\internal.py:22 in wrapper
result = f(*args, **kwargs)

File D:\environment\AIfluid\lib\site-packages\deepxde\model.py:632 in train
self._train_sgd(iterations, display_every)

File D:\environment\AIfluid\lib\site-packages\deepxde\model.py:650 in _train_sgd
self._train_step(

File D:\environment\AIfluid\lib\site-packages\deepxde\model.py:541 in _train_step
self.train_step(inputs, targets, auxiliary_vars)

File D:\environment\AIfluid\lib\site-packages\tensorflow\python\util\traceback_utils.py:153 in error_handler
raise e.with_traceback(filtered_tb) from None

File D:\environment\AIfluid\lib\site-packages\tensorflow\python\eager\execute.py:54 in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

InternalError: libdevice not found at ./libdevice.10.bc [Op:__inference_train_step_1114]

2023-05-24 20:55:37.934421: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2023-05-24 20:55:39.267746: W tensorflow/compiler/xla/service/gpu/nvptx_helper.cc:56] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice.
Searched for CUDA in the following directories:
./cuda_sdk_lib
C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2
/usr/local/cuda
.
You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions. For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work.
2023-05-24 20:55:40.921130: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc
2023-05-24 20:55:40.923168: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at xla_ops.cc:296 : INTERNAL: libdevice not found at ./libdevice.10.bc

[lululxvi]

File D:\environment\AIfluid\lib\site-packages\tensorflow\python\eager\execute.py:54 in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

InternalError: libdevice not found at ./libdevice.10.bc [Op:__inference_train_step_1114]

2023-05-24 20:55:37.934421: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2023-05-24 20:55:39.267746: W tensorflow/compiler/xla/service/gpu/nvptx_helper.cc:56] Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice. This may result in compilation or runtime failures, if the program we try to run uses routines from libdevice. Searched for CUDA in the following directories: ./cuda_sdk_lib C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.2 /usr/local/cuda . You can choose the search directory by setting xla_gpu_cuda_data_dir in HloModule's DebugOptions. For most apps, setting the environment variable XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda will work. 2023-05-24 20:55:40.921130: W tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/gpu_backend_lib.cc:327] libdevice is required by this HLO module but was not found at ./libdevice.10.bc 2023-05-24 20:55:40.923168: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at xla_ops.cc:296 : INTERNAL: libdevice not found at ./libdevice.10.bc

See the error above. Tensorflow is not installed correctly. Or you can disable XLA.

[ZhouMengdi1994]

Dear Lu lu:
Thank you for your reply.
I disable XLA by add 'tf.tpu.XLAOptions(use_spmd_for_xla_partitioning=False, enable_xla_dynamic_padder=False) ' in the program but it doesn't work.
Then I install the tensorflow-CPU and run the program again. It shows that the problem has been solved, 'deeponet_POD.py' is running successfully when the backend is tensorflow.
So I guess, maybe Tensorflow-GPU and DeepXDE are not compatible?
Thank you again for your enthusiastic help.

[lululxvi]

https://deepxde.readthedocs.io/en/stable/modules/deepxde.html#deepxde.config.disable_xla_jit

[ZhouMengdi1994]

Thank you so much！Best wishes.