Tensorflow GPU tests failing for TF 2.16 and 2.17

[smuzaffar]

@cms-sw/ml-l2 , we are trying to update Tensorlfow for cmssw. We have TF 2.16 in CMSSW_14_2_TF_X_2024-09-24-1100 IB and TF 2.17 in this weeks IB (will be available soon). We noticed that GPU tests are failing [a] for these newer versions TF. Can you please look in to these failures ?

[a] run on lxplus-gpu node

> ssh lxplus-gpu
> cd /tmp/$(whoami)
> cmssw-el8 --nv
Singularity> scram p CMSSW_14_2_TF_X_2024-09-24-1100
Singularity> cmsenv
Singularity> git cms-addpkg PhysicsTools/TensorFlow
Singularity> scram build -j 8
Singularity> cmsenv
Singularity> ./test/el8_amd64_gcc12/testTFHelloWorldCUDA
Running .
46.0

CUDA service enabled: 1
Testing CUDA backend
E

##Failure Location unknown## : Error
Test name: testHelloWorldCUDA::test
uncaught exception of type std::exception (or derived).
- An exception of category 'UnavailableAccelerator' occurred while
   [0] Calling tensorflow::setBackend()
Exception Message:
Cuda backend requested, NVIDIA GPU visible to cmssw, but not visible to TensorFlow in the job

Failures !!!
Run: 1   Failure total: 1   Failures: 0   Errors: 1

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smuzaffar.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

assign ml

[cmsbuild]

New categories assigned: ml

@valsdav,@y19y19 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[smuzaffar]

maybe it is related to this issue tensorflow/tensorflow#63362 ?

[valsdav]

Having a look thanks!

[smuzaffar]

by the way, running

> python3 -c "import tensorflow as tf;print(tf.config.list_physical_devices('GPU'))"
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02856/el8_amd64_gcc12/external/py3-numpy/1.26.4-3fdea04e550a3b32f8885b1844fc997a/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

shows that TF does see the GPU device 0

[smuzaffar]

May be https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/TensorFlow/src/TensorFlow.cc#L48 should be

if (_options.config.device_count().find("GPU") == _options.config.device_count().end()){

[valsdav]

Testing it and I can open a PR with the fix 👍

[valsdav]

options.config.device_count seems indeed empty. I'm checking if the internal API has changed.