Config help would be appreciated

[JulienLariviere]

Hi,

I am having difficulty setting up this program to use my GPU (nvidia RTX 2070). I am relatively new to configuring nvidia drivers and cuda versions in general and I attempted to do so in Ubuntu 20.04. For your information, I am running the brave program within a Python3.9 venv (not Anaconda).

I tried 2 approaches:

1. On Windows host, I created a UbuntuServer20.04 VM in VirtualBox. I did not go very far with this, because stumbled upon a post suggesting that I cannot use my host's GPU with full functionality through VirtualBox VM (only an emulated version with cuda cores inaccessible).

2. Dual-booted my machine with Ubuntu Desktop 20.04. I was able to install Cuda 10.1 (as per noted is needed for jaxlib in the brave documentation). To install cuda, I followed steps on this website : https://medium.com/@stephengregory_69986/installing-cuda-10-1-on-ubuntu-20-04-e562a5e724a0

When on Linux host, I have no issue displaying my GPU drivers and cuda version with terminal commands, however, there seems to be a default path I forgot to set somewhere so that the brave code knows where to look for my cuda binaries.

When I run the long command (python -m brave.evaluate_video_embeddings) with "--batch_size 8", I see many warnings regarding "cuda.so", then my CPU fan makes a lot of noise, but I do not see any print statements. Eventually "Killed" is printed on the terminal and the program stops.

When I run the long command (python -m brave.evaluate_video_embeddings) with "--batch_size 1", I see the same warnings regarding "cuda.so" and "xla_bridge.py:234] No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)".

However, I get to see the following "completed embedding" progressing, which is nice!

I stopped the program because I had other things to do and clearly have no idea of how long this process could take. This said, this is the most progress I made although the warnings suggest that the code exclusively used my CPU (not GPU).

My questions:

1. Is a cuda enabled GPU necessary to run this program or does it just make it run faster?

2. Is there something obviously wrong with my GPU configuration attempts that I do not see? Please recommend a different approach or point me to relevant documentation.

3. How long does the "brave.evaluate_video_embeddings --batch_size 1" command take to execute approximately? I have a i7 9700k + RTX 2070 and everything stored on SSD.

Thank you very much for your time. I am excited to get this code to run.

[JulienLariviere]

Hi DeepMind team,

I have been trying to replicate this experiment in vain. I think the issue is related to my configuration and I would like some hints as to where to look. I am using Ubuntu 20.04 with kernel 5.13.0, nvidia driver 510, CUDA toolkit 11.6, cuDNN 8.3. My hardware is i7 9700k, RTX 2070 and SSD.

Here is my current pip freeze output.

absl-py==0.12.0 astunparse==1.6.3 attrs==21.2.0 -e git+https://github.com/deepmind/brave.git@0ae20d9afcf6b1fa4d31d70c906d711901b56e9c#egg=brave cached-property==1.5.2 cachetools==4.2.4 certifi==2021.5.30 charset-normalizer==2.0.11 chex==0.0.8 contextlib2==21.6.0 dill==0.3.4 dm-haiku==0.0.4 dm-tree==0.1.6 ffmpeg-python==0.2.0 flatbuffers==1.12 future==0.18.2 gast==0.4.0 google-auth==1.34.0 google-auth-oauthlib==0.4.5 google-pasta==0.2.0 googleapis-common-protos==1.53.0 grpcio==1.34.1 h5py==3.1.0 idna==3.3 importlib-metadata==4.6.4 importlib-resources==5.2.3 iniconfig==1.1.1 jax==0.2.17 jaxlib @ https://storage.googleapis.com/jax-releases/cuda110/jaxlib-0.1.69+cuda110-cp39-none-manylinux2010_x86_64.whl jaxline==0.0.3 joblib==1.0.1 keras-nightly==2.5.0.dev2021032900 Keras-Preprocessing==1.1.2 Markdown==3.3.6 ml-collections==0.1.0 mock==4.0.3 numpy==1.19.5 oauthlib==3.1.1 opt-einsum==3.3.0 optax==0.0.9 packaging==21.3 pluggy==0.13.1 promise==2.3 protobuf==3.17.3 py==1.10.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pyparsing==2.4.7 pytest==6.2.5 PyYAML==5.4.1 requests==2.26.0 requests-oauthlib==1.3.1 rsa==4.7.2 scikit-learn==0.24.2 scipy==1.5.4 six==1.15.0 sklearn==0.0 tabulate==0.8.9 tensorboard==2.5.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.5.3 tensorflow-datasets==4.4.0 tensorflow-estimator==2.5.0 tensorflow-metadata==1.2.0 termcolor==1.1.0 threadpoolctl==2.2.0 toml==0.10.2 toolz==0.11.1 tqdm==4.62.3 typing-extensions==3.7.4.3 urllib3==1.26.8 Werkzeug==2.0.3 wrapt==1.12.1 zipp==3.5.1

Tensorflow sees my GPU and cuDNN:

Jax sees my GPU with currently installed cuda11 wheel:

'pytest brave' fails early on:

evaluate_video_embeddings crashes with OOM error:

With the default config (in clean environment after pip install git+https://github.com/deepmind/brave.git), I get the following:

'pytest brave' passes with a few warnings:

Jax does not see my GPU with default jaxlib==0.1.69 wheel:

evaluate_video_embeddings silently stops after ~10 minutes with "Killed" output:

I tried using the jaxlib==0.1.69 wheel for CUDA 10.1 to no avail. pytest brave fails with that wheel installed. Also, I tried various other configuration combinations including: kernel 5.4.0/5.8.0, driver 418/450/495, CUDA toolkit 10.1/10.2/11.2, cuDNN 7.6/8.05/8.1. I tried in both Python3.9 venv and Anaconda environments.

I am looking for other config combos to try, a list of hardware/software prerequisites and/or any general hints to troubleshoot these issues.

Thank you for your time.

[JulienLariviere]

I also tried --batch_size 1 with thee same errors.