GPU is not working when running the ResNet18 example

[whcjimmy]

Summary

I am trying to run the ResNet18 inference with a GPU. First, I followed the GPU acceleration guide to set up the running environment, and the checking progress passed successfully.

However, when I actually run the ResNet18 example, I do not see that the program utilizes any GPU resources for FHE computations.

My running command is python3 run_resnet18_fhe.py --use_gpu --run_fhe.

Description

• versions affected: 1.8.0
• python version: 3.8.20
• config (optional: HW, OS): Ubuntu 22.04
• workaround (optional): if you’ve a way to workaround the issue
• proposed fix (optional): if you’ve a way to fix the issue

Step by step procedure someone should follow to trigger the bug:

minimal POC to trigger the bug

print("Minimal POC to reproduce the bug")

[jfrery]

Hi,

Did you wait for the whole execution? The script runs in this order:

• torch evaluation top1 top5
• quantized evaluation top1 top5
• FHE simulation top1 top5
  All the above run on CPU
• actual FHE evaluation (this time on GPU)

[whcjimmy]

Yes, I saw that the program returns “Running FHE execution” and I’ve been waiting for several hours. However, the process is utilizing all CPU resources for the FHE computations.

I’m using a GTX 4090, and the pip packages I installed are shown below. If there is any additional information you need, please let me know, and I will provide it soon.

pip packages

------------------------ -----------
aiohappyeyeballs         2.4.4
aiohttp                  3.10.11
aiosignal                1.3.1
async-timeout            5.0.1
attrs                    24.3.0
brevitas                 0.10.2
certifi                  2024.12.14
charset-normalizer       3.4.1
coloredlogs              15.0.1
concrete-ml              1.8.0
concrete-ml-extensions   0.1.4
concrete-python          2024.12.19
contourpy                1.1.1
cycler                   0.12.1
datasets                 3.1.0
dependencies             2.0.1
dill                     0.3.8
filelock                 3.16.1
flatbuffers              24.12.23
fonttools                4.55.3
frozenlist               1.5.0
fsspec                   2024.9.0
huggingface-hub          0.27.1
humanfriendly            10.0
hummingbird-ml           0.4.11
idna                     3.10
importlib_resources      6.4.5
Jinja2                   3.1.5
joblib                   1.4.2
jsonpickle               4.0.1
kiwisolver               1.4.7
MarkupSafe               2.1.5
matplotlib               3.7.5
mpmath                   1.3.0
multidict                6.1.0
multiprocess             0.70.16
networkx                 3.1
numpy                    1.23.5
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.20.5
nvidia-nvjitlink-cu12    12.6.85
nvidia-nvtx-cu12         12.1.105
onnx                     1.17.0
onnxconverter-common     1.13.0
onnxmltools              1.13.0
onnxoptimizer            0.3.13
onnxruntime              1.18.0
packaging                24.2
pandas                   2.0.3
pillow                   10.4.0
pip                      24.2
propcache                0.2.0
protobuf                 3.20.3
psutil                   6.1.1
pyarrow                  17.0.0
pyparsing                3.1.4
python-dateutil          2.9.0.post0
pytz                     2024.2
PyYAML                   6.0.2
requests                 2.32.3
scikit-learn             1.1.3
scipy                    1.10.1
setuptools               75.3.0
six                      1.17.0
skl2onnx                 1.18.0
skops                    0.5.0
skorch                   0.11.0
sympy                    1.13.3
tabulate                 0.9.0
threadpoolctl            3.5.0
torch                    2.4.1
torchvision              0.19.1
tqdm                     4.67.1
triton                   3.0.0
typing_extensions        4.12.2
tzdata                   2024.2
unfoldNd                 0.2.3
urllib3                  2.2.3
wheel                    0.44.0
xgboost                  1.6.2
xxhash                   3.5.0
yarl                     1.15.2
z3-solver                4.13.0.0
zipp                     3.20.2

[jfrery]

You seem to have the expected libraries. Could you share what's the cuda version on your machine?

If you could provide the full logs from the run until you see the Running FHE execution that would also be useful.

[whcjimmy]

I'm using CUDA 12.6 with Nvidia driver 560.35.03.

Full logs are shown below:

% python3 run_resnet18_fhe.py --use_gpu --run_fhe
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1024/1024 [00:00<00:00, 1801.00it/s]
Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 64/64 [00:00<00:00, 771366.25it/s]
Accuracy of the ResNet18 model on the images: 67.00%
Top-5 Accuracy of the ResNet18 model on the images: 87.00%
Compiling the model with compile_torch_model...
Model compiled successfully.
Quantized Model Accuracy of the FHEResNet18 on the images: 67.00%
Quantized Model Top-5 Accuracy of the FHEResNet18 on the images: 87.00%
FHE simulation Accuracy of the FHEResNet18 on the images: 57.00%
FHE simulation Top-5 Accuracy of the FHEResNet18 on the images: 81.00%
Processing 1 image(s)...

Image 1:
  Running FHE execution...

[bcm-at-zama]

Any update @jfrery ?

[whcjimmy]

Hi,

Sorry for the late update.

I recorded a video describing the problem I encountered. The FHE inference starts at 4:00.

Although I didn’t wait for the final inference result, I waited for six minutes, during which no GPU resources were utilized.

Video: https://drive.google.com/file/d/1lgK7RlSE0yx7Ho_4aB-liSqGc0nupTJ9/view?usp=sharing