Using CUDA

[ItsMeJacky]

How to run faster-whisper using CUDA ?

[rufinus]

cudnn libs + --device cuda

on my LXC setup i followed this article https://sluijsjes.nl/2024/05/18/coral-and-nvidia-passthrough-for-proxmox-lxc-to-install-frigate-video-surveillance-server/

you just have to include the correct libs in the container, or map damn from the host to the container.

i ended up building the cointainer myself with this dockerfile:

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/12.2/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas-12-0
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb

took some time to find the right versions to make it run.

[ssamjh]

👍 For official GPU support, would be nice to speed things up :)

[ssamjh]

👍 for this feature. I already have a GPU in my server for other applications :)

[Mithras]

Also, is it possible to make it unload model based on some --keep-alive param?

[alienatedsec]

How to run faster-whisper using CUDA ?

Simple and works out of the box - https://github.com/linuxserver/docker-faster-whisper

docker run -d \
  --name=faster-whisper-gpu \
  --gpus=all \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Etc/UTC \
  -e WHISPER_MODEL=tiny-int8 \
  -e WHISPER_BEAM=1 `#optional` \
  -e WHISPER_LANG=en `#optional` \
  -p 10300:10300 \
  -v /path/to/faster-whisper/data:/config \
  --restart unless-stopped \
  lscr.io/linuxserver/faster-whisper:gpu

[AnkushMalaker]

I made PR #44 to allow building it directly from this repository, and use CUDA.
It works perfectly with just docker compose up -d or using the build scripts in the README.
@synesthesiam will need to upload a gpu based image to rhasspy dockerhub if they want to allow people to pull without building locally.

[especially-relative]

RUN apt update && apt install -y wget software-properties-common \
  && wget https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && dpkg -i cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && cp /var/cudnn-local-repo-debian11-9.1.0/cudnn-*-keyring.gpg /usr/share/keyrings/ \
  && wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb \
  && dpkg -i cuda-keyring_1.1-1_all.deb \
  && add-apt-repository contrib && apt --allow-releaseinfo-change update \
  && apt -y install libcudnn9-cuda-12 libcublas-12-0 \
  && rm cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb

working with 2.4.0

[xiasi0]

Using CUDNN9.6 and 2.4.0 versions does not work properly. I hope someone can give me a hint

Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}

Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

Do I have to use CUDNN9.1？

#35 (comment)
It can now work using CUDA imag

[jjsmackay]

Thanks to @rufinus, I got this working on a GTX 970M.

Sadly this poxy little GPU only supports CUDA Compute Capability 5.2, and I couldn't get it working on CUDA 12. The CUDA 11 workaround discussed here worked a treat.

Getting <1s translation using small.en model and beam-size 3 with just 3GB VRAM. Happy days!

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/11.8/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && add-apt-repository non-free && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas11
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN pip install --force-reinstall ctranslate2==3.24.0

[kds69]

On ubuntu 22.04 desktop, whatever I am trying with all nvidia drivers version (from 470 to 550... 570 not available for desktop), and rhasspy version, I get the below error:
RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

Please nite that I'd like to stick to ubuntu desktop and preferably nvidia driver 470, because it saves a lot of watts (14w vs 27w on idle, no matter gpu P8 state or not). For an server running 24/7 this is not neglectible.

You are just adding --device cuda option for docker run, right?

[kds69]

solved with:

• RUN pip install --force-reinstall ctranslate2==4.4.0
• a CUDA=12 driver installed on host (535 for instance)

but this didn't help on saving watts. I'll try with the script installed locally + change requirements.txt to force ctranslate2 version.

--edit--
solved with a script on start-up (a start up service that runs the script in fact) with basically do the following:
echo "suspend" > /proc/driver/nvidia/suspend
echo "resume" > /proc/driver/nvidia/suspend

This somehow force driver to idle back at lowest consumption possible, regaining 13w power saving from 27w on idle to 14w on idle. No need to re-run above commands, wished behavior remains until next reboot.

[AnkushMalaker]

FYI guys, I've been running this fork for a few months and its been working great. Could someone help out with a review?
#44

[juan11perez]

@alienatedsec
Thank you for sharing this. The linuxserver image does have gpu capability and it's updated to 2.5.0!