switch GPU workers to generic-worker & dockerize GPU tasks (#700)

* switch GPU workers to d2g images

* Switch GPU tasks to run within docker

The new image we're upgrading GPU workers to uses Ubuntu 24.04, which makes it incompatible with various parts of the pipeline (mostly due to Python package pinning). As it turns out, the easiest way to fix this is to dockerize the GPU tasks.

We need slight updates to GPU task payloads to accommodate this.

This will fix #391.

* pull in cuda-toolkit in GPU tasks to make necessary libraries available

Now that we're running inside a docker image we don't have these available on the filesystem already. I explored the idea of installing them into the Docker image, but it's quite impractical. The host image is Ubuntu 24.04, and the containers Ubuntu 22.04. We need to have a matching toolkit version, and we require version 12 at this point, which isn't available on 22.04.

* enable cache for ~/.task-cache/pip

Without this we end up with these files being inaccessible in subsequent tasks.

* fix: add 'requests' to ctranslate2 requirements

This has always been needed, but it was found on the host system on the previous image.

* update interactivate task documentation

All tasks now use the docker-worker format; no need to distinguish between GPU and non-GPU tasks.

Also add a bit more detail about working in the container.

* drop now-unused cuda-toolkit-11 toolchain

* feat: use volume mount for artifacts

This ensures artifacts will be uploaded if a spot termination happens.

Author: bhearsum

docs/training/task-cluster.md

@@ -136,24 +136,16 @@ To start an interactive task, follow these steps:
 
 5. Reduce the maxRunTime to a best guess at how long you'll need the task and worker running for. (We pay for every minute a worker runs - so they should not be kept running, eg: overnight.)
 
-6. Adjust the payload to simply run bash and sleep (instead of a full pipeline step). For docker-worker tasks use something like:
+6. Adjust the payload to simply run bash and sleep (instead of a full pipeline step):
 ```
      command:
     - bash
     - '-c'
     - 'sleep 7200'
 ```
 
-For generic-worker tasks (those needing a GPU), use:
-```
-     command:
-    - - bash
-      - '-c'
-      - 'sleep 7200'
-```
-
-(docker-worker tasks have an `image` section in the payload)
-
 7. Click "Create Task"
 
-After a few minutes you should be able to get a shell (a link will show up in the tab when it's ready).
+After a few minutes you should be able to get a shell (a link will show up in the tab when it's ready). This shell should drop you inside of docker container as root, running the same image as the task you started this process with. Most tasks drop privileges to the `worker` user before doing any work, so you may want to run `su - worker` before doing anything of note.
+
+When you are done with the worker you can use "Cancel" from the three dots menu to immediately shut it down. (This should happen within a few minutes of closing your last shell to the worker, but it's good practice to do it yourself to minimize costs.)

pipeline/bicleaner/bicleaner.sh

@@ -13,10 +13,6 @@ echo "###### Bicleaner filtering"
 test -v SRC
 test -v TRG
 test -v CUDA_DIR
-test -v CUDNN_DIR
-
-# cuda and cudnn libs
-export LD_LIBRARY_PATH=${CUDA_DIR}/lib64:${CUDNN_DIR}:${LD_LIBRARY_PATH:+LD_LIBRARY_PATH:}
 
 corpus_prefix=$1
 output_prefix=$2

pipeline/translate/requirements/translate-ctranslate2.in

@@ -1,3 +1,4 @@
 ctranslate2==4.3.1
 sentencepiece==0.2.0
 gpustat==1.1.1
+requests==2.32.3

pipeline/translate/requirements/translate-ctranslate2.txt

@@ -93,39 +93,39 @@ workers:
             worker-type: 'b-linux-large-gcp-1tb-64-512-std-d2g'
         b-linux-v100-gpu:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g'
         b-linux-v100-gpu-4:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4'
         b-linux-v100-gpu-4-300gb:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4-300gb'
         b-linux-v100-gpu-4-300gb-standard:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4-300gb-standard'
         b-linux-v100-gpu-4-1tb:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4-1tb'
         b-linux-v100-gpu-4-2tb:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4-2tb'
         b-linux-v100-gpu-4-1tb-standard:
             provisioner: '{trust-domain}-{level}'
-            implementation: generic-worker
+            implementation: docker-worker
             os: linux
-            worker-type: '{alias}'
+            worker-type: 'b-linux-v100-gpu-d2g-4-1tb-standard'
         images:
             provisioner: '{trust-domain}-{level}'
             implementation: docker-worker

taskcluster/config.yml

@@ -59,6 +59,6 @@ ENV SHELL=/bin/bash \
     TERM=hterm-256color
 
 VOLUME /builds/worker/checkouts
-VOLUME /builds/worker/.cache
+VOLUME /builds/worker/.task-cache/pip
 
 USER root

taskcluster/docker/base/Dockerfile

@@ -68,6 +68,6 @@ ENV SHELL=/bin/bash \
     PATH="/builds/worker/.local/bin:$PATH"
 
 VOLUME /builds/worker/checkouts
-VOLUME /builds/worker/.cache
+VOLUME /builds/worker/.task-cache/pip
 
 USER root

taskcluster/docker/inference/Dockerfile

@@ -27,4 +27,4 @@ RUN curl -sSLf "https://github.com/go-task/task/releases/download/v3.35.1/task_l
     | tar -xz -C /usr/local/bin
 
 VOLUME /builds/worker/checkouts
-VOLUME /builds/worker/.cache
+VOLUME /builds/worker/.task-cache/pip

taskcluster/docker/test/Dockerfile

@@ -45,6 +45,6 @@ ENV SHELL=/bin/bash \
     PATH="/builds/worker/.local/bin:$PATH"
 
 VOLUME /builds/worker/checkouts
-VOLUME /builds/worker/.cache
+VOLUME /builds/worker/.task-cache/pip
 
 USER root

taskcluster/docker/toolchain-build/Dockerfile

@@ -16,5 +16,5 @@ RUN apt-get update -qq \
                           wget \
     && apt-get clean
 
-
 VOLUME /builds/worker/checkouts
+VOLUME /builds/worker/.task-cache/pip