Skip to content

Commit b451858

Browse files
Mr-SunglassesMr-Sunglasses
authored
perf(docker): dispatch CPU kernels at runtime and unify the three images (#48)
1 parent 1e25322 commit b451858

14 files changed

Lines changed: 707 additions & 302 deletions

‎.dockerignore‎

Lines changed: 17 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,19 @@
1-
.venv
2-
.pytest_cache
3-
.ruff_cache
4-
.mypy_cache
5-
__pycache__
6-
tests
7-
data
8-
*.db
9-
*.sqlite*
10-
*.wav
11-
*.log
1+
# Allowlist, not a denylist. The build copies exactly four things — pyproject.toml,
2+
# uv.lock, README.md (hatchling reads it for the wheel metadata) and app/ — so
3+
# everything else is excluded up front and re-admitted below. A denylist has to be
4+
# amended every time the repo grows a directory; this does not, and it keeps the
5+
# context small enough that a build never waits on it.
6+
#
7+
# A populated .env holds the bearer token. No stage copies it, but a `*` rule means
8+
# it is never sent to the daemon or captured in a build cache layer either.
9+
*
1210

13-
# A populated .env holds the bearer token. No Dockerfile copies it, but keeping
14-
# it out of the build context means it is never sent to the daemon or captured
15-
# in a build cache layer.
16-
.env
17-
.envrc
11+
!pyproject.toml
12+
!uv.lock
13+
!README.md
14+
!app
15+
!app/**
1816

19-
# Never needed by a build stage; .git in particular is large and changes on
20-
# every commit, which would invalidate the context for no reason.
21-
.git
22-
.gitignore
23-
.github
24-
docs
25-
scripts
26-
LICENSE
27-
justfile
17+
# Re-exclude what the allowlist above lets back in through app/.
18+
app/**/__pycache__
19+
app/**/*.pyc

‎.env.example‎

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
# 1. Required: bearer token
2828
# 2. Where Docker publishes the gateway
2929
# 3. Address the pairing QR gives the phone
30-
# 4. Which image to run
30+
# 4. Which image to run, and how it is built
3131
# 5. Gateway behaviour
3232
# 6. Listener inside the container
3333
# 7. Settings that do not belong here
@@ -103,7 +103,7 @@ VOCAGATEWAY_PUBLISH_PORT=8765
103103

104104

105105
# ============================================================================
106-
# 4. Which image to run
106+
# 4. Which image to run, and how it is built
107107
# ============================================================================
108108

109109
# Tag for the `gateway` service. Setting it does not by itself switch Compose
@@ -114,9 +114,51 @@ VOCAGATEWAY_PUBLISH_PORT=8765
114114
# docker compose pull
115115
# docker compose up --detach --no-build
116116
#
117-
# The native/cuda/vulkan profile services carry fixed tags and ignore this.
117+
# The cuda/vulkan profile services carry fixed tags and ignore this.
118118
#VOCAGATEWAY_IMAGE=ghcr.io/your-user/vocagateway:latest
119119

120+
# Extra cmake flags for the whisper.cpp build, appended after the Dockerfile's
121+
# own, so a flag here overrides the default. Unlike the git-commit build args in
122+
# section 7, this one belongs in .env: it is a property of the machine you build
123+
# for, not of the moment you built.
124+
#
125+
# The two that come up:
126+
#
127+
# -DCMAKE_CUDA_ARCHITECTURES=89-real
128+
# Build CUDA kernels for one known GPU (89 = Ada / RTX 40-series) instead
129+
# of the portable spread ggml picks. Much faster nvcc and a far smaller
130+
# binary; the image then runs only on that GPU generation.
131+
#
132+
# -DGGML_BLAS=OFF
133+
# Drop OpenBLAS from the CPU image. Measured on arm64 this is a ~1.6x
134+
# slowdown (tiny.en on an 11 s clip: ~2.0 s with BLAS, ~3.2 s without), so
135+
# BLAS is on by default. If your host is x86 with AVX-512 the balance may
136+
# differ; there is a timing recipe under "Tuning the whisper.cpp build" in
137+
# docs/deployment.md.
138+
#
139+
# There is no CPU "native" profile to set here. The image builds every ggml CPU
140+
# micro-architecture variant and loads the best one the host reports at startup.
141+
#VOCAGATEWAY_WHISPER_CMAKE_EXTRA=-DCMAKE_CUDA_ARCHITECTURES=89-real
142+
143+
# How many compile jobs the whisper.cpp build runs at once. Blank is resolved to
144+
# the builder's CPU count. The cuda image is the usual reason to set a lower
145+
# value: nvcc instantiates a great many templates and each job can want most of a
146+
# gigabyte, so a memory-constrained build can be OOM-killed partway through.
147+
# Roughly one job per 2 GB of builder memory. An 8 GB machine normally wants 3.
148+
#VOCAGATEWAY_BUILD_JOBS=3
149+
150+
# Supplementary groups the vulkan profile's container joins so uid 10001 can
151+
# open /dev/dri/renderD128. Without the render group the open fails with EACCES,
152+
# Vulkan finds no device, and whisper.cpp silently falls back to Mesa's software
153+
# rasteriser — slower than the CPU image. The GID differs per distribution (993
154+
# on Ubuntu, 104 on Debian); read your host's with:
155+
#
156+
# stat -c '%g' /dev/dri/renderD128
157+
#
158+
# Ignored by the default and cuda services.
159+
#VOCAGATEWAY_RENDER_GID=993
160+
#VOCAGATEWAY_VIDEO_GID=44
161+
120162

121163
# ============================================================================
122164
# 5. Gateway behaviour

‎.github/workflows/container.yml‎

Lines changed: 169 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,32 @@
11
name: Container
22

33
# Building the image compiles whisper.cpp from source, which is by far the most
4-
# expensive thing in CI. Nothing under app/ can break that build — the runtime
5-
# image only copies the package, it never imports it — so this runs on the files
6-
# that actually decide whether the image assembles: the Dockerfiles and the
7-
# dependency set they install.
4+
# expensive thing in CI. Application behavior is covered by the Quality
5+
# workflow; this one runs on the container definition and dependency inputs
6+
# that decide whether the image itself assembles. Its CPU smoke test then proves
7+
# the installed package starts and serves from that image.
8+
#
9+
# One Dockerfile now builds all three accelerator images, so a change to it can
10+
# break cuda or vulkan while cpu still succeeds. The matrix below builds each
11+
# one. They are the reason this belongs in CI rather than on a laptop. A
12+
# portable cuda image compiles kernels for a broad GPU architecture spread; the
13+
# matrix narrows its compile-only validation to one architecture because no GPU
14+
# executes the cache-only result. Release/operator builds keep the portable
15+
# Dockerfile default.
816
on:
917
push:
1018
branches: [main]
11-
paths:
12-
- 'Dockerfile*'
19+
paths: &container-paths
20+
- 'Dockerfile'
1321
- '.dockerignore'
22+
- 'compose.yaml'
1423
- 'pyproject.toml'
1524
- 'uv.lock'
1625
- '.github/workflows/container.yml'
1726
pull_request:
1827
branches: [main]
1928
types: [opened, synchronize, reopened, ready_for_review]
20-
paths:
21-
- 'Dockerfile*'
22-
- '.dockerignore'
23-
- 'pyproject.toml'
24-
- 'uv.lock'
25-
- '.github/workflows/container.yml'
29+
paths: *container-paths
2630
workflow_dispatch:
2731

2832
permissions:
@@ -33,28 +37,171 @@ concurrency:
3337
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
3438

3539
jobs:
36-
container:
40+
build:
3741
if: github.event_name != 'pull_request' || !github.event.pull_request.draft
3842
runs-on: ubuntu-24.04
39-
timeout-minutes: 30
43+
timeout-minutes: ${{ matrix.timeout }}
44+
strategy:
45+
# One accelerator failing says nothing about the others, and the cuda job
46+
# is long enough that cancelling it on a fast vulkan failure wastes the
47+
# whole run.
48+
fail-fast: false
49+
matrix:
50+
include:
51+
# The cpu image is the one every deployment gets by default, so it is
52+
# also the one that is loaded and actually run below.
53+
- accel: cpu
54+
timeout: 40
55+
smoke: true
56+
reclaim-disk: false
57+
build_jobs: 4
58+
cmake_extra: ''
59+
- accel: vulkan
60+
timeout: 40
61+
smoke: false
62+
reclaim-disk: false
63+
build_jobs: 4
64+
cmake_extra: ''
65+
# The CUDA devel/runtime bases and Python engine layers are still much
66+
# larger than the other variants, so reclaim the runner's unused SDKs.
67+
- accel: cuda
68+
timeout: 40
69+
smoke: false
70+
reclaim-disk: true
71+
build_jobs: 3
72+
# Compiling every CUDA architecture dominates the job despite the
73+
# image never running on a GPU. Ada is a representative real-code
74+
# compile; CPU dispatch is already built and exercised above.
75+
cmake_extra: >-
76+
-DCMAKE_CUDA_ARCHITECTURES=89-real
77+
-DGGML_CPU_ALL_VARIANTS=OFF
78+
name: build (${{ matrix.accel }})
4079
steps:
4180
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
4281
with:
4382
persist-credentials: false
83+
84+
- name: Reclaim runner disk space
85+
if: matrix.reclaim-disk
86+
run: |
87+
# Preinstalled SDKs this build never touches. Worth ~25 GB, which is
88+
# the difference between the cuda image building and the runner
89+
# filling up partway through.
90+
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc
91+
sudo rm -rf /usr/local/share/boost "${AGENT_TOOLSDIRECTORY}"
92+
df -h /
93+
4494
- uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
45-
- name: Build gateway image
95+
96+
- name: Build ${{ matrix.accel }} image
4697
run: |
47-
# cacheonly skips exporting the assembled image to the local daemon.
48-
# The question this job answers is whether the build succeeds, and
49-
# nothing downstream runs the image.
98+
# A smoke-tested image has to reach the local daemon; the others only
99+
# need to prove they assemble, so they stop at the cache.
100+
if [ "${SMOKE}" = "true" ]; then
101+
output="type=docker"
102+
else
103+
output="type=cacheonly"
104+
fi
50105
# Only the sha is stamped here: a commit subject is attacker-supplied
51106
# text on a pull request, and expanding it into this shell would be an
52107
# injection. Release builds that publish an image add the subject from
53108
# a checkout instead.
54109
docker buildx build \
55-
--tag vocagateway:test \
110+
--tag "vocagateway:test-${ACCEL}" \
111+
--build-arg "ACCEL=${ACCEL}" \
112+
--build-arg "BUILD_JOBS=${BUILD_JOBS}" \
113+
--build-arg "WHISPER_CMAKE_EXTRA=${CMAKE_EXTRA}" \
56114
--build-arg VOCAGATEWAY_GIT_COMMIT="${GITHUB_SHA}" \
57-
--cache-from type=gha,scope=gateway \
58-
--cache-to type=gha,mode=max,scope=gateway \
59-
--output type=cacheonly \
115+
--cache-from "type=gha,scope=gateway-${ACCEL}" \
116+
--cache-to "type=gha,mode=max,scope=gateway-${ACCEL}" \
117+
--output "${output}" \
60118
.
119+
env:
120+
ACCEL: ${{ matrix.accel }}
121+
BUILD_JOBS: ${{ matrix.build_jobs }}
122+
CMAKE_EXTRA: ${{ matrix.cmake_extra }}
123+
SMOKE: ${{ matrix.smoke }}
124+
125+
# Everything below is the cpu image only. It is what a laptop build was
126+
# being used to check by hand, and it catches the failures a successful
127+
# `docker build` does not: a backend that is present but never loaded, a
128+
# wheel that is missing its templates, a container that cannot start under
129+
# the hardening compose applies.
130+
- name: Check ggml loads a CPU backend variant
131+
if: matrix.smoke
132+
run: |
133+
set -euo pipefail
134+
curl --fail --location --silent --show-error \
135+
--output jfk.wav \
136+
https://raw.githubusercontent.com/ggml-org/whisper.cpp/v1.9.1/samples/jfk.wav
137+
curl --fail --location --silent --show-error \
138+
--output ggml-tiny.en.bin \
139+
https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin
140+
141+
docker run --rm --volume "${PWD}:/bench:ro" \
142+
--entrypoint whisper-cli vocagateway:test-cpu \
143+
-m /bench/ggml-tiny.en.bin -f /bench/jfk.wav -nt -t 2 -bs 2 -bo 2 \
144+
> transcript.txt 2> backend.log || { cat backend.log; exit 1; }
145+
146+
grep 'load_backend: loaded CPU backend' backend.log
147+
148+
# GGML_CPU_ALL_VARIANTS exists so a portable image still runs the
149+
# host's instruction set. A GitHub runner is well past Haswell, so
150+
# loading the x64 baseline means dispatch silently regressed and every
151+
# deployment is running scalar kernels.
152+
if grep -q 'libggml-cpu-x64\.so' backend.log; then
153+
echo "::error::ggml fell back to the baseline x64 CPU backend" >&2
154+
cat backend.log >&2
155+
exit 1
156+
fi
157+
158+
grep -qi 'ask not what your country can do for you' transcript.txt
159+
160+
- name: Check the container serves under compose's hardening
161+
if: matrix.smoke
162+
run: |
163+
set -euo pipefail
164+
# The same restrictions compose.yaml applies, so a capability the
165+
# gateway turns out to need fails here rather than on a deployment.
166+
docker run --detach --name vg-smoke --publish 18765:8765 \
167+
--cap-drop ALL \
168+
--security-opt no-new-privileges:true \
169+
--tmpfs /tmp:size=64m,mode=1777 \
170+
vocagateway:test-cpu
171+
172+
for _ in $(seq 1 30); do
173+
status="$(docker inspect --format '{{.State.Health.Status}}' vg-smoke)"
174+
[ "${status}" = "healthy" ] && break
175+
[ "${status}" = "unhealthy" ] && break
176+
sleep 2
177+
done
178+
179+
if [ "${status}" != "healthy" ]; then
180+
echo "::error::container never became healthy (${status})" >&2
181+
docker logs vg-smoke >&2
182+
exit 1
183+
fi
184+
185+
curl --fail --silent --show-error --output /dev/null http://127.0.0.1:18765/health/live
186+
# The WebUI renders Jinja templates from the installed wheel, so a 200
187+
# here is what proves the non-editable install carried app/templates.
188+
curl --fail --silent --show-error --output /dev/null http://127.0.0.1:18765/
189+
190+
- name: Container logs
191+
if: failure() && matrix.smoke
192+
run: docker logs vg-smoke || true
193+
194+
compose:
195+
if: github.event_name != 'pull_request' || !github.event.pull_request.draft
196+
runs-on: ubuntu-24.04
197+
timeout-minutes: 5
198+
steps:
199+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
200+
with:
201+
persist-credentials: false
202+
- name: Validate every profile interpolates
203+
run: |
204+
# `just compose` checks the default service. The profile services carry
205+
# the device wiring, so they need naming explicitly to be parsed.
206+
VOCAGATEWAY_TOKEN=test-token-with-at-least-thirty-two-characters \
207+
docker compose --profile cuda --profile vulkan config --quiet

‎AGENTS.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -119,9 +119,9 @@ Docker Desktop cannot use macOS MLX / WhisperKit / Core ML. Host-only engines
119119
(`vocamac`, `handy`, `mlx-audio`, `whisperkit`) are hidden in the WebUI on Linux
120120
and in containers; API select is `422 invalid_engine`. Compose default publish is
121121
`127.0.0.1:8765`. `VOCAGATEWAY_NETWORK_MODE=host` is Linux Docker Engine only
122-
(not Docker Desktop) and ignores `VOCAGATEWAY_PUBLISH_HOST`/`PORT`. Profiles
123-
`native` / `cuda` / `vulkan` share the same port and `vocagateway-data` volume —
124-
run one service at a time.
122+
(not Docker Desktop) and ignores `VOCAGATEWAY_PUBLISH_HOST`/`PORT`. The default
123+
CPU service and the `cuda` / `vulkan` profile services share the same port and
124+
`vocagateway-data` volume — run one service at a time.
125125

126126
## Consumers
127127

@@ -160,7 +160,7 @@ behind a reverse proxy at a **domain root**, not a subpath.
160160
| Workflow | When | What |
161161
| --- | --- | --- |
162162
| `quality.yml` | `app/`, `tests/`, `scripts/`, `pyproject.toml`, `uv.lock`, `compose.yaml` | ffmpeg, ruff, format `--check`, `mypy app`, pytest, compose config |
163-
| `container.yml` | `Dockerfile*`, `.dockerignore`, `pyproject.toml`, `uv.lock` | `docker buildx` of `Dockerfile` only (`cacheonly`; not CUDA/Vulkan) |
163+
| `container.yml` | `Dockerfile`, `.dockerignore`, `compose.yaml`, `pyproject.toml`, `uv.lock`, workflow | `docker buildx` CPU/CUDA/Vulkan matrix (representative CUDA architecture), CPU backend/runtime smoke test, all-profile Compose config |
164164
| `verify-model-pins.yml` | pin/catalog/harvester paths + weekly | `scripts/verify-model-pins.py` |
165165
| `deploy-pages.yml` | `web/**` on `main` | GitHub Pages |
166166

@@ -177,7 +177,7 @@ hostnames. Diagnostics omit those. Keep `.env` and token files local.
177177
- Never commit or push to `main`. Do not merge PRs; wait for review.
178178
- PR body should match [`.github/pull_request_template.md`](.github/pull_request_template.md):
179179
**Summary** (what/why), **Verification** (`just test` or the equivalent ruff /
180-
mypy / pytest / compose checks; container build if Dockerfiles or the lockfile
180+
mypy / pytest / compose checks; container build if the Dockerfile or lockfile
181181
changed; docs if setup/network/config changed), **Privacy and security**
182182
checklist.
183183
- One logical change per PR.

0 commit comments

Comments
 (0)