perf(docker): dispatch CPU kernels at runtime and unify the three images by Mr-Sunglasses · Pull Request #48 · VocaHQ/vocagateway

Mr-Sunglasses · 2026-09-03T22:11:26Z

Problem

The portable CPU image disabled native code generation without replacing it with
runtime dispatch, leaving x86 hosts on baseline kernels and arm64 hosts without
dot-product or i8mm acceleration. The separate CPU, CUDA, and Vulkan Dockerfiles
also duplicated most of the image and had already drifted: the Vulkan service
could see /dev/dri but its unprivileged user could not open the render node.

The first CI run exposed one more build problem. An empty BUILD_JOBS reached
CMake as bare --parallel; with Make that means unlimited jobs, not one job per
CPU. The Vulkan runner launched all ggml variants together and was terminated
with signal 143 under the resulting resource pressure. The full portable CUDA
architecture spread also made a compile-only PR check take more than 45 minutes.

Summary

Replace the three Dockerfiles with one ACCEL=cpu|cuda|vulkan multi-stage
Dockerfile and share the common Compose service configuration.
Build ggml CPU variants for runtime dispatch, retaining portable images while
loading the best supported host backend.
Give the Vulkan container the host render/video supplementary groups, with
distribution-specific GIDs configurable in .env.
Keep OpenBLAS enabled based on the measured arm64 result: roughly 2.0 seconds
with BLAS versus 3.2 seconds without it for the same tiny.en sample.
Add BuildKit apt, uv, and ccache mounts; precompile Python bytecode; keep the uv
cache out of the final image; and expose build-job/CMake tuning controls.
Add log rotation, cap_drop: ALL, no-new-privileges, and a bounded tmpfs for
per-request temporary files.
Build CPU, Vulkan, and CUDA images in GitHub Actions and smoke-test the CPU
backend, transcript, health endpoint, WebUI, and runtime hardening.
Resolve blank BUILD_JOBS to nproc, and give CI explicit memory-safe job
caps: 4 for CPU/Vulkan and 3 for CUDA.
Make the compile-only CUDA CI job target one representative Ada architecture
and omit CPU variants already covered by the CPU job. Normal Dockerfile,
Compose, operator, and release builds retain the portable CUDA defaults.
Align the README, deployment, configuration, troubleshooting, docs index, and
contributor guidance with the unified images and current CI behavior.

Verification

just test (Ruff, format, WPS, strict mypy, package/lock checks, audit,
543 tests, and default Compose validation)
All Compose profiles interpolate with a valid dummy token
actionlint .github/workflows/container.yml
zizmor .github/workflows/container.yml
docker buildx build --check . (only the expected false positive for the
VOCAGATEWAY_TOKEN_FILE secret-file path)
GitHub Actions Container matrix: CPU 5m38s, Vulkan 8m21s, CUDA 15m15s,
Compose 5s
Operator and contributor documentation audited and updated

The M1 laptop is useful for native Linux arm64 and static checks, but it does not
cover the Linux amd64 or NVIDIA CUDA paths. The GitHub-hosted matrix is the
acceptance test for those images. Its CUDA result is compile validation, not a
published or release-ready image.

Privacy and security

No secrets, recordings, transcripts, or private hostnames added
No weakening of bearer auth, upload limits, retention, or default binding
without discussion

The CPU image was built with GGML_NATIVE=OFF, which on x86 means baseline scalar kernels — no AVX2, no FMA — and on arm64 a plain armv8.0 build with neither dotprod nor i8mm. The `native` profile existed to escape that, at the cost of an image that could not be published. Build with GGML_CPU_ALL_VARIANTS instead. ggml compiles one CPU backend per micro-architecture and dlopens the best one the host reports at startup, so a portable image runs the host's instruction set. Verified on arm64: the image loads libggml-cpu-armv8.2_2.so (dotprod + fp16 arithmetic) rather than the armv8.0 baseline. The `native` profile and its service are gone. This needs GGML_BACKEND_DL, so the backends are sorted into a compiled-in GGML_BACKEND_DIR and the build no longer targets whisper-cli alone — the variants are dlopen'd modules nothing links against, and a whisper-cli-only build produced a binary with no CPU backend to load. Two measurements corrected assumptions rather than confirming them: - OpenBLAS earns its place and stays on. tiny.en on an 11 s clip, --cpus 4: ~2.0 s with BLAS against ~3.2 s without, over five runs each. - OPENBLAS_NUM_THREADS is deliberately left unset. OpenBLAS-pthread cannot see the cgroup quota, which looks like guaranteed oversubscription, but ggml issues BLAS matmuls from one thread at a time so the pools never nest. Pinned and unset were within noise for tiny.en and base.en at --cpus 4 and --cpus 2 on a 10-CPU host. Fix the Vulkan profile, which could not reach the GPU at all: /dev/dri was passed through but the render node is root:render and the gateway runs as uid 10001, so the open failed with EACCES and whisper.cpp fell back to Mesa's software rasteriser — slower than the CPU image, and silent about it. Collapse Dockerfile.cuda and Dockerfile.vulkan into the one Dockerfile, chosen by an ACCEL build arg, and lift the repeated service body in compose.yaml into an anchor. The runtime half was copy-pasted three times and each service repeated ten lines. Build and runtime work besides: - apt, uv and ccache cache mounts; the uv cache no longer ships in the image - UV_COMPILE_BYTECODE, so a restart does not pay to compile the dependency tree on its first request - gcc-14 on the Ubuntu builders: GCC 13 rejects -march=...+sme and fails the armv9.2 variants on an arm64 builder - BUILD_JOBS, because nvcc with one job per core OOM-kills a small builder partway through and reports only Error 137 - json-file log caps, cap_drop, no-new-privileges, and a tmpfs /tmp that also takes per-request transcripts off the overlay filesystem - .dockerignore becomes an allowlist, so a new directory is out of the build context by default CI builds all three accelerators on a matrix rather than leaving cuda and vulkan to be discovered on a laptop, and smoke-tests the cpu image: that a non-baseline CPU backend loads, that a clip transcribes, and that the container serves the WebUI under the same hardening compose applies. Claude-Session: https://claude.ai/code/session_014r21LLf1huM8UNq7tGgk2x

netlify · 2026-09-03T22:11:32Z