Add WhisperX ASR container (EC2 + SageMaker, AL2023 GPU)#6396
Draft
Yadan-Wei wants to merge 9 commits into
Draft
Add WhisperX ASR container (EC2 + SageMaker, AL2023 GPU)#6396Yadan-Wei wants to merge 9 commits into
Yadan-Wei wants to merge 9 commits into
Conversation
Onboard the whisperx framework: Dockerfile (CUDA 12.8 / Python 3.12, ec2 + sagemaker targets), two image configs, a pre-build hook that stages the pyannote diarization weights into the build context, and a CI pipeline running build + sanity + security + telemetry for both variants. Functional EC2/SageMaker endpoint tests to follow.
added 8 commits
July 13, 2026 20:32
- deep_learning_container.py: add "whisperx" to the --framework argparse choices so container telemetry fires (argparse was rejecting the value with exit 2, so telemetry never ran and the telemetry-instance tests failed). - Dockerfile.amzn2023: apply dockerfmt (2-space indent) to satisfy the pre-commit docker formatter. - server.py: apply ruff import sorting.
The Silero bake ran torch.hub.load('snakers4/silero-vad', ...), which
makes a live GitHub call at build time and failed with HTTP 403 rate
limit exceeded under CI's shared-IP load.
Silero is not needed: whisperx 3.8.6 defaults to vad_method="pyannote"
and server.py never selects silero. The default pyannote VAD's
segmentation model ships inside the whisperx wheel
(whisperx/assets/pytorch_model.bin) and loads from a local file with no
network (verified with docker run --network=none), so the default
serving path is fully offline. Removing the bake eliminates the
container's only build/runtime GitHub dependency with no loss on the
default or API-exposed path. HF_HOME/TORCH_HOME are kept for lazy
runtime caching of Whisper + per-language wav2vec2 aligners.
…CVEs ECR enhanced scan flagged 19 CRITICAL/HIGH CVEs. Remediation: - python-multipart 0.0.22 -> 0.0.30 fixes CVE-2026-42561 and CVE-2026-53539 (both on the multipart request-parsing path). - Allowlist 8 CVEs whose fix requires a major bump that breaks the whisperx/fastapi/CUDA contract, each with a specific reason and a review_by date: - transformers CVE-2026-4372, CVE-2026-1839 (whisperx caps transformers<5; Trainer path unused by inference) - torch CVE-2026-24747, CVE-2025-55552 (whisperx caps torch~=2.8.0) - starlette CVE-2026-54283, CVE-2026-48818 (fastapi pins starlette<0.50; SSRF is Windows-only and no StaticFiles is served) - cuda-toolkit CVE-2025-23339, CVE-2025-23308 (nvidia/cuda 12.8 base; fix needs CUDA 13; dev-only cuobjdump/nvdisasm tools) The 9 Rust CVEs (openssl/tar/bytes/rustls-webpki) were resident only in the Silero torch.hub download and are removed by dropping that bake.
Phase 2 of the whisperx onboarding: functional test suites plus their reusable workflows, gated per customer_type. EC2 (test/whisperx/ec2/, x86-g6xl-runner, docker run --gpus all -p 8000): ping/models health, basic transcription, word timestamps, non-English (zh) + language echo, diarization (>=2 speakers), response formats (text/srt/vtt), and error handling (404/400). SageMaker (test/whisperx/sagemaker/, ml.g6.xlarge, default-runner): sync /invocations transcription + diarization, and an async S3-in/S3-out long-audio test. Endpoints pin INFERENCE_AMI_VERSION_CU12 (CUDA 12.8 image) and clean up in a finally block. Wiring: whisperx.tests-ec2.yml + whisperx.tests-sagemaker.yml reusable workflows; whisperx.pipeline.yml gains a ci-config job plus ec2-test (customer_type==ec2) and sagemaker-test (customer_type==sagemaker) jobs; pr-amzn2023.yml adds test/whisperx paths, change filters, and the two run-*-test inputs. Assertions are structural (non-empty text, speaker count, status codes) to stay deterministic against nondeterministic ASR. Audio fixtures staged at s3://dlc-cicd-models/test-fixtures/audio/ (reuse asr_en.wav + asr_zh.wav; add asr_diarize_2spk.wav + asr_long_60s.wav).
Model-launch-config and response-contract coverage, plus the new
transformers CVE from the latest ECR scan.
EC2 (test/whisperx/ec2/):
- common.py: launch containers with custom env (-e) via a
run_container_with_env context manager (always tears down).
- Group A (test_ec2_model_config.py): custom WHISPERX_DEFAULT_MODEL,
served-model alias, ALLOW_MODEL_OVERRIDE on (accepts other models),
and override-denied 404 response-body shape.
- Group B (test_ec2_gpu.py): verbose_json field set, json-minimal
(text only), diarize=false emits no speaker fields, max_speakers cap.
SageMaker (test/whisperx/sagemaker/):
- C1 (test_sm_endpoint.py): unknown model over /invocations surfaces
the 404 (ModelError / ClientError) naming the served model.
- C2 (test_sm_model_config.py): a second endpoint deployed with
environment={WHISPERX_DEFAULT_MODEL: tiny} proves ContainerDefinition
env propagates to the container (large-v2 rejected, tiny served).
Security: allowlist CVE-2026-5241 (transformers, fix needs >=5.5.0;
whisperx 3.8.6 caps transformers<5) with the same contract reason and
review_by as the sibling transformers CVEs.
Assertions are structural (ids, status, error text, key presence) to
stay deterministic against nondeterministic ASR output.
The entrypoints exec uvicorn directly, so the bashrc-sourced telemetry hook (which needs an interactive/login shell) never runs at real container start — only the telemetry CI test triggered it via bash -ic. Add an explicit `bash bash_telemetry.sh` call at the top of both the EC2 and SageMaker entrypoints, matching the vllm/sglang/ray convention, so telemetry fires once per container launch. Fire-and-forget with errors suppressed (|| true) so it never blocks or fails startup.
…ing, CI unit tests Server (scripts/docker/whisperx/server.py): - Expose full WhisperX decoding/VAD/model config as WHISPERX_* launch env vars (asr_options + vad_options), read once into immutable module globals. - Remove temperature/prompt from the request body (now launch-only); they are load-time asr_options in WhisperX and passing them to transcribe() raised TypeError. This is the one breaking change vs the prior request shape. - Make WHISPERX_TASK effective (pass task per-call; load_model discards it when no language is pinned) and WHISPERX_ALIGN_MODEL effective (forward model_name to load_align_model). - Offload the blocking transcribe to a worker thread (anyio.to_thread) so the event loop / GET /ping stays responsive under long requests. - Narrow the alignment except to (ValueError, NotImplementedError, KeyError) so real errors surface as 500 instead of a silently-degraded 200; return 422 when diarization was requested but alignment is unavailable. - Guard the model LRU caches with per-cache locks to dedupe concurrent loads and avoid an evict/touch race now that transcribe runs off-loop. - Add srt/vtt line-formatting (max_line_width/max_line_count/highlight_words), WhisperX-CLI-named, srt/vtt-only, reusing WhisperX's WriteSRT/WriteVTT so output matches the CLI; forces alignment internally when set. Ignored for json/text/verbose_json. CI: - Add whisperx.tests-unit.yml (CPU-only, image-independent) and wire it into the pipeline + PR caller; runs immediately (no build dependency). - Run the whole whisperx/ec2 dir so test_ec2_model_config.py is executed. Tests: add GPU-free unit suites for options, transcribe behavior, and subtitle formatting (49 tests). Signed-off-by: Yadan Wei <weiyadan@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Onboards a new WhisperX ASR inference container into the V2 config-driven CI, with both EC2 and SageMaker variants (AL2023, CUDA 12.8, Python 3.12).
WhisperX is a BSD-2-Clause ASR pipeline (Silero VAD → faster-whisper transcription → wav2vec2 word alignment → optional pyannote diarization) served over an OpenAI-compatible FastAPI/uvicorn API.
What's included (Phase 1 — build + common tests)
docker/whisperx/Dockerfile.amzn2023(targetswhisperx-ec2-amzn2023,whisperx-sagemaker-amzn2023) +NOTICEscripts/docker/whisperx/{server.py,dockerd_entrypoint.sh,sagemaker_entrypoint.sh,cuda_compat.sh}scripts/ci/build/whisperx/pre_build.sh— stages pyannote diarization weights into the build context.github/config/image/whisperx/{ec2,sagemaker}-amzn2023.ymlwhisperx.pipeline.yml(build + sanity + security + telemetry),whisperx.pr-amzn2023.ymltest/security/data/ecr_scan_allowlist/whisperx/framework_allowlist.json(empty starting allowlist)The Dockerfile follows the established
nvidia/cuda:*-amzn2023pattern (as used by sglang/vllm/ray) and adds the standard DLC telemetry + OSS-compliance overlay so it passes the cross-framework sanity/telemetry checks.Scope / follow-ups
release: falseon both configs — this onboards CI validation, not a release.docker run --gpus all, HTTP :8000) and SageMaker endpoint tests are intentionally deferred to a Phase 2 PR, to land the build + common-test foundation first.Test plan