fix(whisperx): serialize GPU inference, one model per container, admission control + readiness

Addresses code-review findings on the WhisperX serving container.

BREAKING (request API): the `model` request field is removed. The container
serves exactly one model, chosen at launch via WHISPERX_DEFAULT_MODEL. FastAPI
ignores an undeclared `model` field, so OpenAI-SDK clients that always send it
still work — the value is a no-op. /v1/models still advertises the served id.

Concurrency + safety:
- Serialize GPU inference with an anyio CapacityLimiter (default 1,
  WHISPERX_MAX_CONCURRENT_REQUESTS). WhisperX's pipeline mutates shared state
  mid-call and the diarization pipeline is not thread-safe, so concurrent
  inference on one instance is unsafe; excess requests wait on the event loop
  so /ping stays responsive.
- Remove per-request model selection and the model LRU; cache a single model
  under a lock. This makes it structurally impossible to hold two Whisper
  models on one GPU. Align LRU (keyed by language) retained; it now frees CUDA
  memory on eviction.

Correctness + operations:
- Fix the response `task` field to reflect WHISPERX_TASK (was hardcoded
  "transcribe", so translate mode reported the wrong task).
- Admission control: 503 when the inference queue is full (WHISPERX_MAX_QUEUE),
  413 when an upload exceeds WHISPERX_MAX_UPLOAD_BYTES (default 1 GiB).
- Readiness health: /ping returns 503 after a fatal CUDA/device fault corrupts
  the process GPU context, so orchestration can drain/replace the host.
  Transient failures (CUDA OOM, unsupported-language degrade, 4xx) do not flip
  health. This is a passive signal — it reflects faults observed by the
  inference path, not idle GPU death.
- Validate request params at the boundary (min/max_speakers, line widths) with
  422 instead of opaque downstream errors.

Security: allowlist CVE-2025-55551 (torch 2.8.0 torch.linalg.lu DoS; fix needs
torch>=2.9.0 which breaks the whisperx 3.8.6 pin; off the ASR inference path).

Tests: GPU-free unit coverage for inference serialization, readiness/fatal-CUDA
classification, admission control, task-from-env, and param validation; update
the EC2/SageMaker model tests for the model-field-ignored contract.

Signed-off-by: Yadan Wei <weiyadan@amazon.com>

Author: Yadan Wei

scripts/docker/whisperx/server.py

@@ -24,6 +24,11 @@
         "reason": "torch is pinned to 2.8.0 by whisperx 3.8.6 (torch~=2.8.0); the fix requires torch>=2.9.0 and cannot be taken without breaking the whisperx contract. The reported torch.rot90/torch.randn_like interaction is not on the ASR inference path. Tracked for a whisperx version that raises its torch cap.",
         "review_by": "2026-10-13"
     },
+    {
+        "vulnerability_id": "CVE-2025-55551",
+        "reason": "torch is pinned to 2.8.0 by whisperx 3.8.6 (torch~=2.8.0); the fix requires torch>=2.9.0 and cannot be taken without breaking the whisperx contract. The reported torch.linalg.lu slice denial-of-service is not on the ASR inference path (transcription/alignment/diarization do not call torch.linalg.lu). Tracked for a whisperx version that raises its torch cap.",
+        "review_by": "2026-10-13"
+    },
     {
         "vulnerability_id": "CVE-2026-54283",
         "reason": "starlette is a transitive dependency of fastapi 0.121.0, which pins starlette<0.50.0; the fix requires starlette>=1.3.1 and cannot be taken without a major fastapi bump that breaks the serving stack. Tracked for a fastapi release that allows starlette 1.x.",

test/security/data/ecr_scan_allowlist/whisperx/framework_allowlist.json

@@ -161,20 +161,17 @@ def test_response_formats(container, aws_session, tmp_path):
 
 
 def test_errors(container, tmp_path):
-    """Contract errors: unknown model -> 404, empty upload -> 400, bad format -> 400.
+    """Contract errors: bad response_format -> 400, empty upload -> 400.
 
-    The server validates the model, then the response_format, then reads the
-    file, so the model/format cases need only a present (unread) file part.
+    The server validates response_format before reading the file, so the format
+    case needs only a present (unread) file part.
     """
     port = container["port"]
 
     # A present-but-never-read file part (validation short-circuits before read).
     dummy = tmp_path / "dummy.wav"
     dummy.write_bytes(b"RIFF0000WAVE")
 
-    resp = post_transcription(port, str(dummy), model="does-not-exist-xyz")
-    assert resp.status_code == 404, resp.text
-
     resp = post_transcription(port, str(dummy), response_format="xml")
     assert resp.status_code == 400, resp.text
 

test/whisperx/ec2/test_ec2_gpu.py

@@ -13,19 +13,23 @@
 """EC2 model-launch-config tests for the WhisperX ASR DLC (GPU).
 
 Group A: these tests validate how the server's launch-time env vars shape the
-model-resolution contract — WHISPERX_DEFAULT_MODEL (the pinned/served model),
-WHISPERX_SERVED_MODEL_NAME (a client-facing alias), and
-WHISPERX_ALLOW_MODEL_OVERRIDE (per-request on-demand loading). Each case needs a
-container launched with different env, so unlike test_ec2_gpu.py they do NOT use
-the shared default `container` fixture; instead each spins up (and tears down) a
-custom-env container via common.run_container_with_env.
-
-WHISPERX_DEFAULT_MODEL=tiny is used throughout so the startup warm-load (and any
-on-demand override load) finishes in seconds — tiny is a valid faster-whisper
-model that downloads far faster than the large-v2 default.
-
-Assertions target the resolution contract only (served model id, 200 vs 404, and
-the 404 error-body shape) — never exact transcript text, which is nondeterministic.
+served-model contract — WHISPERX_DEFAULT_MODEL (the one model per container) and
+WHISPERX_SERVED_MODEL_NAME (a client-facing alias advertised by /v1/models).
+Each case needs a container launched with different env, so unlike test_ec2_gpu.py
+they do NOT use the shared default `container` fixture; instead each spins up (and
+tears down) a custom-env container via common.run_container_with_env.
+
+The Whisper model is one-per-container: the request `model` field is accepted but
+ignored (an OpenAI-compat no-op), so every transcription runs the launched model
+regardless of the `model` value sent.
+
+WHISPERX_DEFAULT_MODEL=tiny is used throughout so the startup warm-load finishes
+in seconds — tiny is a valid faster-whisper model that downloads far faster than
+the large-v2 default.
+
+Assertions target the served contract only (advertised model id, and that any
+`model` value transcribes 200) — never exact transcript text, which is
+nondeterministic.
 """
 
 import requests
@@ -38,17 +42,18 @@
 
 DEVICE = "gpu"
 DOCKER_RUN_FLAGS = ["--gpus", "all"]
-# Small, fast-downloading Whisper model so warm-load and on-demand override load
-# take seconds, not the minutes large-v2 needs. tiny is a valid faster-whisper id.
+# Small, fast-downloading Whisper model so the startup warm-load takes seconds,
+# not the minutes large-v2 needs. tiny is a valid faster-whisper id.
 FAST_MODEL = "tiny"
 
 
 def test_custom_default_model(image_uri, aws_session, tmp_path):
-    """WHISPERX_DEFAULT_MODEL pins the served id; only that id (or none) resolves.
+    """WHISPERX_DEFAULT_MODEL sets the one served model; the `model` field is ignored.
 
     Launches with WHISPERX_DEFAULT_MODEL=tiny and asserts /v1/models advertises
     `tiny`, that omitting `model` and naming `tiny` both transcribe (200), and
-    that a different real model (`large-v2`) is rejected 404 (override off).
+    that naming a *different* id (`large-v2`) also transcribes 200 — the field is
+    an ignored no-op, so the launched model runs regardless.
     """
     audio = download_fixture(aws_session, AUDIO_EN, str(tmp_path / AUDIO_EN))
     with run_container_with_env(
@@ -61,27 +66,29 @@ def test_custom_default_model(image_uri, aws_session, tmp_path):
         data = resp.json()["data"]
         assert data[0]["id"] == FAST_MODEL, f"expected served id {FAST_MODEL!r}, got {data}"
 
-        # Omitted model -> resolves to the pinned default -> 200 with real text.
+        # Omitted model -> serves the launched model -> 200 with real text.
         resp = post_transcription(port, audio, response_format="json")
         assert resp.status_code == 200, resp.text
         assert resp.json().get("text", "").strip(), "expected non-empty text with default model"
 
-        # Naming the pinned model explicitly -> 200.
+        # Naming the launched model explicitly -> 200.
         resp = post_transcription(port, audio, model=FAST_MODEL, response_format="json")
         assert resp.status_code == 200, resp.text
 
-        # A different real model with override off -> 404.
+        # A different id -> still 200: the `model` field is ignored, so the
+        # launched model serves the request instead of being rejected.
         resp = post_transcription(port, audio, model="large-v2", response_format="json")
-        assert resp.status_code == 404, resp.text
+        assert resp.status_code == 200, resp.text
+        assert resp.json().get("text", "").strip(), "expected launched model to serve any model id"
 
 
 def test_served_model_alias(image_uri, aws_session, tmp_path):
-    """WHISPERX_SERVED_MODEL_NAME exposes an alias; the 404 detail names that alias.
+    """WHISPERX_SERVED_MODEL_NAME sets the id /v1/models advertises for OpenAI clients.
 
     Launches tiny behind the alias `whisper-1` and asserts /v1/models reports
-    `whisper-1`, that both the alias and the real backing id `tiny` transcribe
-    (200), and that an unknown id is rejected 404 with a detail that names the
-    served alias (not the internal `tiny`).
+    `whisper-1`, and that the alias, the real backing id `tiny`, and an unrelated
+    id all transcribe (200) — the `model` field is an ignored no-op, so every
+    request runs the launched model.
     """
     audio = download_fixture(aws_session, AUDIO_EN, str(tmp_path / AUDIO_EN))
     env = {"WHISPERX_DEFAULT_MODEL": FAST_MODEL, "WHISPERX_SERVED_MODEL_NAME": "whisper-1"}
@@ -93,52 +100,14 @@ def test_served_model_alias(image_uri, aws_session, tmp_path):
         data = resp.json()["data"]
         assert data[0]["id"] == "whisper-1", f"expected alias id 'whisper-1', got {data}"
 
-        # The alias resolves to the pinned model -> 200.
+        # The advertised alias transcribes -> 200.
         resp = post_transcription(port, audio, model="whisper-1", response_format="json")
         assert resp.status_code == 200, resp.text
 
-        # The real backing model id still resolves too -> 200.
+        # The real backing model id transcribes too -> 200.
         resp = post_transcription(port, audio, model=FAST_MODEL, response_format="json")
         assert resp.status_code == 200, resp.text
 
-        # An unknown id -> 404 whose detail advertises the served alias.
+        # An unrelated id -> still 200: the `model` field is ignored.
         resp = post_transcription(port, audio, model="whisper-large", response_format="json")
-        assert resp.status_code == 404, resp.text
-        detail = resp.json().get("detail", "")
-        assert "whisper-1" in detail, f"404 detail should name the served alias: {detail!r}"
-
-
-def test_model_override_allowed(image_uri, aws_session, tmp_path):
-    """WHISPERX_ALLOW_MODEL_OVERRIDE=true loads an unlisted model on demand.
-
-    Launches tiny with override enabled and asserts that naming a *different*
-    valid model (`base`) transcribes 200 — loaded on demand — rather than 404.
-    Kept to `base` (small, fast) so the on-demand load stays quick.
-    """
-    audio = download_fixture(aws_session, AUDIO_EN, str(tmp_path / AUDIO_EN))
-    env = {"WHISPERX_DEFAULT_MODEL": FAST_MODEL, "WHISPERX_ALLOW_MODEL_OVERRIDE": "true"}
-    with run_container_with_env(image_uri, env, device=DEVICE, flags=DOCKER_RUN_FLAGS) as c:
-        # `base` is not the pinned model, but override loads it on demand -> 200.
-        resp = post_transcription(c["port"], audio, model="base", response_format="json")
         assert resp.status_code == 200, resp.text
-        assert resp.json().get("text", "").strip(), "expected non-empty text from override model"
-
-
-def test_model_override_denied_response(image_uri, tmp_path):
-    """Override off: an unknown model yields a 404 whose body names the served id.
-
-    Asserts the error *shape* (FastAPI's {"detail": ...}), not just the status:
-    the detail must say the model "does not exist" and name the served id
-    (`tiny`). The model is resolved before the upload is read, so a present but
-    unread dummy file part is enough — no S3 audio needed.
-    """
-    dummy = tmp_path / "dummy.wav"
-    dummy.write_bytes(b"RIFF0000WAVE")
-    with run_container_with_env(
-        image_uri, {"WHISPERX_DEFAULT_MODEL": FAST_MODEL}, device=DEVICE, flags=DOCKER_RUN_FLAGS
-    ) as c:
-        resp = post_transcription(c["port"], str(dummy), model="nonexistent-model")
-        assert resp.status_code == 404, resp.text
-        detail = resp.json().get("detail", "")
-        assert "does not exist" in detail, f"expected 'does not exist' in 404 detail: {detail!r}"
-        assert FAST_MODEL in detail, f"expected served id {FAST_MODEL!r} in 404 detail: {detail!r}"

test/whisperx/ec2/test_ec2_model_config.py

@@ -23,7 +23,6 @@
 import uuid
 
 import pytest
-from botocore.exceptions import ClientError
 from sagemaker.core.resources import Endpoint, EndpointConfig, Model
 from sagemaker.core.shapes import ContainerDefinition, ProductionVariant
 from test_utils import random_suffix_name
@@ -46,11 +45,6 @@
 AUDIO_EN = "asr_en.wav"  # English ~15s
 AUDIO_DIARIZE = "asr_diarize_2spk.wav"  # 2 speakers ~21s
 
-# The model this (no-env) endpoint pins and serves: the server's DEFAULT_MODEL,
-# since WHISPERX_DEFAULT_MODEL is not overridden here. The override-denied 404
-# names this id.
-SERVED_MODEL_ID = "large-v2"
-
 
 def _cleanup(resources):
     """Best-effort delete for a list of v3 resource objects (None-safe).
@@ -227,47 +221,3 @@ def test_invocations_diarization(model_endpoint, audio_cache):
     assert speakers, f"verbose_json missing non-empty 'speakers': {body!r}"
     assert len(speakers) >= 2, f"Expected >=2 speakers, got {speakers!r}"
     LOGGER.info(f"Diarization detected {len(speakers)} speakers: {speakers}")
-
-
-def test_invocations_unknown_model_404(model_endpoint, audio_cache):
-    """An unknown `model` is rejected — the model pin holds through SageMaker invoke.
-
-    Model override is off by default, so the server validates the request `model`
-    field against the pinned served id and 404s any other value (server contract:
-    ``_resolve_model``). This proves that contract survives the SageMaker
-    /invocations hop rather than only over a local HTTP call.
-
-    Error path: sagemaker-core ``Endpoint.invoke`` calls sagemaker-runtime
-    ``invoke_endpoint`` with no try/except, so a container 4xx surfaces as a
-    ``ModelError`` — a ``botocore.exceptions.ClientError`` subclass raised as HTTP
-    424 that wraps the container's status and body. We therefore assert on the
-    raised ClientError (and, defensively, on a returned body should a runtime path
-    hand one back instead of raising). Validation short-circuits before the audio
-    is read, so no model load happens and no retry/cold-start handling is needed.
-    """
-    endpoint = model_endpoint
-    audio = audio_cache(AUDIO_EN)
-
-    # A `file` part is required by the route signature (FastAPI 422s without it),
-    # but `_resolve_model` rejects the unknown id before the upload is ever read.
-    body, content_type = _build_multipart(audio, {"model": "does-not-exist", "language": "en"})
-
-    try:
-        result = endpoint.invoke(body=body, content_type=content_type)
-    except ClientError as e:
-        err = f"{e}\n{json.dumps(e.response, default=str)}"
-        lowered = err.lower()
-        assert "does not exist" in lowered or SERVED_MODEL_ID in lowered, (
-            f"ModelError did not surface the 404 detail / served model id: {err!r}"
-        )
-        LOGGER.info(f"Unknown model correctly rejected via ClientError/ModelError: {e}")
-        return
-
-    # Fallback: some runtime paths may return the container body rather than raise.
-    raw = result.body.read()
-    text = raw.decode(errors="replace") if isinstance(raw, (bytes, bytearray)) else str(raw)
-    lowered = text.lower()
-    assert "does not exist" in lowered or SERVED_MODEL_ID in lowered, (
-        f"Expected a 404 naming the served model `{SERVED_MODEL_ID}`, got: {text!r}"
-    )
-    LOGGER.info(f"Unknown model rejected in returned body: {text[:200]!r}")