feat(pytorch): add PyTorch 2.13 currency scaffold#6365
Open
bhanutejagk wants to merge 32 commits into
Open
Conversation
Add DLC image configs, workflow callers, and release schedule entries
for PyTorch 2.13.0 on cu130 (Amazon Linux 2023). Covers ec2 and
sagemaker variants across cuda and cpu devices.
- docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock} pinned to
torch 2.13.0, torchvision 0.28.0, torchaudio 2.11.0 (held; upstream
audio has no 2.12/2.13 tag yet)
- 4 image configs and 4 workflow callers (pr + autorelease x ec2/sagemaker)
- autorelease crons: 30 19 UTC ec2, 00 23 UTC sagemaker on Tue/Thu
Align 2.13 currency image with newer upstream releases that were
available when scaffolding but held back from the initial 2.12-copy.
- nccl 2.26.2 -> 2.30.7-1: matches torch 2.13's bundled NCCL pin
(torch 2.13 requires source builds to use NCCL >= 2.23 and pins its
bundled runtime NCCL to 2.30)
- deepspeed 0.18.8 -> 0.19.2: 0.19.x train carries the torch >= 2.11
compatibility patches; 0.18.8 predates them
- transformer_engine 2.12.0 -> 2.16.1: security patch + cuDNN 9.18.1
requirement in the latest stable train (2.12 is 4 months old)
- efa 1.47.0 -> 1.49.0: latest EFA installer (libfabric 2.4.0amzn5.0,
efa driver 3.1.0, OFI NCCL Plugin 1.20.0)
- gdrcopy 2.4.4 -> 2.6: latest stable; new optional DMA-BUF backend
requires cuda driver 13.3+ but falls back to gdrdrv (current path)
- pyproject deepspeed==0.18.8 -> 0.19.2 mirrors the config bump
- uv.lock regenerated in 2.13/{cuda,cpu}
cryptography>=49.0.0 bump was attempted but reverted: mlflow>=3.9.0
caps cryptography<49 in the sagemaker extra. Revisit when mlflow's
upper cap loosens.
CPU configs unchanged (no gpu-only pins to bump).
The wheel-cache upload path in scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs `docker buildx build --target wheel-export`, which re-executes the entire Dockerfile to reach a scratch stage that COPYs the built wheels out of the flash-attn and TE builder stages. With MAX_JOBS=8, the concurrent flash-attn 2.8.3 + transformer-engine 2.16.1 source builds during that re-execution exceed the CI runner cgroup memory limit and get OOM-killed (SIGKILL / exit code 137, PR #6365 attempt 2). Halving MAX_JOBS to 4 roughly halves peak compile RAM, keeping the re-execution under the runner ceiling at the cost of a slower flash-attn / TE compile step. CPU configs unaffected.
bhanutejagk
force-pushed
the
pytorch-2.13-currency
branch
from
51e3d4c to
eb84c58
Compare
added 24 commits
July 9, 2026 13:41
Revert the temporary max_jobs=8->4 workaround (eb84c58). Root cause of the wheel-export OOM is x86-build-runner is CPU-sized and can't handle the concurrent flash-attn + TE source recompile that `docker buildx build --target wheel-export` triggers. Route pytorch GPU builds (device_type=gpu) to x86-vllm-build-runner, the existing GPU-compilation fleet already used by vllm and vllm-omni. CPU builds stay on x86-build-runner. No new CDK / infra changes needed; the fleet is already provisioned with GPU-compilation-appropriate RAM.
The dedicated x86-pytorch-build-runner fleet is defined in DLContainersInfraCDK mainline (commit 85a06e065, rebalanced by f4bcbc9c on 2026-06-30) - identical hardware to x86-vllm-build-runner and x86-sglang-build-runner, but reserved for pytorch. Route all pytorch builds (both CPU and GPU) to the dedicated fleet unconditionally, matching the vllm.pipeline.yml and sglang.pipeline.yml pattern. This replaces the temporary conditional that routed GPU builds to x86-vllm-build-runner as a stopgap. Cache locality on a dedicated fleet is expected to fix the wheel-export OOM: BuildKit finds the primary build's flash-attn + TE layers still cached on the same host, so the wheel-export re-run reduces to a COPY --from=<cached-layer> instead of a concurrent recompile. - pytorch.pipeline.yml: unconditional fleet:x86-pytorch-build-runner - ci-architecture.md: document the new fleet row
PR-6365 ECR vulnerability scan flagged 2 HIGH CVEs on 2.13 CPU images (also on SageMaker CPU variant). Both are unbounded-memory / catastrophic-backtracking bugs in soupsieve's CSS selector parser, fixed upstream in 2.8.4. soupsieve is transitive via beautifulsoup4 -> python-fasthtml -> fastai. Add an explicit direct pin `soupsieve>=2.8.4` to both cpu and cuda pyproject.toml (following prior transitive-CVE bump convention: msgpack, tornado, cryptography, aiohttp), regenerate uv.lock via `uv lock`. CVEs cleared: - GHSA-2wc2-fm75-p42x (unbounded memory on large selector lists) - GHSA-836r-79rf-4m37 (regex catastrophic backtracking on attribute selectors) Upstream fix pin: soupsieve>=2.8.4
The wheel-cache upload path re-runs `docker buildx build --target wheel-export` to extract compiled wheels from the wheel-export stage. Because that re-invocation passed NO --build-arg while the primary build passes ~15 of them, BuildKit computed a different cache key for every stage and had to recompile flash-attn + TE from source concurrently, OOM-killing the CodeBuild runner (exit 137 at upload_wheels.sh:45). Forward the same --build-args the primary build uses (read from EXTRA_BUILD_ARGS env var, the space-separated list of arg names the build-image action already computes via resolve_build_args.py). The wheel-export stage now hits BuildKit's local cache for every prior stage and reduces to a COPY --from=<cached-layer>, avoiding the recompile entirely.
Reverts commit 2705344. That change forwarded primary-build --build-args to the wheel-export re-invocation to make BuildKit cache-hit and avoid recompiling flash-attn + TE. No longer needed: the x86-pytorch-build-runner fleet has been bumped from m6a.8xlarge (128 GB) to m6a.12xlarge (192 GB) via DLContainersInfraCDK CR CR-288126860. The runner now has enough memory headroom to complete the cold-cache wheel-export recompile without OOM. Reverting keeps upload_wheels.sh in its upstream shape for other frameworks that share the pattern (though currently only pytorch_runtime uses this exact copy).
Starting with EFA 1.48, the installer auto-detects NGC containers via /opt/nvidia/nvidia_entrypoint.sh (present in nvidia/cuda:*-amzn2023 base images) and skips installing the AL2023 libnccl-ofi RPM. This left the pytorch runtime image without libnccl-net.so, breaking every unit / sanity / single-gpu / efa test that asserts the OFI plugin is present. The sibling scripts/docker/common/install_efa_amzn2023.sh already has this fix (lines 37-48); mirror the version-gated --disable-ngc flag here so aws-ofi-nccl installs regardless of the base-image NGC detection. Only affects pytorch - sglang, vllm, and vllm-omni use the common installer via their own paths.
Both fetch_wheels.sh and upload_wheels.sh keyed the S3 wheel cache
by ${PKG}/cu${MAJOR}${MINOR}/${FNAME}, omitting the torch version the
wheel was compiled against. Since pytorch 2.12 and 2.13 both pin
flash_attn 2.8.3 + cuda 13.0.2, the 2.13 build fetched 2.12's cached
wheel and loaded it into torch 2.13 - producing an undefined symbol
at import (c10::impl::cow::materialize_cow_storage, removed in
torch 2.13).
Key by torch major.minor as well, so cross-torch-version cache reuse
is impossible by construction. transformer-engine-torch would hit the
same bug at the next torch bump.
Note: stale cu130/torch212 wheels remain in s3://dlc-cicd-wheels/
under the old key format. They're now orphaned and unreferenced -
safe to delete via a separate S3 cleanup, or leave as is.
TE 2.16.1 calls cublasLtGroupedMatrixLayoutInit_internal which requires libcublasLt.so.13.3, but our runtime base image (nvidia/cuda:13.0.2- runtime-amzn2023) ships an older cuBLASLt. The build succeeds against devel headers but fails to load at runtime with: ImportError: /path/libtransformer_engine.so: undefined symbol cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13 TE 2.13.0 predates the cuBLAS 13.3 dependency (grouped GEMM with on-device group sizes landed in TE 2.14 via NVIDIA/TransformerEngine #2748) and is torch-2.13 compatible. Bump base CUDA to 13.3.x-amzn2023 in a follow-up once that image is available on Docker Hub.
Reverts commit e1c8715. That commit rolled TE back to 2.13.0 based on deep-analysis that assumed the cuBLAS 13.3 grouped-GEMM dependency was introduced in TE 2.14+. That analysis was wrong: NVIDIA's PyPI distribution transformer_engine_cu13-2.13.0 is a PRECOMPILED wheel built against cuBLAS 13.3, so the undefined symbol persists on 2.13.0 too. Follow-up commits pin TE to the latest release + force source build against our cuBLAS 13.0, which excludes the grouped-GEMM code path via the compile-time #if CUBLAS_VERSION >= 130300 guard.
Restores TE currency to the latest release. Combined with the source- build change in the next commit, this uses NVIDIA's newest features while avoiding the cuBLAS 13.3 binary dep that ships in their PyPI wheels. Compat verified: torch 2.13 (declared torch>=2.1 in build_tools/ pytorch.py), CUDA 13.0 build-time headers (source guards on runtime symbols), cuDNN 9.x bundled with torch wheel.
nvidia/cuda:13.3.0-{base,devel,runtime}-amzn2023 are now published on
Docker Hub. Bumping the base ships cuBLAS 13.3 in the runtime, which
resolves the undefined-symbol import error on transformer_engine 2.17.0:
OSError: libtransformer_engine.so: undefined symbol:
cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13
The symbol was added in cuBLAS 13.3. NVIDIA's precompiled
transformer_engine_cu13-2.17.0 wheel expects it, but our previous base
(nvidia/cuda:13.0.2-runtime-amzn2023) shipped cuBLAS 13.0.
This replaces the source-build workaround we were considering
(git+https install of TE with --no-build-isolation). Now we can consume
NVIDIA's precompiled wheel directly and stay current on TE.
Also updates prod_image tag from cu130 -> cu133 to match the new base.
Both CVEs are actively exploited per NVD: - CVE-2026-7246 (click) - fixed in 8.3.3 - CVE-2026-55379 (pillow) - fixed in 12.3.0 Follows the same transitive-pin pattern as commit 7f3372f (soupsieve).
… errors Two related fixes to the CUDA 13.3 rebuild path: 1. NCCL headers changed shape in the torch 2.13 wheel - the nccl_device/ subdirectory (containing coop.h) was missed by the previous flat `cp *.h` glob. Copy the include tree recursively so TE's source build can find <nccl_device/coop.h>. 2. Remove the `|| true` at the end of the TE install RUN block. It was masking source-build compile failures - TE would silently be missing from the runtime image and only surface as `ModuleNotFoundError` in downstream tests. Fail loudly at build time.
nvidia-container-toolkit 1.19.1 (current AWS DLAMI version) has a stale semver parser and compat table that reject `com.nvidia.cuda.requires=cuda>=13.3` even though the runner's actual driver (580.159.04) exceeds CUDA 13.3's declared minimum (580.65). Container prestart hook errors with "Auto-detected mode as 'legacy'" and single-gpu tests never run. Upstream nvidia-container-toolkit v1.20.0-rc.1 (2026-07-09) contains the fix (Masterminds/semver parser + updated compat tables), but no AWS AMI has consumed the new version yet. Bypass with NVIDIA_DISABLE_REQUIRE=true until the DLAMI ships toolkit >=1.20. The bypass only skips the label version-check; all other container startup logic (GPU mount, driver library injection) remains active. Revert this line once the DLAMI updates.
Commit 51a936c removed || true broadly to stop masking TE build failures. But one `cp` in that RUN block is legitimately best-effort: the pip wheel-cache save-off at the tail of the source-build branch. TE compile and install were succeeding on 2.13-ec2-cuda in PR CI run 29302027689, but the trailing `cp /root/.cache/pip/wheels/*/*.whl /tmp/built_wheels/` returned non-zero (glob mismatch or empty cache dir) and failed the whole Dockerfile stage. Restore || true only on that specific cache-save line. Actual TE build failures still fail the build loudly — that fix from 51a936c stays intact.
Adds an inline TODO comment near the bypass added in 30c2a47 so future maintainers can identify when and how to remove it without spelunking commit history.
The wheel-cache entry at s3://dlc-cicd-wheels/wheels/flash_attn/cu133/torch213/ got poisoned during earlier CI iterations on this PR — it contains a flash-attn wheel compiled against torch 2.12 ABI, causing `undefined symbol: c10::impl::cow::materialize_cow_storage` at runtime in every 2.13 GPU test since (run 29302504659, jobs 86997450461 etc). Adds a `build.skip_wheel_cache` config knob honored by pre_build.sh — when set, forces a source build regardless of what's in S3. Enables it on the two 2.13 GPU configs for one CI cycle. Source-built wheels then overwrite the poisoned S3 entries via post_build.sh's normal upload path. Follow-up commit removes the skip_wheel_cache flags after CI is green so subsequent builds hit the freshly-populated (clean) cache. The pre_build.sh mechanism stays as reusable infrastructure — future cache-corruption incidents can be resolved by flipping the same flag.
…onfigs Reverts the YAML additions from c0e3285 (skip_wheel_cache: "true" on 2.13-{ec2,sagemaker}-cuda.yml). The wheel cache at cu133/torch213/ has been refreshed with valid torch-2.13-ABI wheels via one source- build cycle. Re-enable normal cache fetch for subsequent builds. The pre_build.sh SKIP_WHEEL_CACHE mechanism itself stays as reusable infrastructure — future cache-corruption incidents can be resolved by flipping the same flag.
Restores the SKIP_WHEEL_CACHE guard in pre_build.sh (previously reverted in cbca116) and extends it: when a config sets skip_wheel_cache:"true", we now also export FORCE_WHEEL_CACHE_UPLOAD=true so upload_wheels.sh knows to overwrite any existing S3 wheel at the target key. Needed to replace a poisoned flash-attn wheel at cu133/torch213/ that the prior skip-only mechanism couldn't dislodge (upload was blocked by "already cached" guard, leaving S3 untouched).
When FORCE_WHEEL_CACHE_UPLOAD=true (set by pre_build.sh when skip_wheel_cache is enabled on a config), upload_wheels.sh overwrites any existing S3 wheel at the target key instead of skipping. Used to replace poisoned cache entries with freshly source-built wheels.
Forces one CI cycle to source-build flash-attn (and TE) then overwrite the poisoned S3 cache entry at wheels/flash_attn/cu133/torch213/. The existing wheel is torch-2.12 ABI (confirmed via undefined-symbol at import: c10::impl::cow::materialize_cow_storage), preventing 2.13 GPU tests from running. Follow-up commit removes this flag after CI is green so subsequent builds hit the freshly overwritten (clean) cache.
test_import_gpu[transformer_engine] fails on the CPU-only default-runner that unit-test runs on because TE 2.17 dlopens libcuda.so.1 at import time; libcuda is only mounted into containers via nvidia-container-cli when a GPU is present. The test is correctly gated with @pytest.mark.skipif(not IS_CUDA) — that gate is about the IMAGE variant (CUDA vs CPU), not about the test runner's GPU availability. Moving both parametrized cases (flash_attn, transformer_engine) into single_gpu/ where the workflow runs on x86-g6xl-runner with --gpus all gives them a real driver mount at import time. flash_attn passed on the CPU fleet only because it defers libcuda dlopen until the first CUDA call — the import itself succeeded. Moving it too keeps GPU-only import checks in a single suite and avoids relying on that lazy-loading behavior. The IS_CUDA skip is preserved as a belt-and-suspenders guard for any future CPU-variant coverage that reuses the single_gpu suite. Unit suite retains COMMON_PACKAGES (torch, torchvision, torchaudio, deepspeed, boto3, requests, yaml, packaging) — none of which need a GPU driver at import.
…onfigs The wheel cache at wheels/flash_attn/cu133/torch213/ and wheels/transformer_engine_torch/cu133/torch213/ has been overwritten with clean torch-2.13-ABI wheels via the FORCE_WHEEL_CACHE_UPLOAD mechanism (a67ea66 + b5b0a8d). PR #6365 2.13 unit-test and single-gpu-test both pass. Re-enable normal cache fetch for subsequent builds.
added 5 commits
July 14, 2026 17:37
Reverts b5b0a8d. Cache poisoning has been resolved for this incident; keep upload_wheels.sh minimal without the force-overwrite escape hatch. Future incidents can restore this mechanism on-demand.
Reverts a67ea66. Cache poisoning has been resolved; keep pre_build.sh minimal. Future incidents can restore this mechanism on-demand.
Empty commit. S3 wheel cache and ECR image tags for 2.13 GPU builds were manually cleaned up to recover from a poisoned flash-attn wheel (torch-2.12 ABI at a torch213-labeled S3 key). This commit forces a fresh workflow run so build -> test dependencies resolve in order rather than being racing rerun-failed re-queues.
Adds an inline comment inside the builder-flash-attn RUN block to change its BuildKit stage hash, forcing a fresh source build against the current torch 2.13 pin. Background: earlier CI cycles on this branch cached a flash-attn wheel that was compiled against torch 2.12 (undefined symbol c10::impl::cow::materialize_cow_storage at import time on torch 2.13). The bad wheel persisted through S3 delete + ECR tag delete because BuildKit's stage cache lives across untagged image layers and the fleet host's local buildkitd cache, both of which we cannot enumerate and clean cleanly. Changing the RUN command text is the only way to guarantee BuildKit re-executes the stage. The comment is intentionally verbose and self-documenting so future readers understand its purpose and don't reflexively delete it.
…n stage cache The `#` comment block added in ad73db2 didn't invalidate BuildKit's stage cache — Docker's Dockerfile parser strips `#` comment lines from RUN commands before hashing, so the RUN command text was unchanged and BuildKit continued serving the poisoned wheel from cache (evidenced by 18s cached build on run 29384147865). Replace the inline comment with a `: "string" && \` shell no-op — a real POSIX shell command that Docker's parser MUST preserve. The `:` builtin evaluates its args and returns success (zero runtime cost). BuildKit sees new RUN text → cache miss → re-executes stage against current torch 2.13 pin. The explanatory comment is moved ABOVE the RUN (Dockerfile-level, not shell-level), so future readers understand why the no-op line exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PyTorch 2.13 currency
Ships the PyTorch 2.13.0 DLC image (EC2 and SageMaker variants, GPU + CPU) as part of the ongoing currency cadence.
What's shipped
Infrastructure
x86-pytorch-build-runnerCodeBuild fleet (m6a.12xlarge, per an internal infra CR)--disable-ngcon 1.48+ (required to install libnccl-net.so on NGC base images)Known tech debt
NVIDIA_DISABLE_REQUIRE=truebypass onpytorch.tests-single-gpu.yml: works around a stalenvidia-container-toolkit(1.19.1) that rejects thecom.nvidia.cuda.requires=cuda>=13.3OCI label. The runner's actual driver (580.159.04) exceeds CUDA 13.3's declared minimum (580.65), so the check is a false positive. Fix landed in nvidia-container-toolkit v1.20.0-rc.1 (2026-07-09); waiting on the upstream base AMI to consume the newer toolkit. Bypass will be reverted in a follow-up PR once the base image updates.Testing
CI on this PR exercises: unit-test, sanity-test, single-gpu-test, telemetry-test, sagemaker-test, security-test (ECR CVE scan).