Skip to content

Commit ad73db2

Browse files
author
Bhanu Teja Goshikonda
committed
fix(pytorch): invalidate builder-flash-attn stage cache
Adds an inline comment inside the builder-flash-attn RUN block to change its BuildKit stage hash, forcing a fresh source build against the current torch 2.13 pin. Background: earlier CI cycles on this branch cached a flash-attn wheel that was compiled against torch 2.12 (undefined symbol c10::impl::cow::materialize_cow_storage at import time on torch 2.13). The bad wheel persisted through S3 delete + ECR tag delete because BuildKit's stage cache lives across untagged image layers and the fleet host's local buildkitd cache, both of which we cannot enumerate and clean cleanly. Changing the RUN command text is the only way to guarantee BuildKit re-executes the stage. The comment is intentionally verbose and self-documenting so future readers understand its purpose and don't reflexively delete it.
1 parent 9637c9e commit ad73db2

1 file changed

Lines changed: 6 additions & 0 deletions

File tree

docker/pytorch/Dockerfile.cuda

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,12 @@ ARG MAX_JOBS
8989
# docker/pytorch/${FRAMEWORK_SHORT_VERSION}/wheels/ is created by CI (fetch_cached_wheels.sh); may not exist locally.
9090
COPY docker/pytorch/${FRAMEWORK_SHORT_VERSION}/cuda/pyproject.toml docker/pytorch/${FRAMEWORK_SHORT_VERSION}/wheel[s]/ /tmp/wheels/
9191
RUN --mount=type=cache,target=/root/.cache/uv \
92+
# Cache invalidation nudge 2026-07-14: prior BuildKit stage cache on this \
93+
# branch contained a flash-attn wheel compiled against torch 2.12 ABI \
94+
# (undefined symbol c10::impl::cow::materialize_cow_storage on torch 2.13). \
95+
# This comment changes the RUN text hash so BuildKit re-executes the stage \
96+
# against the current torch 2.13 pin. Do NOT remove until poisoned cache \
97+
# layers are GC'd out of ECR + fleet host BuildKit local caches (~30 days). \
9298
mkdir -p /tmp/built_wheels && \
9399
WHL=$(find /tmp/wheels -name "flash*attn*.whl" 2>/dev/null | head -1) && \
94100
if [ -n "${WHL}" ]; then \

0 commit comments

Comments
 (0)