Add a job summary listing promoted files to the release workflows (#8479)

The three release workflows -- `release-download-pytorch-org.yml`,
`release-stage-pypi.yml` and `release-pypi.yml` -- print their S3
transfers into a step log and nothing else. Finding out what a promotion
actually moved means expanding the step and scrolling past `--debug`
output and xtrace noise, and after a partial failure it means diffing
the bucket by hand.

This adds a job summary to each of them, listing every file the run
transferred, grouped into collapsed sections by destination directory.

Each promotion step now tees its transfer output to a log under
`${RUNNER_TEMP}`, and a new `release/summarize_promoted_files.sh` parses
the `copy:` / `upload:` / `download:` lines out of it and writes
markdown to `${GITHUB_STEP_SUMMARY}`. Parsing the transfer log rather
than listing the destination afterwards means the summary reports what
this run moved, not whatever already happened to be there.

Details worth noting:

- The summary steps are `if: always()`, because a promotion that dies
partway through is exactly when you want to know which files already
made it.
- Dry runs are labelled with a banner and still list the files, since
`--dryrun` emits `(dryrun) copy:` lines carrying the real destinations.
- Zero transferred files gets an explicit callout instead of an empty
list -- that state means the source pattern matched nothing, which
currently passes silently.
- `set -ex` became `set -exo pipefail` in the teed steps so a failing
`aws` command is not masked by `tee` succeeding.
- Progress output is carriage-return delimited, so the script converts
CRs to newlines rather than stripping them; otherwise `Completed 3
file(s)...\rcopy: ...` would hide the transfer line.

Sample output:

> ## torchvision 0.27.1 - promoted to download.pytorch.org
>
> **Dry run** -- these transfers were simulated, nothing was published.
>
> **6 file(s) transferred.**
>
> <details><summary><code>s3://pytorch/whl/cu128</code> -- 2
file(s)</summary>
>
> - torchvision-0.27.1%2Bcu128-cp311-cp311-linux_x86_64.whl
> - torchvision-0.27.1%2Bcu128-cp312-cp312-linux_x86_64.whl
>
> </details>

### Test plan

Ran the script against a synthetic log covering CR-interleaved progress
lines, `(dryrun)` prefixes, a duplicate line from a retried transfer,
and two sibling destination directories sharing a prefix (`whl/cpu` and
`whl/cpu2`) -- output groups correctly, dedupes the retry, and keeps the
siblings apart. Also verified the empty-log and no-transfers paths.

`actionlint` (with `shellcheck` enabled) passes on all three workflows
and `shellcheck` passes on the new script.

Still needs a real `workflow_dispatch` dry run of each workflow to
confirm the AWS CLI output in these containers matches what the parser
expects.

Author: atalman

.github/workflows/release-download-pytorch-org.yml

@@ -58,7 +58,7 @@ jobs:
           R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
           R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
         run: |
-            set -ex
+            set -exo pipefail
             cd release
             # Install requirements
             pip install awscli==1.32.18
@@ -97,8 +97,26 @@ jobs:
               export PACKAGE_INCLUDE_SUFFIX="-*"
             fi
 
+            # Tee the transfer output so the next step can list what moved.
             # shellcheck disable=SC2086
-            promote_s3 ${PACKAGE} whl "${!version}"
+            promote_s3 ${PACKAGE} whl "${!version}" 2>&1 | tee -a "${RUNNER_TEMP}/promote.log"
+
+      # always(): a promotion that dies partway through is exactly when you want to
+      # know which files already made it to prod.
+      - name: Summarize promoted files
+        if: always()
+        shell: bash
+        env:
+          PACKAGE: ${{ inputs.package || 'torchvision' }}
+          DRY_RUN: ${{ inputs.dryrun || 'enabled' }}
+        run: |
+            set -euo pipefail
+            source ./release/release_versions.sh
+            version="${PACKAGE^^}_VERSION"
+            ./release/summarize_promoted_files.sh \
+              "${RUNNER_TEMP}/promote.log" \
+              "${PACKAGE} ${!version} - promoted to download.pytorch.org" \
+              "${DRY_RUN}"
 
       - name: Recompute SHA256 on whl/ (prod) for promoted package
         if: ${{ inputs.dryrun == 'disabled' }}

.github/workflows/release-pypi.yml

@@ -53,17 +53,37 @@ jobs:
           PACKAGE: ${{ inputs.package || 'torchvision' }}
           DRY_RUN: ${{ inputs.dryrun || 'enabled' }}
         run: |
-            set -ex
+            set -exo pipefail
 
             # Init release versions variables
             source ./release/release_versions.sh
             # shellcheck disable=SC2086
             version="${PACKAGE^^}_VERSION"
             mkdir dist/
+            # Only stdout is teed; --debug goes to stderr and stays in the job log.
             # shellcheck disable=SC2086
-            aws s3 sync "s3://pytorch-backup/${PACKAGE}-${!version}-pypi-staging/" dist/ --debug
+            aws s3 sync "s3://pytorch-backup/${PACKAGE}-${!version}-pypi-staging/" dist/ --debug \
+              | tee -a "${RUNNER_TEMP}/download.log"
       - name: Display structure of downloaded files
         run: ls -R dist/
+
+      # Runs before the publish step, so the title says what will be published
+      # rather than asserting that it was.
+      - name: Summarize files to publish
+        if: always()
+        shell: bash
+        env:
+          PACKAGE: ${{ inputs.package || 'torchvision' }}
+          DRY_RUN: ${{ inputs.dryrun || 'enabled' }}
+        run: |
+            set -euo pipefail
+            source ./release/release_versions.sh
+            version="${PACKAGE^^}_VERSION"
+            ./release/summarize_promoted_files.sh \
+              "${RUNNER_TEMP}/download.log" \
+              "${PACKAGE} ${!version} - files to publish to PyPI" \
+              "${DRY_RUN}"
+
       - name: Publish package to PyPI
         if: ${{ inputs.dryrun == 'disabled' }}
         uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/release-stage-pypi.yml

@@ -46,7 +46,7 @@ jobs:
           PACKAGE: ${{ inputs.package || 'torchvision' }}
           DRY_RUN: ${{ inputs.dryrun || 'enabled' }}
         run: |
-            set -ex
+            set -exo pipefail
             cd release/pypi
 
             # Default Version to promote
@@ -66,10 +66,12 @@ jobs:
               local promote_version
               promote_version=$2
               echo "=-=-=-= Promoting ${package_name}'s v${promote_version} to pypi staging' =-=-=-="
+              # Tee every platform's transfers into one log so the summary step can
+              # list them all together.
               (
                   set -x
                   PACKAGE_VERSION="${promote_version}" PACKAGE_NAME="${package_name}" DRY_RUN="${DRY_RUN}" bash ./upload_pypi_to_staging.sh
-              )
+              ) 2>&1 | tee -a "${RUNNER_TEMP}/stage.log"
               echo
             }
 
@@ -118,3 +120,20 @@ jobs:
               # shellcheck disable=SC2086
               PLATFORM="win_amd64"              VERSION_SUFFIX="${CPU_VERSION_SUFFIX}"                      upload_pypi_to_staging ${PACKAGE} "${!version}"
             fi
+
+      # always(): staging fans out over several platforms, so a mid-run failure
+      # still leaves a partial set in the staging bucket worth listing.
+      - name: Summarize staged files
+        if: always()
+        shell: bash
+        env:
+          PACKAGE: ${{ inputs.package || 'torchvision' }}
+          DRY_RUN: ${{ inputs.dryrun || 'enabled' }}
+        run: |
+            set -euo pipefail
+            source ./release/release_versions.sh
+            version="${PACKAGE^^}_VERSION"
+            ./release/summarize_promoted_files.sh \
+              "${RUNNER_TEMP}/stage.log" \
+              "${PACKAGE} ${!version} - staged for PyPI" \
+              "${DRY_RUN}"

release/summarize_promoted_files.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Write a GitHub Actions job summary listing the files a release step transferred.
+#
+# The release workflows all drive `aws s3 cp` / `aws s3 sync`, which print one line
+# per object:
+#
+#   copy: s3://pytorch/whl/test/cu128/foo.whl to s3://pytorch/whl/cu128/foo.whl
+#   (dryrun) copy: s3://... to s3://...
+#   upload: ./foo.whl to s3://pytorch-backup/foo-1.0-pypi-staging/foo.whl
+#   download: s3://pytorch-backup/... to dist/foo.whl
+#
+# Capture that output to a log and pass it here. Reading the transfer log rather
+# than listing the destination afterwards means the summary reports what this run
+# actually moved, not whatever already happened to be there.
+#
+# Usage: summarize_promoted_files.sh <logfile> <title> [dry_run]
+#   dry_run: "enabled" (default) or "disabled" -- only controls the banner.
+
+set -euo pipefail
+
+LOG_FILE=${1:?usage: summarize_promoted_files.sh <logfile> <title> [dry_run]}
+TITLE=${2:?usage: summarize_promoted_files.sh <logfile> <title> [dry_run]}
+DRY_RUN=${3:-enabled}
+
+# Fall back to stdout when run outside Actions, so the script stays testable.
+SUMMARY_FILE=${GITHUB_STEP_SUMMARY:-/dev/stdout}
+
+if [[ ! -s "${LOG_FILE}" ]]; then
+    {
+        echo "## ${TITLE}"
+        echo
+        echo "No transfer log was produced, so there is nothing to report."
+    } >>"${SUMMARY_FILE}"
+    exit 0
+fi
+
+# The destination of a transfer is whatever follows the last " to ". Progress
+# output is carriage-return delimited, so turn CRs into newlines (rather than
+# deleting them) to keep the "copy:" prefix at the start of its own line.
+# sort -u because a retried transfer logs the same object twice.
+DESTS=$(tr '\r' '\n' <"${LOG_FILE}" \
+    | grep -E '^(\(dryrun\) )?(copy|upload|download|move): ' \
+    | sed -E -e 's/.* to (.*)$/\1/' -e 's/[[:space:]]*$//' \
+    | grep -v '^$' \
+    | sort -u || true)
+
+COUNT=$(printf '%s' "${DESTS}" | grep -c '^' || true)
+
+{
+    echo "## ${TITLE}"
+    echo
+    if [[ "${DRY_RUN}" != "disabled" ]]; then
+        echo "> **Dry run** -- these transfers were simulated, nothing was published."
+        echo
+    fi
+
+    if [[ "${COUNT}" -eq 0 ]]; then
+        echo "**No files were transferred.**"
+        echo
+        echo "The commands ran but moved nothing, which usually means the source"
+        echo "pattern matched no objects. Check the package version and channel"
+        echo "before treating this run as a success."
+    else
+        echo "**${COUNT} file(s) transferred.**"
+        echo
+
+        # Group by destination directory: a multi-accelerator promotion then reads
+        # as a handful of collapsed buckets rather than one flat list of ~200 wheels.
+        printf '%s\n' "${DESTS}" | sed -E 's:/[^/]+$::' | sort -u | while read -r dir; do
+            if [[ -z "${dir}" ]]; then
+                continue
+            fi
+            files=$(printf '%s\n' "${DESTS}" | awk -v d="${dir}/" '
+                index($0, d) == 1 {
+                    rest = substr($0, length(d) + 1)
+                    if (index(rest, "/") == 0) { print rest }
+                }')
+            n=$(printf '%s' "${files}" | grep -c '^' || true)
+            echo "<details><summary><code>${dir}</code> -- ${n} file(s)</summary>"
+            echo
+            printf '%s\n' "${files}" | sed 's/^/- /'
+            echo
+            echo "</details>"
+        done
+    fi
+} >>"${SUMMARY_FILE}"