ci: embed immediate Slack failure notification in release pipelines (alt to #6337)

[Eren-Jeager123]

Context

This is the embedded alternative to the scheduled failure digest in #6337, per the design discussion (G's proposal). Both target the same problem — auto-release workflows fail silently — and use the same SLACK_RELEASE_FAILURE_WEBHOOK_URL secret, so the team can compare and pick one.

What this does

Shared logic lives in a composite action, .github/actions/notify-slack-failure. Each of the 10 *.pipeline.yml orchestrators adds a thin notify-failure job that needs every job in the pipeline, runs on if: failure() && <release context>, and calls the action — so any failed job posts to Slack immediately.

Per-pipeline addition is ~20 lines (job wiring only); the curl/jq logic is defined once in the action.

Guard (same release context as release-gate):

if: >-
  failure() &&
  github.ref == 'refs/heads/main' &&
  (contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
  !contains(github.workflow_ref, '.pr') &&
  inputs.release

→ fires only for autorelease / dispatch-release runs on main; PR runs never notify.

Why embedded (vs. the #6337 cron digest)

	#6337 cron digest	This (embedded)
Latency	up to ~17h (fixed sweep times)	immediate
Scan timing risk	yes (best-effort cron)	none
Dedup logic	window-based, fragile	none — 1 failed run = 1 message
Reruns reported	maybe double / missed	each rerun re-posts
Can desync from release	possible	impossible (part of the run)
Files touched	1	1 action + 10 thin job wirings

Trade-off: emits one message per failed config (no batching), so a bad night is noisier than a digest. Given failures are ~0–2/day this is acceptable; if noise becomes real we can add a Slack-side throttle.

Payload / message

The action posts the workflow label + run URL (no count — one run per message):

{ "workflow": "PyTorch 2.12 EC2 (2.12-ec2-cuda.yml)", "url": "https://github.com/aws/deep-learning-containers/actions/runs/111" }

The label strips the Auto Release - prefix and adds the config basename (one workflow releases several configs via matrix). Rendered with a Slack template of 🚨 {{workflow}} → {{url}}:

🚨 PyTorch 2.12 EC2 (2.12-ec2-cuda.yml) → https://github.com/aws/deep-learning-containers/actions/runs/111

Configuration

webhook-url is a required action input, wired to the SLACK_RELEASE_FAILURE_WEBHOOK_URL secret. If the secret is unset the notify job fails loudly (::error:: + non-zero exit) rather than silently skipping, so a misconfiguration is visible instead of dropping alerts.

Testing

• Action + all 10 pipeline YAMLs validated.
• Payload logic simulated locally (prefix strip + config basename + jq payload).