Skip to content

Commit 375bf13

Browse files
Eren-Jeager123Eren-Jeager123
committed
ci: embed immediate Slack failure notification in release pipelines
Alternative to the scheduled failure digest (#6337): add a notify-failure job to each of the 10 *.pipeline.yml orchestrators. It needs every job in the pipeline and runs on failure(), so any failed job posts to Slack immediately — no scan timing, no dedup logic, and a failed rerun re-posts. Guarded on the same release context as release-gate (main + autorelease/ dispatch-release ref + inputs.release), so PR runs never notify. Reuses the same Slack Workflow Builder payload ({count, failures}) and the same SLACK_RELEASE_FAILURE_WEBHOOK_URL secret as the digest, so the two designs are drop-in comparable. Label includes the config basename since one workflow releases several configs via matrix.
1 parent 79320df commit 375bf13

10 files changed

Lines changed: 520 additions & 0 deletions

‎.github/workflows/base.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,3 +119,55 @@ jobs:
119119
environment: ${{ needs.release-gate.outputs.environment }}
120120
aws-region: ${{ vars.AWS_REGION }}
121121
secrets: inherit
122+
123+
notify-failure:
124+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
125+
# Embedded in the pipeline so it cannot desync from the release and needs no
126+
# scan timing or dedup logic: one failed run posts exactly one message, and a
127+
# failed rerun posts again. Guarded on release context so PR runs never notify.
128+
# Reuses the same Slack Workflow Builder payload as the failure digest
129+
# ({count, failures}) so the same webhook/secret works either way.
130+
if: >-
131+
failure() &&
132+
github.ref == 'refs/heads/main' &&
133+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
134+
!contains(github.workflow_ref, '.pr') &&
135+
inputs.release
136+
needs: [build, sanity-test, security-test, release-gate, release]
137+
runs-on: ubuntu-latest
138+
steps:
139+
- name: Send Slack notification
140+
env:
141+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
142+
WORKFLOW_NAME: ${{ github.workflow }}
143+
CONFIG_FILE: ${{ inputs.config-file }}
144+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
145+
run: |
146+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
147+
echo "Slack notifications: webhook URL not configured. Skipping."
148+
exit 0
149+
fi
150+
151+
# Strip the "Auto Release - " prefix from the label; add the config
152+
# basename since one workflow can release several configs in a matrix.
153+
label="${WORKFLOW_NAME#Auto Release - }"
154+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
155+
156+
payload=$(jq -n \
157+
--arg count "1" \
158+
--arg failures "${failures}" \
159+
'{
160+
count: $count,
161+
failures: $failures
162+
}')
163+
164+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
165+
--max-time 10 \
166+
-X POST "${SLACK_WEBHOOK_URL}" \
167+
-H 'Content-Type: application/json' \
168+
--data "${payload}")
169+
170+
echo "Slack webhook responded with HTTP ${http_code}"
171+
if [[ "${http_code}" != "200" ]]; then
172+
echo "::warning::Slack notification failed (HTTP ${http_code})"
173+
fi

‎.github/workflows/huggingface-vllm.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,3 +151,55 @@ jobs:
151151
environment: ${{ needs.release-gate.outputs.environment }}
152152
aws-region: ${{ vars.AWS_REGION }}
153153
secrets: inherit
154+
155+
notify-failure:
156+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
157+
# Embedded in the pipeline so it cannot desync from the release and needs no
158+
# scan timing or dedup logic: one failed run posts exactly one message, and a
159+
# failed rerun posts again. Guarded on release context so PR runs never notify.
160+
# Reuses the same Slack Workflow Builder payload as the failure digest
161+
# ({count, failures}) so the same webhook/secret works either way.
162+
if: >-
163+
failure() &&
164+
github.ref == 'refs/heads/main' &&
165+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
166+
!contains(github.workflow_ref, '.pr') &&
167+
inputs.release
168+
needs: [build, sanity-test, security-test, telemetry-test, sagemaker-tests, release-gate, release]
169+
runs-on: ubuntu-latest
170+
steps:
171+
- name: Send Slack notification
172+
env:
173+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
174+
WORKFLOW_NAME: ${{ github.workflow }}
175+
CONFIG_FILE: ${{ inputs.config-file }}
176+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
177+
run: |
178+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
179+
echo "Slack notifications: webhook URL not configured. Skipping."
180+
exit 0
181+
fi
182+
183+
# Strip the "Auto Release - " prefix from the label; add the config
184+
# basename since one workflow can release several configs in a matrix.
185+
label="${WORKFLOW_NAME#Auto Release - }"
186+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
187+
188+
payload=$(jq -n \
189+
--arg count "1" \
190+
--arg failures "${failures}" \
191+
'{
192+
count: $count,
193+
failures: $failures
194+
}')
195+
196+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
197+
--max-time 10 \
198+
-X POST "${SLACK_WEBHOOK_URL}" \
199+
-H 'Content-Type: application/json' \
200+
--data "${payload}")
201+
202+
echo "Slack webhook responded with HTTP ${http_code}"
203+
if [[ "${http_code}" != "200" ]]; then
204+
echo "::warning::Slack notification failed (HTTP ${http_code})"
205+
fi

‎.github/workflows/lambda.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,3 +202,55 @@ jobs:
202202
environment: ${{ needs.release-gate.outputs.environment }}
203203
aws-region: ${{ vars.AWS_REGION }}
204204
secrets: inherit
205+
206+
notify-failure:
207+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
208+
# Embedded in the pipeline so it cannot desync from the release and needs no
209+
# scan timing or dedup logic: one failed run posts exactly one message, and a
210+
# failed rerun posts again. Guarded on release context so PR runs never notify.
211+
# Reuses the same Slack Workflow Builder payload as the failure digest
212+
# ({count, failures}) so the same webhook/secret works either way.
213+
if: >-
214+
failure() &&
215+
github.ref == 'refs/heads/main' &&
216+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
217+
!contains(github.workflow_ref, '.pr') &&
218+
inputs.release
219+
needs: [build, sanity-test, security-test, telemetry-test, validation-test, release-gate, release]
220+
runs-on: ubuntu-latest
221+
steps:
222+
- name: Send Slack notification
223+
env:
224+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
225+
WORKFLOW_NAME: ${{ github.workflow }}
226+
CONFIG_FILE: ${{ inputs.config-file }}
227+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
228+
run: |
229+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
230+
echo "Slack notifications: webhook URL not configured. Skipping."
231+
exit 0
232+
fi
233+
234+
# Strip the "Auto Release - " prefix from the label; add the config
235+
# basename since one workflow can release several configs in a matrix.
236+
label="${WORKFLOW_NAME#Auto Release - }"
237+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
238+
239+
payload=$(jq -n \
240+
--arg count "1" \
241+
--arg failures "${failures}" \
242+
'{
243+
count: $count,
244+
failures: $failures
245+
}')
246+
247+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
248+
--max-time 10 \
249+
-X POST "${SLACK_WEBHOOK_URL}" \
250+
-H 'Content-Type: application/json' \
251+
--data "${payload}")
252+
253+
echo "Slack webhook responded with HTTP ${http_code}"
254+
if [[ "${http_code}" != "200" ]]; then
255+
echo "::warning::Slack notification failed (HTTP ${http_code})"
256+
fi

‎.github/workflows/openfold3.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,3 +145,55 @@ jobs:
145145
environment: ${{ needs.release-gate.outputs.environment }}
146146
aws-region: ${{ vars.AWS_REGION }}
147147
secrets: inherit
148+
149+
notify-failure:
150+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
151+
# Embedded in the pipeline so it cannot desync from the release and needs no
152+
# scan timing or dedup logic: one failed run posts exactly one message, and a
153+
# failed rerun posts again. Guarded on release context so PR runs never notify.
154+
# Reuses the same Slack Workflow Builder payload as the failure digest
155+
# ({count, failures}) so the same webhook/secret works either way.
156+
if: >-
157+
failure() &&
158+
github.ref == 'refs/heads/main' &&
159+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
160+
!contains(github.workflow_ref, '.pr') &&
161+
inputs.release
162+
needs: [ci-config, build, sanity-test, security-test, sagemaker-tests, release-gate, release]
163+
runs-on: ubuntu-latest
164+
steps:
165+
- name: Send Slack notification
166+
env:
167+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
168+
WORKFLOW_NAME: ${{ github.workflow }}
169+
CONFIG_FILE: ${{ inputs.config-file }}
170+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
171+
run: |
172+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
173+
echo "Slack notifications: webhook URL not configured. Skipping."
174+
exit 0
175+
fi
176+
177+
# Strip the "Auto Release - " prefix from the label; add the config
178+
# basename since one workflow can release several configs in a matrix.
179+
label="${WORKFLOW_NAME#Auto Release - }"
180+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
181+
182+
payload=$(jq -n \
183+
--arg count "1" \
184+
--arg failures "${failures}" \
185+
'{
186+
count: $count,
187+
failures: $failures
188+
}')
189+
190+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
191+
--max-time 10 \
192+
-X POST "${SLACK_WEBHOOK_URL}" \
193+
-H 'Content-Type: application/json' \
194+
--data "${payload}")
195+
196+
echo "Slack webhook responded with HTTP ${http_code}"
197+
if [[ "${http_code}" != "200" ]]; then
198+
echo "::warning::Slack notification failed (HTTP ${http_code})"
199+
fi

‎.github/workflows/pytorch.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,3 +249,55 @@ jobs:
249249
environment: ${{ needs.release-gate.outputs.environment }}
250250
aws-region: ${{ vars.AWS_REGION }}
251251
secrets: inherit
252+
253+
notify-failure:
254+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
255+
# Embedded in the pipeline so it cannot desync from the release and needs no
256+
# scan timing or dedup logic: one failed run posts exactly one message, and a
257+
# failed rerun posts again. Guarded on release context so PR runs never notify.
258+
# Reuses the same Slack Workflow Builder payload as the failure digest
259+
# ({count, failures}) so the same webhook/secret works either way.
260+
if: >-
261+
failure() &&
262+
github.ref == 'refs/heads/main' &&
263+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
264+
!contains(github.workflow_ref, '.pr') &&
265+
inputs.release
266+
needs: [ci-config, build, sanity-test, security-test, telemetry-test, unit-test, single-gpu-test, multi-gpu-test, multi-node-gpu-test, efa-test, sagemaker-test, release-gate, release]
267+
runs-on: ubuntu-latest
268+
steps:
269+
- name: Send Slack notification
270+
env:
271+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
272+
WORKFLOW_NAME: ${{ github.workflow }}
273+
CONFIG_FILE: ${{ inputs.config-file }}
274+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
275+
run: |
276+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
277+
echo "Slack notifications: webhook URL not configured. Skipping."
278+
exit 0
279+
fi
280+
281+
# Strip the "Auto Release - " prefix from the label; add the config
282+
# basename since one workflow can release several configs in a matrix.
283+
label="${WORKFLOW_NAME#Auto Release - }"
284+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
285+
286+
payload=$(jq -n \
287+
--arg count "1" \
288+
--arg failures "${failures}" \
289+
'{
290+
count: $count,
291+
failures: $failures
292+
}')
293+
294+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
295+
--max-time 10 \
296+
-X POST "${SLACK_WEBHOOK_URL}" \
297+
-H 'Content-Type: application/json' \
298+
--data "${payload}")
299+
300+
echo "Slack webhook responded with HTTP ${http_code}"
301+
if [[ "${http_code}" != "200" ]]; then
302+
echo "::warning::Slack notification failed (HTTP ${http_code})"
303+
fi

‎.github/workflows/ray.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,3 +195,55 @@ jobs:
195195
environment: ${{ needs.release-gate.outputs.environment }}
196196
aws-region: ${{ vars.AWS_REGION }}
197197
secrets: inherit
198+
199+
notify-failure:
200+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
201+
# Embedded in the pipeline so it cannot desync from the release and needs no
202+
# scan timing or dedup logic: one failed run posts exactly one message, and a
203+
# failed rerun posts again. Guarded on release context so PR runs never notify.
204+
# Reuses the same Slack Workflow Builder payload as the failure digest
205+
# ({count, failures}) so the same webhook/secret works either way.
206+
if: >-
207+
failure() &&
208+
github.ref == 'refs/heads/main' &&
209+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
210+
!contains(github.workflow_ref, '.pr') &&
211+
inputs.release
212+
needs: [ci-config, build, sanity-test, security-test, telemetry-test, ffmpeg-test, serve-test, sagemaker-test, release-gate, release]
213+
runs-on: ubuntu-latest
214+
steps:
215+
- name: Send Slack notification
216+
env:
217+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
218+
WORKFLOW_NAME: ${{ github.workflow }}
219+
CONFIG_FILE: ${{ inputs.config-file }}
220+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
221+
run: |
222+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
223+
echo "Slack notifications: webhook URL not configured. Skipping."
224+
exit 0
225+
fi
226+
227+
# Strip the "Auto Release - " prefix from the label; add the config
228+
# basename since one workflow can release several configs in a matrix.
229+
label="${WORKFLOW_NAME#Auto Release - }"
230+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
231+
232+
payload=$(jq -n \
233+
--arg count "1" \
234+
--arg failures "${failures}" \
235+
'{
236+
count: $count,
237+
failures: $failures
238+
}')
239+
240+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
241+
--max-time 10 \
242+
-X POST "${SLACK_WEBHOOK_URL}" \
243+
-H 'Content-Type: application/json' \
244+
--data "${payload}")
245+
246+
echo "Slack webhook responded with HTTP ${http_code}"
247+
if [[ "${http_code}" != "200" ]]; then
248+
echo "::warning::Slack notification failed (HTTP ${http_code})"
249+
fi

‎.github/workflows/sglang.pipeline.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,3 +217,55 @@ jobs:
217217
environment: ${{ needs.release-gate.outputs.environment }}
218218
aws-region: ${{ vars.AWS_REGION }}
219219
secrets: inherit
220+
221+
notify-failure:
222+
# Alert Slack immediately when an auto-release (or dispatch-release) run fails.
223+
# Embedded in the pipeline so it cannot desync from the release and needs no
224+
# scan timing or dedup logic: one failed run posts exactly one message, and a
225+
# failed rerun posts again. Guarded on release context so PR runs never notify.
226+
# Reuses the same Slack Workflow Builder payload as the failure digest
227+
# ({count, failures}) so the same webhook/secret works either way.
228+
if: >-
229+
failure() &&
230+
github.ref == 'refs/heads/main' &&
231+
(contains(github.workflow_ref, '.autorelease') || contains(github.workflow_ref, '.dispatch-release')) &&
232+
!contains(github.workflow_ref, '.pr') &&
233+
inputs.release
234+
needs: [ci-config, build, sanity-test, security-test, telemetry-test, upstream-tests, model-tests, sagemaker-tests, efa-test, release-gate, release]
235+
runs-on: ubuntu-latest
236+
steps:
237+
- name: Send Slack notification
238+
env:
239+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_RELEASE_FAILURE_WEBHOOK_URL }}
240+
WORKFLOW_NAME: ${{ github.workflow }}
241+
CONFIG_FILE: ${{ inputs.config-file }}
242+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
243+
run: |
244+
if [[ -z "${SLACK_WEBHOOK_URL}" ]]; then
245+
echo "Slack notifications: webhook URL not configured. Skipping."
246+
exit 0
247+
fi
248+
249+
# Strip the "Auto Release - " prefix from the label; add the config
250+
# basename since one workflow can release several configs in a matrix.
251+
label="${WORKFLOW_NAME#Auto Release - }"
252+
failures="• ${label} ($(basename "${CONFIG_FILE}")) → ${RUN_URL}"
253+
254+
payload=$(jq -n \
255+
--arg count "1" \
256+
--arg failures "${failures}" \
257+
'{
258+
count: $count,
259+
failures: $failures
260+
}')
261+
262+
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
263+
--max-time 10 \
264+
-X POST "${SLACK_WEBHOOK_URL}" \
265+
-H 'Content-Type: application/json' \
266+
--data "${payload}")
267+
268+
echo "Slack webhook responded with HTTP ${http_code}"
269+
if [[ "${http_code}" != "200" ]]; then
270+
echo "::warning::Slack notification failed (HTTP ${http_code})"
271+
fi

0 commit comments

Comments
 (0)