daily: notify #crash-bot-ci on workflow failure

Yesterday's 403 (NJSP feed) and today's GH-API 429 (crash-log walker) both
silently failed — there was no whole-pipeline failure notification, only
the narrow `summaries.dvc` PDF/XML-mismatch hook from `notify.py`. Both
cron-scheduled runs went unnoticed, missing a day of NJSP refresh.

Add a final `if: failure()` step that delegates to
`.github/scripts/notify-ci-failure.sh`:

- Looks up which step failed via `gh run view --json jobs --jq …`
- Posts ":x: *Daily pipeline failed* at \`<step>\` (<branch> <sha> <subject>,
  event=<schedule|workflow_dispatch>)" with a "view run" link
- No-op (prints to stdout instead of raising) when `SLACK_CI_CHANNEL_ID` or
  `SLACK_BOT_TOKEN` isn't set, so dev/test runs stay quiet
- Surfaces Slack API failures via `::warning::` annotation rather than
  blocking — the pipeline already failed; we don't want notification
  failure to mask the real cause

Pure-bash so it still runs even when the failure is in early dependency-
install steps (before `.venv/bin` lands on `PATH`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: ryan-williams

.github/scripts/notify-ci-failure.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+# Post a CI-failure summary to Slack channel `$SLACK_CI_CHANNEL_ID`.
+#
+# Wired into `.github/workflows/daily.yml` as the final step under
+# `if: failure()`. Pure-bash so it still runs even when the failure is in
+# Python / dependency-install steps before `.venv/bin` is on PATH.
+#
+# Required env (set by the GHA shell automatically unless noted):
+#   GITHUB_RUN_ID        — for `gh run view` lookup of the failed step
+#   GITHUB_SHA           — commit SHA
+#   GITHUB_REF_NAME      — branch
+#   GITHUB_SERVER_URL    — e.g. https://github.com
+#   GITHUB_REPOSITORY    — owner/repo
+#   GH_TOKEN             — must be passed in step env (gh CLI auth)
+#   EVENT_NAME           — caller-set; the workflow's event (schedule / workflow_dispatch / …)
+#   SLACK_BOT_TOKEN, SLACK_CI_CHANNEL_ID
+#                        — Slack post target; if either is unset the script
+#                          prints the would-be message and exits 0 so the
+#                          underlying CI failure remains the loud signal.
+#
+# Slack post failures are logged as a `::warning::` GHA annotation rather
+# than re-raising — the pipeline already failed and we don't want a notify
+# bug to mask the real cause.
+
+set -euo pipefail
+
+run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+
+failed_step=$(gh run view "${GITHUB_RUN_ID}" --json jobs \
+    --jq '.jobs[].steps[] | select(.conclusion == "failure") | .name' \
+    | head -1)
+failed_step="${failed_step:-unknown step}"
+short_sha="${GITHUB_SHA:0:7}"
+commit_subject=$(git log -1 --format=%s "${GITHUB_SHA}" 2>/dev/null || echo '(unavailable)')
+
+text=":x: *Daily pipeline failed* at \`${failed_step}\` (\`${GITHUB_REF_NAME}\` ${short_sha} _${commit_subject}_, event=${EVENT_NAME:-?})"$'\n'"<${run_url}|view run>"
+
+if [[ -z "${SLACK_CI_CHANNEL_ID:-}" || -z "${SLACK_BOT_TOKEN:-}" ]]; then
+    echo "SLACK_CI_CHANNEL_ID or SLACK_BOT_TOKEN unset; would-be message:"
+    printf '%s\n' "$text"
+    exit 0
+fi
+
+payload=$(jq -nc \
+    --arg channel "${SLACK_CI_CHANNEL_ID}" \
+    --arg text "$text" \
+    '{channel: $channel, text: $text, unfurl_links: false, unfurl_media: false}')
+
+resp=$(curl -sS -X POST https://slack.com/api/chat.postMessage \
+    -H "Authorization: Bearer ${SLACK_BOT_TOKEN}" \
+    -H 'Content-Type: application/json; charset=utf-8' \
+    -d "$payload")
+echo "Slack response: $resp"
+
+ok=$(printf '%s' "$resp" | jq -r '.ok // false')
+if [[ "$ok" != "true" ]]; then
+    echo "::warning::Slack CI notification failed: $resp"
+fi

.github/workflows/daily.yml

@@ -151,3 +151,9 @@ jobs:
       - name: Run custom targets
         if: ${{ inputs.targets }}
         run: $DVX ${{ inputs.targets }}
+      - name: "Notify #crash-bot-ci on failure"
+        if: failure()
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+          EVENT_NAME: ${{ github.event_name }}
+        run: bash .github/scripts/notify-ci-failure.sh