Add Claude triage step to canary ferry workflows

On scheduled canary failures (TPU and GPU), Claude now runs before
cluster teardown to diagnose the failure, file a GitHub issue, and
produce a Slack summary that replaces the static one-liner.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: yoblin

.agents/skills/canary-triage/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: canary-triage
+description: Triage a failed canary ferry run. Gather diagnostics, identify root cause, file a GitHub issue, and write a Slack summary. Used by CI on scheduled canary failures.
+---
+
+# Skill: Canary Triage
+
+Triage a failed canary ferry run. Diagnose root cause, file a GitHub issue,
+write a Slack summary. Diagnosis and reporting only — no code changes, no PRs.
+
+## Inputs (environment variables)
+
+| Variable | Description |
+|---|---|
+| `CANARY_LANE` | `gpu` (CoreWeave) or `tpu` (GCP) |
+| `CANARY_JOB_ID` | Iris job ID |
+| `CANARY_RUN_ID` | W&B run ID |
+| `IRIS_CONFIG` | Path to Iris cluster config |
+| `IRIS_NAMESPACE` | Kubernetes namespace (CW only) |
+| `WANDB_ENTITY` | W&B entity |
+| `WANDB_PROJECT` | W&B project |
+| `GHA_RUN_URL` | Full URL to the GitHub Actions run |
+
+## Steps
+
+### 1. Gather diagnostics
+
+The cluster is still live. Collect signal now — it will be torn down after you.
+
+- Iris job state via `.venv/bin/iris --config=$IRIS_CONFIG job list --json`
+- **GPU lane:** you have kubectl at `~/.kube/coreweave-iris`, namespace `$IRIS_NAMESPACE`.
+  Get pod status, controller logs, task pod logs, warning events, pod describe.
+- **TPU lane:** use `iris process logs` and `iris job list`.
+- Re-run `scripts/canary/validate_canary_metrics.py` if you need the validation output.
+
+### 2. Identify root cause
+
+Classify into one of: **infra/scheduling**, **training crash**, **metric regression**,
+**controller bug**, **data/storage**.
+
+Use hypothesis-driven diagnosis: state hypothesis, gather evidence, narrow. 
+Attempt to reproduce the issue locally and minimally.
+Triple check that you're narrowing down on the same issue as the one that actually broke the canary.
+
+### 3. File a GitHub issue
+
+Follow the `file-issue` skill. Use the bug-report template.
+
+- **Title:** `[canary-{lane}] {short failure description}`
+- **Labels:** `bug`, `agent-generated`, `canary`
+- **Body must include** a "Canary run context" section with: lane, job ID,
+  GHA run URL, W&B run URL, date.
+- Support your claims using supporting data (e.g. runtime logs)
+- Keep the issue concise and maximally readable for humans. 
+- Use GFM to make the details (e.g. log traces, code to reproduce issue) optional and declutter the issue.
+- Use `--body-file` with a temp file (see `file-issue` skill for the pattern).
+
+### 4. Write `slack_message.md`
+
+Write to the repo root. The workflow reads this file and sends it to Slack.
+Always write this file, even if issue creation failed.
+
+Format — keep to 4 lines max:
+
+```
+:red_circle: *{GPU|TPU} Canary failed* — {one-line summary}
+*Root cause:* {category} — {1 sentence}
+*Issue:* {github issue URL}
+*GHA run:* {GHA_RUN_URL}
+```
+
+If root cause is unclear, say so: `root cause unclear` with your best-guess signals.
+

.github/workflows/marin-canary-ferry-cw.yaml

@@ -17,6 +17,8 @@ on:
 permissions:
   contents: read   # actions/checkout
   packages: write  # docker login ghcr.io for iris cluster start
+  issues: write    # claude triage files issues
+  id-token: write  # claude-code-action OIDC
 
 jobs:
   canary-ferry-cw:
@@ -182,6 +184,30 @@ jobs:
           kubectl --kubeconfig ~/.kube/coreweave-iris -n ${{ env.IRIS_NAMESPACE }} \
             get events --sort-by='.lastTimestamp' --field-selector type!=Normal || true
 
+      - name: Claude triage
+        id: claude_triage
+        if: failure() && github.event_name == 'schedule'
+        uses: anthropics/claude-code-action@v1
+        timeout-minutes: 30
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN || secrets.CLAUDE_MAX_OAUTH_TOKEN }}
+          prompt: |
+            Read .agents/skills/canary-triage/SKILL.md and follow it.
+          claude_args: |
+            --model opus
+            --max-turns 50
+            --allowedTools "Bash(kubectl:*),Bash(gh:*),Bash(.venv/bin/iris:*),Bash(.venv/bin/python:*),Bash(cat:*),Bash(jq:*),Bash(head:*),Bash(tail:*),Bash(grep:*)"
+        env:
+          CANARY_LANE: gpu
+          CANARY_JOB_ID: ${{ steps.submit.outputs.job_id }}
+          CANARY_RUN_ID: ${{ env.RUN_ID }}
+          IRIS_CONFIG: ${{ env.IRIS_CONFIG }}
+          IRIS_NAMESPACE: ${{ env.IRIS_NAMESPACE }}
+          WANDB_ENTITY: ${{ env.WANDB_ENTITY }}
+          WANDB_PROJECT: ${{ env.WANDB_PROJECT }}
+          WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
+          GHA_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+
       # `cluster stop` only deletes Pods; NodePools survive and rely on the
       # CW autoscaler to scale down. Delete them explicitly to avoid lingering
       # H100 costs.
@@ -196,11 +222,23 @@ jobs:
             echo "Keeping node pool alive (keep_nodepool=true)"
           fi
 
+      - name: Read triage summary
+        id: slack_message
+        if: failure() && github.event_name == 'schedule'
+        run: |
+          if [ -f slack_message.md ]; then
+            # Escape for JSON embedding: newlines to \n, quotes escaped
+            MSG=$(cat slack_message.md | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')
+          else
+            MSG='":red_circle: *GPU Canary failed*\nRun: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"'
+          fi
+          echo "text=$MSG" >> "$GITHUB_OUTPUT"
+
       - name: Notify Slack on failure
         if: failure() && github.event_name == 'schedule'
         uses: slackapi/slack-github-action@v2
         with:
           webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
           webhook-type: incoming-webhook
           payload: |
-            text: ":red_circle: *GPU Canary failed*\nRun: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            text: ${{ steps.slack_message.outputs.text }}

.github/workflows/marin-canary-ferry.yaml

@@ -12,6 +12,8 @@ on:
 
 permissions:
   contents: read
+  issues: write    # claude triage files issues
+  id-token: write  # claude-code-action OIDC
 
 jobs:
   canary-ferry:
@@ -149,11 +151,45 @@ jobs:
           .venv/bin/iris --config=${{ env.IRIS_CONFIG }} \
             job list --json 2>/dev/null | jq '.[0:5]' || true
 
+      - name: Claude triage
+        id: claude_triage
+        if: failure() && github.event_name == 'schedule'
+        uses: anthropics/claude-code-action@v1
+        timeout-minutes: 30
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN || secrets.CLAUDE_MAX_OAUTH_TOKEN }}
+          prompt: |
+            Read .agents/skills/canary-triage/SKILL.md and follow it.
+          claude_args: |
+            --model opus
+            --max-turns 50
+            --allowedTools "Bash(gh:*),Bash(.venv/bin/iris:*),Bash(.venv/bin/python:*),Bash(cat:*),Bash(jq:*),Bash(head:*),Bash(tail:*),Bash(grep:*)"
+        env:
+          CANARY_LANE: tpu
+          CANARY_JOB_ID: ${{ steps.submit.outputs.job_id }}
+          CANARY_RUN_ID: ${{ env.RUN_ID }}
+          IRIS_CONFIG: ${{ env.IRIS_CONFIG }}
+          WANDB_ENTITY: ${{ env.WANDB_ENTITY }}
+          WANDB_PROJECT: ${{ env.WANDB_PROJECT }}
+          WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
+          GHA_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+
+      - name: Read triage summary
+        id: slack_message
+        if: failure() && github.event_name == 'schedule'
+        run: |
+          if [ -f slack_message.md ]; then
+            MSG=$(cat slack_message.md | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')
+          else
+            MSG='":red_circle: *TPU Canary failed*\nRun: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"'
+          fi
+          echo "text=$MSG" >> "$GITHUB_OUTPUT"
+
       - name: Notify Slack on failure
         if: failure() && github.event_name == 'schedule'
         uses: slackapi/slack-github-action@v2
         with:
           webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
           webhook-type: incoming-webhook
           payload: |
-            text: ":red_circle: *TPU Canary failed*\nRun: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            text: ${{ steps.slack_message.outputs.text }}