Nightly Slack notification: perf+eval canvas with peak / 7d-avg±σ / current tables

[khluu]

What

Reworks the nightly perf/eval Slack notification from a plain text message into a Slack Canvas (real tables) plus a short summary message in #ci-notifications that links to it.

Each nightly run now compares, per model/config (perf) and per model/task/metric (eval):

• Peak (best over the trailing 30 days)
• 7-day moving average ±σ
• Current nightly value
• Δ vs avg and Δ vs peak (relative %)
• a leading status dot: 🔴 = regression vs the 7-day average (≥2σ and ≥1% move), 🟢 = otherwise

The feed message carries the headline counts + any flagged rows and links to the full canvas.

Why

Slack messages can't render real tables, so the previous code-block layout was cramped and hard to scan. A canvas gives proper tables; the summary message keeps at-a-glance regression visibility in the channel feed.

Changes

• compare.ts — computeEvalHistory() (mirrors computePerfHistory()): per (model, task, n_shot, metric, filter) current / peak / trailing 7d mean ±σ / z / status. Added deltaPeakPct to both perf and eval history rows. Existing exports untouched (still used by /api/nightly, /compare, /perf).
• nightly-template.ts — renderNightlyCanvas() (Canvas-flavored Markdown: perf + eval tables with the status dot and Δ-vs-avg / Δ-vs-peak columns) and renderChannelSummary() (feed message). renderNightlySummary() retained as a fallback.
• slack.ts — createCanvas() and shareCanvasToChannel() (canvases.create / canvases.access.set).
• cron/nightly-summary route — loads the nightly window, computes perf + eval history, creates a canvas, shares it to SLACK_CI_NOTIFICATIONS_CHANNEL, and posts the linking message. Falls back to the text-table message if the canvas call fails, so notifications never silently break.

Testing

• tsc --noEmit, eslint, and next build all pass.
• Canvas + summary rendering validated end-to-end against live Databricks data (latest nightly).
• Preview canvas (rendered from real data): https://inferact.slack.com/docs/T09RC6UHDNW/F0B8C2H60G3

⚠️ Required before this posts canvases in prod

1. Add the canvases:write scope to the runner_queue_alert Slack app and reinstall it to the workspace. The bot currently has only incoming-webhook, chat:write, reactions:write; until the scope is added the cron uses the text-message fallback (no breakage).
2. Confirm SLACK_CI_NOTIFICATIONS_CHANNEL points to #ci-notifications.

Note: canvases.create's request shape is confirmed (a scope-less probe failed only on missing_scope, not arguments), but canvases.access.set (the channel-share call) is unverified until the bot has the scope — will confirm with a live run and adjust the field format if Slack requires it.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
vllm-ci-dashboard	Ready	Preview, Comment	Jun 7, 2026 8:49am

[khluu]

✅ Live end-to-end test passed against the production vllm_ci_bot + #ci-notifications.

• canvases.create → ok
• canvases.access.set (channel share) → ok — the previously-unverified share call works with the shape implemented here ({ canvas_id, access_level: "read", channel_ids: [...] }), no code change needed.
• chat.postMessage summary (links the canvas) → ok

Ran the committed lib code path (renderNightlyCanvas → createCanvas → shareCanvasToChannel → postMessage) against the latest nightly; canvas rendered with the perf + eval tables, 🔴/🟢 dots, and Δ-vs-avg / Δ-vs-peak columns.

Prod checklist: set Vercel SLACK_BOT_TOKEN to the vllm_ci_bot bot token (it has canvases:write) and SLACK_CI_NOTIFICATIONS_CHANNEL to the #ci-notifications channel id; ensure the bot is a member of that channel.