Skip to content

Commit 14627f6

Browse files
authored
Add ClickHouse query for nightly binary pipeline failure alert (#8231)
## Summary Adds the ClickHouse query that backs a new Grafana alert for **nightly binary build pipeline failures** (e.g. the `linux-aarch64-binary-manywheel` / `manywheel-cpu-aarch64-build` failures on the `nightly` branch). Stored under `clickhouse_db_schema/grafana_alerts/` to match the existing convention (queries in source control; the Grafana rule itself is created in the UI). ## Query Counts nightly-branch binary-build jobs that completed with `conclusion = 'failure'` in the last 24h; the Grafana rule fires when count > 0. ## Tunables (call out for review) - `workflow_name LIKE '%-binary-%'` scopes to the binary pipeline; drop it to alert on *any* `nightly`-branch failure. - 24h window matches the daily nightly cadence. - Threshold (`> 0`) / team / priority live in the Grafana rule annotations, not the SQL. ## Next step (manual, not in this repo) Create the rule in `pytorchci.grafana.net/alerting/` using this query, with `Teams` + `Priority` annotations and advanced notification routing -> flows to alerting-infra -> GitHub issue. Then backfill the rule URL into the SQL comment.
1 parent 8a8caea commit 14627f6

1 file changed

Lines changed: 17 additions & 0 deletions

File tree

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
-- Alert when the nightly binary build pipeline has failures.
2+
--
3+
-- The nightly binary builds (manywheel / libtorch / conda, across
4+
-- linux / linux-aarch64 / windows / macos / s390x) run on the `nightly` branch,
5+
-- e.g. workflow `linux-aarch64-binary-manywheel`, job `manywheel-cpu-aarch64-build`.
6+
-- This counts jobs from those workflows that completed with a failure in the
7+
-- evaluation window; the Grafana alert fires when the count is > 0.
8+
--
9+
-- Used by Grafana alert: <add URL after creating the rule in pytorchci.grafana.net>
10+
SELECT count() AS failed_nightly_jobs
11+
FROM default.workflow_job
12+
WHERE
13+
head_branch = 'nightly'
14+
AND workflow_name LIKE '%-binary-%'
15+
AND status = 'completed'
16+
AND conclusion = 'failure'
17+
AND completed_at >= now() - INTERVAL 24 HOUR

0 commit comments

Comments
 (0)