Skip to content

Commit 1832f45

Browse files
committed
fix(alerts): Add new alerts for submit latency and kudos accounting components
- Mapped new alerts for HordeSubmitLatencyHigh for both image and text components. - Introduced internal alerts for kudos accounting, including silent applier and backlog conditions. - Added application background job failure alerts to improve monitoring of internal processes.
1 parent 5a7aec6 commit 1832f45

3 files changed

Lines changed: 47 additions & 1 deletion

File tree

packages/ai-horde-service-alerts/config/alert_component_map.yaml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,11 @@ mappings:
3737
label_match:
3838
horde_gentype: image
3939
status: degraded
40+
- component: image
41+
alertname: HordeSubmitLatencyHigh
42+
label_match:
43+
horde_gentype: image
44+
status: degraded
4045

4146
# ---- Text generation ----
4247
- component: text
@@ -47,6 +52,11 @@ mappings:
4752
label_match:
4853
horde_gentype: text
4954
status: degraded
55+
- component: text
56+
alertname: HordeSubmitLatencyHigh
57+
label_match:
58+
horde_gentype: text
59+
status: degraded
5060

5161
# ---- Worker coordination ----
5262
- component: workers
@@ -61,6 +71,30 @@ mappings:
6171
alertname: HordeWebhookFailureBurst
6272
status: degraded
6373

74+
# ---- Kudos accounting (internal component) ----
75+
# A silent applier freezes balance materialization entirely; sustained lag or
76+
# backlog means balances and counters are stale by that amount. Postings stay
77+
# durable throughout, so these drive the internal component rather than any
78+
# public pill. HordeKudosFloorCreationHigh is a finance anomaly signal, not a
79+
# service-health one, and is deliberately unmapped.
80+
- component: kudos
81+
alertname: HordeKudosApplierSilent
82+
status: down
83+
- component: kudos
84+
alertname: HordeKudosApplierLagHigh
85+
status: degraded
86+
- component: kudos
87+
alertname: HordeKudosBacklogHigh
88+
status: degraded
89+
- component: kudos
90+
alertname: HordeKudosReservationStuck
91+
status: degraded
92+
93+
# ---- Application background jobs (internal component) ----
94+
- component: app-jobs
95+
alertname: HordeBackgroundJobFailures
96+
status: degraded
97+
6498
# alchemy: pill driven only by the external prober — the deployed rules
6599
# carry no alchemy/interrogation-specific alert, so there is nothing curated
66100
# to map. Add an entry here once a public-determinable alchemy alert exists.

packages/ai-horde-service-alerts/config/components.yaml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,18 @@ components:
4646
display_order: 60
4747

4848
# ---- Internal-only infrastructure ----
49+
- id: kudos
50+
name: Kudos Accounting
51+
description: Ledger applier folding balances and derived counters.
52+
audience: internal
53+
display_order: 100
54+
55+
- id: app-jobs
56+
name: Application Background Jobs
57+
description: Quorum-node periodic jobs (queue checks, stats, kudos applier).
58+
audience: internal
59+
display_order: 105
60+
4961
- id: postgres
5062
name: PostgreSQL
5163
description: Primary application database.

packages/ai-horde-service-alerts/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
44

55
[project]
66
name = "ai-horde-service-alerts"
7-
version = "0.7.0"
7+
version = "0.8.0"
88
description = "Exposes AI Horde service alert signals as a FastAPI service."
99
readme = "README.md"
1010
requires-python = ">=3.12"

0 commit comments

Comments
 (0)