Fix/webhook cascade

[Aryainguz]

Summary

Deleting a webhook left every alert whose channel.webhookId pointed at it broken indefinitely silently logging WEBHOOK_ERROR on every evaluation tick without any user-visible notice.

A user can delete a webhook (to rotate credentials, rename it, etc.) not realising that every alert attached to it just died silently.

This fix nulls the channel field on all referencing alerts before the webhook is removed, so they continue to evaluate but no longer attempt delivery to the deleted endpoint. The alert data, thresholds, and history are fully preserved.

Dashboard deletion cascades via deleteDashboardAlerts; SavedSearch deletion via deleteSavedSearchAlerts. Webhook deletion had no equivalent this PR closes that gap.

Fixes #2590

Changed file: packages/api/src/routers/api/webhooks.ts

+import Alert, { AlertState } from '@/models/alert';
 import Webhook, { WebhookService } from '@/models/webhook';

 router.delete('/:id', ..., async (req, res, next) => {
   const teamId = req.user?.team;
   if (teamId == null) return res.sendStatus(403);

+  // Null the channel on referencing alerts so they don't fire WEBHOOK_ERROR on every tick.
+  await Alert.updateMany(
+    { 'channel.webhookId': req.params.id, team: teamId },
+    { $set: { channel: { type: null } } },
+  );
+
   await Webhook.findOneAndDelete({ _id: req.params.id, team: teamId });
   res.json({});
 });

Changes:

• packages/api/src/routers/api/webhooks.ts: Added Alert.updateMany inside the DELETE /:id handler to cascade-null the channel of all attached alerts.
• packages/api/src/routers/api/__tests__/webhooks.test.ts: Added an integration test verifying the cascade correctly nulls the alert channel in the database when the webhook is deleted.
• /.changeset/fix-webhook-cascade-delete.md

Local Validations

[changeset-bot]

🦋 Changeset detected

Latest commit: da12dcd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@Aryainguz is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR adds a guard to the DELETE /:id webhook endpoint that returns 409 when one or more alerts still reference the webhook, preventing silent broken-alert state after deletion. The changeset description and tests are consistent with the actual implementation.

• webhooks.ts: A countDocuments check (scoped to the team) blocks deletion with a 409 and a count message when referencing alerts exist; no changes to the webhook delete path otherwise.
• webhooks.test.ts: Two new integration tests verify that the 409 fires when an alert references the webhook and that deletion succeeds once those alerts are removed.
• .changeset/fix-webhook-cascade-delete.md: Changeset correctly describes the blocking behaviour (updated from an earlier iteration that described cascade-nulling).

Confidence Score: 5/5

Safe to merge — the guard is correctly scoped to the requesting team, the string-typed webhookId comparison is consistent across the codebase, and both the happy path and the blocking path are covered by integration tests.

The change is a small, well-contained pre-delete check. Team isolation is enforced by including team: teamId in the countDocuments filter, the webhookId string comparison is consistent with how every other part of the codebase stores and queries that field, and the tests exercise both the 409 and the clean-delete paths. No logic paths are left uncovered.

No files require special attention.

Important Files Changed

Filename	Overview
packages/api/src/routers/api/webhooks.ts	Adds a pre-delete countDocuments guard scoped to the team; logic, team isolation, and string-typed webhookId comparison are all correct.
packages/api/src/routers/api/tests/webhooks.test.ts	Two new tests verify the 409 path and the successful-after-removal path; fixture fields (threshold, thresholdType) match the Alert schema requirements.
.changeset/fix-webhook-cascade-delete.md	Changeset correctly describes the blocking behaviour after being updated from the earlier cascade-null description.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Client
    participant WebhookRouter as DELETE /webhooks/:id
    participant AlertModel as Alert.countDocuments
    participant WebhookModel as Webhook.findOneAndDelete

    Client->>WebhookRouter: "DELETE /webhooks/{id}"
    WebhookRouter->>WebhookRouter: Validate teamId (403 if null)
    WebhookRouter->>AlertModel: "countDocuments({ channel.webhookId: id, team: teamId })"
    alt "count > 0"
        AlertModel-->>WebhookRouter: N referencing alerts
        WebhookRouter-->>Client: "409 { message: "N alert(s) still reference it" }"
    else "count == 0"
        AlertModel-->>WebhookRouter: 0
        WebhookRouter->>WebhookModel: "findOneAndDelete({ _id: id, team: teamId })"
        WebhookModel-->>WebhookRouter: deleted doc (or null)
        WebhookRouter-->>Client: "200 {}"
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Client
    participant WebhookRouter as DELETE /webhooks/:id
    participant AlertModel as Alert.countDocuments
    participant WebhookModel as Webhook.findOneAndDelete

    Client->>WebhookRouter: "DELETE /webhooks/{id}"
    WebhookRouter->>WebhookRouter: Validate teamId (403 if null)
    WebhookRouter->>AlertModel: "countDocuments({ channel.webhookId: id, team: teamId })"
    alt "count > 0"
        AlertModel-->>WebhookRouter: N referencing alerts
        WebhookRouter-->>Client: "409 { message: "N alert(s) still reference it" }"
    else "count == 0"
        AlertModel-->>WebhookRouter: 0
        WebhookRouter->>WebhookModel: "findOneAndDelete({ _id: id, team: teamId })"
        WebhookModel-->>WebhookRouter: deleted doc (or null)
        WebhookRouter-->>Client: "200 {}"
    end

Loading

_{Reviews (5): Last reviewed commit: "fix: prevent webhook deletion if referen..." | Re-trigger Greptile}

[github-actions]

Deep Review

✅ No critical issues found. The change is correctly team-scoped on both the count query and the delete, webhookId is consistently a string on both sides of the match, and both DB calls are wrapped in try/catch → next(err). The recommendations below are reliability and coverage improvements, not merge blockers.

🟡 P2 — recommended

• packages/api/src/routers/api/webhooks.ts:360 — The countDocuments guard and findOneAndDelete are two separate operations, so an alert created or repointed at this webhook between them slips through and becomes a dangling reference — the exact condition the guard exists to prevent.
  • Fix: Run the reference count and the delete inside one Mongo transaction/session, re-checking the count immediately before deleting.
  • _{reliability, correctness, maintainability, kieran-typescript}
• packages/api/src/routers/api/__tests__/webhooks.test.ts:190 — Every test alert is created under the same team as the webhook, so the team: teamId clause on the new count query is never exercised; dropping that filter would not be caught by any test.
  • Fix: Add a test where a referencing alert belongs to a different team and assert the delete still returns 200.
  • _{testing, correctness, security}

🔵 P3 nitpicks (5)

• packages/api/src/routers/api/webhooks.ts:364 — DELETE /webhooks/:id changes from always-200 to a possible 409; the existing WebhooksSection.tsx client surfaces the message generically, but no frontend test locks that in and out-of-repo callers assuming 200 are unhandled.
  • Fix: Add a frontend test asserting the 409 message is surfaced, and call out the status-code change in release notes.
• packages/api/src/routers/api/webhooks.ts:360 — The handler imports the Alert model and builds the query inline, whereas every other alert-deletion path routes through controllers/alerts.ts helpers.
  • Fix: Extract a countAlertsReferencingWebhook(webhookId, teamId) helper in controllers/alerts.ts and call it here.
• packages/api/src/routers/api/webhooks.ts:360 — The count query does not filter on AlertState, so a disabled or silenced alert still blocks deletion, which may surprise users.
  • Fix: Decide whether disabled alerts should block, and if not, add a state filter to the query.
• packages/api/src/routers/api/webhooks.ts:361 — Querying the channel.webhookId path works but is untyped because Alert.channel is Schema.Types.Mixed, so a future rename of the field breaks this guard with no compiler feedback (pre-existing pattern, now load-bearing).
  • Fix: Co-locate a typed filter builder with the AlertChannel union so the coupling is visible at the type level.
• .changeset/fix-webhook-cascade-delete.md:5 — The entry opens with Fix: (colon) while every sibling fix-* changeset opens with Fix and no colon.
  • Fix: Drop the colon to match the existing changeset format.
  • _{project-standards}

---

Reviewers (8): correctness, security, testing, api-contract, maintainability, kieran-typescript, reliability, project-standards.

Testing gaps:

• No test creates 2+ referencing alerts to assert the exact count in the 409 message.
• No frontend test verifies the 409 message reaches the user.
• No test covers an alert with channel: { type: null } (no webhookId) leaving unrelated deletes unblocked.

[pulpdrew]

Thanks for taking a look at this @Aryainguz

Deleting a webhook left every alert whose channel.webhookId pointed at it broken indefinitely silently logging WEBHOOK_ERROR on every evaluation tick without any user-visible notice.

A user can delete a webhook (to rotate credentials, rename it, etc.) not realising that every alert attached to it just died silently.

This is indeed a problem, but IMO this change makes it slightly worse than before because the user can still silently kill many alerts, but now the alert history UI does not indicate any problem to the user.

As the alert task is currently implemented, alerts with a broken Webhook reference will still be evaluated and save their state to the AlertHistory. So the alerts haven't died, their state is just shadowed in the AlertHistory UX with the webhook error. So I think that a better approach here might be to either:

1. Error on webhook deletion, encouraging the user to migrate the alerts before deleting the webhook; OR
2. Update the alert history UX to indicate both the WEBHOOK_ERROR and the actual alert evaluation state

[Aryainguz]

Thanks for taking a look at this @Aryainguz

Deleting a webhook left every alert whose channel.webhookId pointed at it broken indefinitely silently logging WEBHOOK_ERROR on every evaluation tick without any user-visible notice.
A user can delete a webhook (to rotate credentials, rename it, etc.) not realising that every alert attached to it just died silently.

This is indeed a problem, but IMO this change makes it slightly worse than before because the user can still silently kill many alerts, but now the alert history UI does not indicate any problem to the user.

As the alert task is currently implemented, alerts with a broken Webhook reference will still be evaluated and save their state to the AlertHistory. So the alerts haven't died, their state is just shadowed in the AlertHistory UX with the webhook error. So I think that a better approach here might be to either:

1. Error on webhook deletion, encouraging the user to migrate the alerts before deleting the webhook; OR
2. Update the alert history UX to indicate both the WEBHOOK_ERROR and the actual alert evaluation state

@pulpdrew yeah agreed that silently nullifying alert channels is worse than erroring. I think we should go with Option 1. The delete endpoint will return a 409 Conflict listing the alert count, so the user knows to migrate those alerts first before deleting. I think this is better but let me know your thoughts. I've updated code and test cases as per this as well

Initial idea of implementation before that user don't delete the alert and usually update webhooks only but I agree it's not correct, and your options 1 sounds best

[pulpdrew]

@Aryainguz thanks! Although now that the API errors, could we show a mantine notification or other indicator the user when the delete errors, so that they are aware why the deletion is failing?

Also cc @jordan-simonovski in regards to #2586 in case we want to make the new external webhooks DELETE API consistent with this behavior.

[Aryainguz]

@Aryainguz thanks! Although now that the API errors, could we show a mantine notification or other indicator the user when the delete errors, so that they are aware why the deletion is failing?

Also cc @jordan-simonovski in regards to #2586 in case we want to make the new external webhooks DELETE API consistent with this behavior.

@pulpdrew The frontend already handles this! The DeleteWebhookButton in WebhooksSection.tsx catches HTTPError, extracts the .message from the JSON response, and shows it as a red Mantine notification toast for 5 seconds. So when the 409 fires, the user will see: "Cannot delete webhook: N alert(s) still reference it. Please update or remove those alerts first." So no frontend changes needed for this

Attaching image and video for same

2026-07-07.20-22-29.mov

[pulpdrew]

My mistake, thanks for confirming that the UI shows the notification! LGTM, thanks for the contribution.