feat: Adding remaining API v2 endpoints - saved searches & webhooks#2586
Open
jordan-simonovski wants to merge 16 commits into
Open
feat: Adding remaining API v2 endpoints - saved searches & webhooks#2586jordan-simonovski wants to merge 16 commits into
jordan-simonovski wants to merge 16 commits into
Conversation
Saved searches and webhooks are the last API endpoints remaining. This frees us up to finalise the tf provider build
🦋 Changeset detectedLatest commit: 79628ef The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
Greptile SummaryThis PR adds full CRUD to the External API v2 for two previously read-limited (or absent) resources: a new
Confidence Score: 3/5The webhook CRUD and pagination additions are solid, but the saved-search PUT grandfathering logic has a key-normalisation bug that will cause a 400 on any GET → PUT round-trip where the saved search was originally created by the UI without an explicit filter type field — a realistic and common case that would break provider tooling on existing data. The filterKey function produces [null, condition] for stored filters whose type was never written, while incoming filters always carry type:'sql' from the schema default, yielding ["sql", condition]. The keys never match, so grandfathering is silently bypassed for those filters and the PUT rejects with 400 on real saved-search data. The fix is a one-line normalisation (f.type ?? 'sql'), but until it lands, any provider that reads a saved search and writes it back unchanged will hit a wall if the search contains a legacy typeless filter. packages/api/src/routers/external-api/v2/savedSearches.ts — specifically the filterKey helper and how it is called in the PUT handler with .lean() results from the database. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant C as Client (Clickstack)
participant MW as Auth + Rate Limiter
participant SS as /api/v2/saved-searches
participant WH as /api/v2/webhooks
participant SC as SavedSearch Controller
participant DB as MongoDB
Note over C,DB: Saved Search CRUD
C->>MW: Bearer token
MW->>SS: GET / (list)
SS->>DB: "find({team}) + countDocuments (parallel)"
DB-->>SS: [docs], total
SS-->>C: "{data, meta:{total,limit,offset}}"
C->>SS: POST / (create)
SS->>SS: validateRequest (Zod body)
SS->>SS: requireValidSourceId (team-scoped)
SS->>SS: firstNonRenderableFilter check
SS->>SC: createSavedSearch(teamId, input)
SC->>DB: SavedSearch.create()
DB-->>SC: doc
SC-->>SS: doc
SS-->>C: "{data: toExternalJSON()}"
C->>SS: PUT /:id (update)
SS->>DB: "findOne({_id,team},{filters:1}).lean()"
DB-->>SS: existing filters
SS->>SS: build grandfathered Set via filterKey()
SS->>SS: firstNonRenderableFilter(req.body.filters, grandfathered)
SS->>SC: updateSavedSearch(teamId, id, input)
SC->>DB: findOneAndUpdate
DB-->>SC: updated doc
SS-->>C: "{data: toExternalJSON()}"
C->>SS: DELETE /:id
SS->>SC: deleteSavedSearch(teamId, id)
SC->>DB: findOne (existence check)
SC->>DB: deleteSavedSearchAlerts (pre-delete)
SC->>DB: deleteOne (parent)
SC->>DB: deleteSavedSearchAlerts (post-delete re-sweep)
SS-->>C: "{}"
Note over C,DB: Webhook CRUD
C->>WH: POST / (create)
WH->>WH: validateRequest (externalWebhookCreateSchema)
WH->>DB: "Webhook.create({team,...})"
DB-->>WH: doc (or 11000 duplicate key error)
WH-->>C: "{data} or 400 duplicate"
C->>WH: PUT /:id (update)
WH->>DB: "findOne({_id,team}) snapshot url+service"
WH->>WH: compute $set/$unset (destination-change secret clearing)
WH->>DB: "findOneAndUpdate({_id,team,url,service}, op, {new:true})"
DB-->>WH: updated doc or null (concurrent mod)
WH-->>C: "{data} or 409 conflict or 404"
C->>WH: DELETE /:id
WH->>DB: "Alert.countDocuments({channel.webhookId,team})"
DB-->>WH: count
WH->>DB: "Webhook.findOneAndDelete (if count=0)"
WH-->>C: "{} or 409 (alerts still reference)"
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant C as Client (Clickstack)
participant MW as Auth + Rate Limiter
participant SS as /api/v2/saved-searches
participant WH as /api/v2/webhooks
participant SC as SavedSearch Controller
participant DB as MongoDB
Note over C,DB: Saved Search CRUD
C->>MW: Bearer token
MW->>SS: GET / (list)
SS->>DB: "find({team}) + countDocuments (parallel)"
DB-->>SS: [docs], total
SS-->>C: "{data, meta:{total,limit,offset}}"
C->>SS: POST / (create)
SS->>SS: validateRequest (Zod body)
SS->>SS: requireValidSourceId (team-scoped)
SS->>SS: firstNonRenderableFilter check
SS->>SC: createSavedSearch(teamId, input)
SC->>DB: SavedSearch.create()
DB-->>SC: doc
SC-->>SS: doc
SS-->>C: "{data: toExternalJSON()}"
C->>SS: PUT /:id (update)
SS->>DB: "findOne({_id,team},{filters:1}).lean()"
DB-->>SS: existing filters
SS->>SS: build grandfathered Set via filterKey()
SS->>SS: firstNonRenderableFilter(req.body.filters, grandfathered)
SS->>SC: updateSavedSearch(teamId, id, input)
SC->>DB: findOneAndUpdate
DB-->>SC: updated doc
SS-->>C: "{data: toExternalJSON()}"
C->>SS: DELETE /:id
SS->>SC: deleteSavedSearch(teamId, id)
SC->>DB: findOne (existence check)
SC->>DB: deleteSavedSearchAlerts (pre-delete)
SC->>DB: deleteOne (parent)
SC->>DB: deleteSavedSearchAlerts (post-delete re-sweep)
SS-->>C: "{}"
Note over C,DB: Webhook CRUD
C->>WH: POST / (create)
WH->>WH: validateRequest (externalWebhookCreateSchema)
WH->>DB: "Webhook.create({team,...})"
DB-->>WH: doc (or 11000 duplicate key error)
WH-->>C: "{data} or 400 duplicate"
C->>WH: PUT /:id (update)
WH->>DB: "findOne({_id,team}) snapshot url+service"
WH->>WH: compute $set/$unset (destination-change secret clearing)
WH->>DB: "findOneAndUpdate({_id,team,url,service}, op, {new:true})"
DB-->>WH: updated doc or null (concurrent mod)
WH-->>C: "{data} or 409 conflict or 404"
C->>WH: DELETE /:id
WH->>DB: "Alert.countDocuments({channel.webhookId,team})"
DB-->>WH: count
WH->>DB: "Webhook.findOneAndDelete (if count=0)"
WH-->>C: "{} or 409 (alerts still reference)"
Reviews (13): Last reviewed commit: "fix(review): address code-review finding..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
|
review content below |
Contributor
E2E Test Results✅ All tests passed • 227 passed • 3 skipped • 1495s
Tests ran across 4 shards in parallel. |
jordan-simonovski
changed the title
Contributor
Deep Review✅ No critical issues found. No P0/P1 defects survived verification — team-scoping, the write-only secret handling for webhook 🟡 P2 -- recommended
🔵 P3 nitpicks (11)
Reviewers (11): correctness, security, adversarial, testing, maintainability, api-contract, reliability, kieran-typescript, performance, project-standards, agent-native. Testing gaps:
Pre-existing / residual (not counted toward verdict): webhook |
pulpdrew
reviewed
Jul 6, 2026
pulpdrew
reviewed
Jul 6, 2026
pulpdrew
reviewed
Jul 6, 2026
Merged
…, field size caps, and implemented shared tag component
pulpdrew
reviewed
Jul 7, 2026
| // expression into filters[].condition/left/right. | ||
| const MAX_FILTER_EXPR_LENGTH = 8 * 1024; | ||
|
|
||
| const boundedFilterSchema = FilterSchema.superRefine((filter, ctx) => { |
Contributor
There was a problem hiding this comment.
We probably want a separate FilterSchema here because saved searches expect a very particular format for filters (type: 'sql', condition: "<col> IN ('<value1>', ...)" - otherwise the filters will not render in the sidebar. The existing savedFilterValues on the dashboards API has the same format, I believe (though there is probably better validation we could do there).
Contributor
Author
There was a problem hiding this comment.
Good call. Resolved
Contributor
|
Closes #2544 |
…ilter schema to ensure it is renderable on UI. Moved check into utils
- savedSearches: remove NUL bytes from filterKey (made the file binary to git);
use a lean {filters:1} query on PUT instead of the fully-populated
getSavedSearch; document that reads may return legacy lucene/sql_ast filters
- filters: reject pinned filters whose parsed key absorbed a NOT/AND/OR operator
(e.g. `col NOT BETWEEN 1 AND 2`), which executed the inverse of the rendered facet
- webhooks: block external DELETE with 409 while an alert still references the
webhook (mirrors the internal router; prevents orphaned alerts); emit a metric
for the persisted-but-unserializable path and extract the duplicated block
- savedSearch controller: make the post-delete alert re-sweep best-effort so its
failure logs a warning instead of turning a successful delete into a 500
- tests: NOT-operator filter rejection; external webhook DELETE 409 guard
pulpdrew
approved these changes
Jul 8, 2026
Comment on lines
+96
to
+103
| try { | ||
| await deleteSavedSearchAlerts(savedSearchId, teamId); | ||
| } catch (e) { | ||
| logger.warn( | ||
| { err: e, savedSearchId, team: teamId }, | ||
| 'Post-delete alert re-sweep failed; a concurrently-created alert may be orphaned for this deleted saved search', | ||
| ); | ||
| } |
Contributor
There was a problem hiding this comment.
If only we had transactions 😢
Comment on lines
+64
to
+66
| tags: tagsSchema.describe( | ||
| `Dashboard tags. Up to ${MAX_TAGS} tags, each at most ${MAX_TAG_LENGTH} characters.`, | ||
| ), |
Contributor
There was a problem hiding this comment.
Curious - are the constraints not already automatically communicated to the agent through the tagsSchema?
Comment on lines
+617
to
+622
| // Grandfather filters that are unchanged from the stored search: a | ||
| // read-modify-write may echo back a legacy non-renderable filter the API | ||
| // itself served on GET. Only new/changed filters must render. | ||
| // Read only the stored filters for the grandfather check — a lean, | ||
| // projected query, not the fully-populated getSavedSearch (which also | ||
| // populates createdBy/updatedBy we don't need here). |
Contributor
There was a problem hiding this comment.
FWIW, I don't think that we should have any saved searches with the sql_ast form, since filters were not saved in saved searches at all until relatively recently, and I don't believe that we've made any changes since then related to sql_ast filters in search.
| 'channel.webhookId': req.params.id, | ||
| team: teamId, | ||
| }); | ||
| if (referencingAlertCount > 0) { |
|
|
||
| // MongoDB duplicate-key error (unique-index violation). Used to translate a | ||
| // racing or colliding write into a 400 instead of a generic 500. | ||
| export const isDuplicateKeyError = (e: unknown): boolean => |
Contributor
There was a problem hiding this comment.
Just a heads up there is another version of this in EE already: https://github.com/DeploySentinel/hyperdx-ee/blob/5c1e0dcac002327cc31cadc33b68ce48631c8f17/packages/api/src/utils/mongo.ts#L8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds bearer-auth CRUD to the External API v2 for two resources so a provider (clickstack) can manage them programmatically:
Why
The v2 external API previously had no way to manage saved searches at all (/api/v2/search only executes a query), and webhooks were read-only. OSS parity only needs the read-only webhook data source, but the provider integration needs to manage both as first-class resources. The webhook CRUD is specifically what unblocks the clickstack_webhook resource (HDX-4490).
Alerts v2 already had full CRUD, so it's untouched.
What changed
Security / sensitive data
Webhook headers and queryParams are write-only: accepted on create/update but never returned by any read endpoint (externalWebhookSchema omits them), so auth tokens and other secrets don't leak. All operations are team-scoped (create/read/update/delete all filter by team); request bodies can't spoof team/_id (Zod strips unknown keys, team is set server-side).
Testing
Coverage includes team-scoping/404 isolation, sourceId ownership validation, duplicate rejection, and assertions that webhook secrets are never returned on create.
Follow-up (not in this PR)
MCP agent tools don't yet cover the new write paths (webhook create/update/delete, saved-search delete) — tracked in HDX-4700.
Changeset
Included (@hyperdx/api minor).
resolves hdx-4673 hdx-4674