Add headless webhook-service: subscriptions CRUD, HMAC-signed deliveries off SNS/SQS, retries, DLQ, delivery log

[devin-ai-integration]

Summary

Adds services/webhook-service: a headless (JSON-only) Go service that lets API users register outbound webhooks and receive HMAC-signed deliveries for events already flowing over the LocalStack SNS topic otterworks-events. Replaces the "poll /files and /documents" pattern; nothing in notification-service is touched.

Data flow

file/document-service --SNS otterworks-events--> SQS otterworks-webhook-events
  -> consumer: events.Parse (SNS wrapper | raw; eventType|event_type; payload) -> Event{ID,Type,Source,OccurredAt,Data}
  -> dispatcher.Enqueue: one delivery per matching active subscription OF THE EVENT'S OWNER
     (ownerId/userId/authorId from the producer payload; exact type or "*"; ownerless events dropped),
     UNIQUE(subscription_id, event_id) makes SQS redelivery idempotent
  -> dispatcher workers: POST signed JSON, record every attempt
       2xx            -> delivered
       else/timeout   -> retrying, next_attempt_at = now + Backoff(attempt)   # exp, jitter [d/2,d], capped
       attempts>=max  -> publish to SQS otterworks-webhook-dlq, THEN dead_letter (row kept);
                         a failed publish leaves the row retrying (no more HTTP) and is re-published next cycle

Signing — X-OtterWorks-Signature: t=<unix>,v1=hex(hmac_sha256(secret, "<unix>.<raw-body>")), plus X-OtterWorks-Event, -Event-ID, -Delivery-ID. Secrets (whsec_…) are returned only on create / rotate-secret; every other read omits them. Redirects are not followed; non-https / private / localhost targets are rejected unless ALLOW_PRIVATE_TARGETS=true (compose sets it for the demo only; the service default rejects them).

API (gateway → /api/v1/webhooks, owner = X-User-ID from the JWT; all ownership-scoped)

GET|POST        /subscriptions              GET|PUT|PATCH|DELETE /subscriptions/{id}
POST            /subscriptions/{id}/rotate-secret   POST /subscriptions/{id}/test
GET             /deliveries?status=&subscriptionId=&eventType=&since=&until=&limit=&offset=
GET             /deliveries/{id}            GET /deliveries/{id}/attempts
POST            /deliveries/{id}/replay     # keeps attempt history, max_attempts = attempts + N
GET             /dead-letters   GET /stats   GET /event-types

404/405/unknown routes/errors are JSON too; requireUUID 404s malformed ids and filterFrom 400s malformed subscriptionId/cursor before they reach Postgres.

Store — store.Store interface with Postgres (pgx, SELECT … FOR UPDATE SKIP LOCKED + lease of DELIVERY_TIMEOUT_SECONDS+30s for due deliveries; migrations applied at boot) and Memory (tests, DATABASE_URL=memory for the CI smoke).

Wiring — gateway WEBHOOK_SERVICE_URL + route; scripts/localstack-init.sh creates the two queues and the SNS→SQS subscription; docker-compose.yml adds webhook-service (not host-published — it trusts X-User-ID, so it is reachable only via the gateway) and webhook-sink (a headless signature-verifying receiver with /hook, /flaky (503,503,200), /fail, /received).

Proving it's headless

• scripts/headless-guard.sh (runs in CI and make lint): fails on any .html/.css/.js/.ts, template dirs/html/template, static/public/assets dirs, frontend toolchain files, http.FileServer/ServeFile, //go:embed, text/html content types or HTML tags.
• scripts/smoke-json-only.sh: hits every route (incl. /index.html, /static/app.js, wrong methods, browser Accept: text/html) and asserts JSON content-type, JSON object body, no HTML, no redirects. CI runs it against a live memory-backed instance; the CI job also builds the image and asserts the container ships exactly /app/server.
• scripts/e2e-compose.sh (make e2e-webhook): full localhost demo — CRUD through the gateway, SNS publish, 3 signed deliveries (ok / flaky-with-retries / dead-lettered), attempt log, stats, DLQ message, replay, gateway JSON smoke.

Verification

go vet, go test -race ./..., staticcheck, headless guard, Docker build and the full compose e2e pass locally; the e2e run log is attached to the Slack thread. Tail of the run:

OK   retry with backoff and dead-letter transitions observed via the delivery log
OK   all 8 requests carried a valid X-OtterWorks-Signature
OK   dead letter published to otterworks-webhook-dlq
OK   replay re-queued the delivery; history kept (4 failed + 1 delivered attempts)
== 27 passed, 0 failed          # JSON-only smoke through the gateway
OK   image ships exactly /app/server
E2E PASSED

Note: README already claims webhooks exist; it's left as-is since this PR makes it true.

Not in this PR: EKS wiring (Helm chart, ECR build, deploy-dev.sh/spinup-dev.sh/tenant lists). The route 502s on EKS like any not-yet-deployed backend until that lands; tenants also have SNS/SQS disabled per AGENTS.md, so the event path there needs its own design.

Link to Devin session: https://partner-workshops.devinenterprise.com/sessions/7ce0574f3d474cda84d608d428a70941
Open in Devin Desktop: https://partner-workshops.devinenterprise.com/desktop/session/7ce0574f3d474cda84d608d428a70941?variant=devin
Requested by: @mbatchelor81

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 8 potential issues.

4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🔴 Deployed webhook route has no backend

ServiceRoutes registers webhooks, but deployment service lists omit the webhook workload. Every EKS webhook request returns 502.

Learn more

The gateway now exposes the webhook prefix in every environment. Compose defines the backend, but the repository's canonical EKS deploy paths enumerate services explicitly. The dev, spin-up, and tenant lists omit this service, and no webhook Helm chart exists. Kubernetes therefore deploys a gateway that resolves webhook-service:8092 for no matching Service.

Example: A user calls POST /api/v1/webhooks/subscriptions on the main EKS tenant. The gateway selects the new route, DNS cannot resolve webhook-service, and the proxy returns 502 instead of creating a subscription.

Recommended fix: Add a ClusterIP-only webhook Helm chart, image build/push wiring, port mapping, database configuration, gateway URL, and service entries to deploy-dev.sh, spinup-dev.sh, and tenant deployment helpers. Keep SNS/SQS consumers disabled for tenants as required by AGENTS.md, or define an isolated event path before enabling them.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Acknowledged, not addressed in this PR. The request was scoped to a localhost/compose demo; the EKS path (Helm chart, ECR build, deploy-dev/spinup/tenant lists, and a per-tenant event path given SNS/SQS is disabled for tenants per AGENTS.md) is a separate piece of work. Until then the route behaves like any other not-yet-deployed backend (gateway 502). Flagged to the requester as a follow-up decision.

[devin-ai-integration]

🟨 Default stack permits private webhook targets

Compose enables ALLOW_PRIVATE_TARGETS for every subscription. Any authenticated user can make the service POST to internal Compose, host, or cloud-network endpoints.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Kept, intentionally: the compose stack is the localhost demo and the signed sink lives on the compose network, so ALLOW_PRIVATE_TARGETS=true is required there. The service default is false (private/loopback/link-local targets rejected), and the compose line is commented as local-demo-only. Not set anywhere outside docker-compose.yml.