Epic 7: Webhooks, events, and gRPC E2E coverage (v1.1.9)

[brandonrc]

Webhook delivery engine, event taxonomy, and gRPC services exist in 1.1.x backend but lack E2E coverage. Tracked under coverage policy #66.

Sub-tasks (1.1.x scope)

7.1 Webhook delivery retry engine (exponential backoff 30s -> 2m -> 15m -> 1h -> 4h)
7.2 Dead-letter queue handling for exhausted attempts
7.3 Max-attempts exhaustion scenario
7.4 Async retry job triggering (process_webhook_retries)
7.5 HMAC signature generation and X-Webhook-Signature header injection
7.6 Webhook secret rotation/revocation
7.7 Custom header injection
7.8 SSRF prevention on webhook URLs (private IPs, metadata servers, loopback)
7.9 URL re-validation at delivery time (DNS rebinding prevention)
7.10 Repository-scoped webhook filtering
7.11 Multi-event filtering (currently only artifact.uploaded tested)
7.12 Webhook test/dry-run endpoint (POST /{id}/test)
7.13 Webhook enable/disable toggle (POST /{id}/enable, /{id}/disable)
7.14 Event delivery list filtering by status
7.15 SSE event stream consumption from a client (with backpressure / lagged-event handling)
7.16 Cross-handler event propagation (artifact.created -> webhook delivery -> SSE broadcast)
7.17 gRPC SbomService (port 9090): GenerateSbom, GetSbom, ListSbomsForArtifact, ConvertSbom, DeleteSbom, RegenerateSbom, CheckLicenseCompliance
7.18 gRPC CveHistoryService: GetCveHistory, UpdateCveStatus, GetCveTrends, TriggerRetroactiveScan
7.19 gRPC SecurityPolicyService: license policy CRUD

[brandonrc]

Sub-issue breakdown (proposed)

PR #149 ships three proof-of-concept tests against this epic (custom headers, multi-event filter, SSE stream). Each remaining sub-task below should become its own sub-issue so the work can be parallelised and the epic stays an aggregator rather than a single-PR mega-task.

Webhook delivery engine

• 7.1 Retry schedule contract (12-attempt, ~24h cap) -- partially covered by PR test(webhooks): rewrite suite against v2 wire contract (#919 E8) #139's test-webhook-retry-recover.sh (first 3 attempts) and test-webhook-deadletter.sh (exhaustion). Sub-issue: extend dead-letter to assert the full 12-attempt schedule with WEBHOOK_FAST_FORWARD=1.
• 7.2 Dead-letter queue handling for exhausted attempts -- covered by test-webhook-deadletter.sh; sub-issue: add a separate test asserting dead-lettered rows are queryable via a list endpoint (separate from disabled-status assertion).
• 7.3 Max-attempts exhaustion scenario -- folded into 7.2.
• 7.4 Async retry job triggering (process_webhook_retries) -- sub-issue: assert that a long-paused receiver eventually drains once it comes back up (the retry job must keep firing).

Webhook security and rotation

• 7.5 HMAC signature contract -- covered by PR test(webhooks): rewrite suite against v2 wire contract (#919 E8) #139's test-webhook-hmac-signature.sh.
• 7.6 Webhook secret rotation/revocation -- covered by PR test(webhooks): rewrite suite against v2 wire contract (#919 E8) #139's test-webhook-rotation-overlap.sh. Sub-issue: add a revocation-only test (rotate without overlap, old secret must be rejected).
• 7.7 Custom header injection -- shipped in PR test: webhook v2 + search/browse E2E coverage (#85, #75, #73) #149 (test-webhook-custom-headers.sh).
• 7.8 SSRF prevention on webhook URLs -- sub-issue: pin the 4xx contract on private-IP / metadata-server / loopback URLs at create time.
• 7.9 URL re-validation at delivery time (DNS rebinding) -- sub-issue: separate test that flips a name from public -> 169.254.169.254 between create and delivery and asserts the delivery is dropped.

Event routing

• 7.10 Repository-scoped webhook filtering -- sub-issue: assert a webhook with repository_id = X only receives events for repo X.
• 7.11 Multi-event filtering -- shipped in PR test: webhook v2 + search/browse E2E coverage (#85, #75, #73) #149 (test-webhook-multi-event-filter.sh).
• 7.12 Webhook test/dry-run endpoint (POST /{id}/test) -- covered by test-webhook-crud.sh; sub-issue: assert response shape (id, status, latency_ms).
• 7.13 Webhook enable/disable toggle -- sub-issue: assert that a disabled webhook drops events instead of queuing them.
• 7.14 Event delivery list filtering by status -- sub-issue: assert GET /deliveries?status=failed returns only failed rows.

Event consumption

• 7.15 SSE event stream consumption -- shipped in PR test: webhook v2 + search/browse E2E coverage (#85, #75, #73) #149 as proof-of-concept (test-events-sse-stream.sh). Sub-issue: extend with back-pressure (slow consumer), lagged-event handling, named event types, Last-Event-ID resumption.
• 7.16 Cross-handler event propagation (artifact.created -> webhook delivery -> SSE broadcast) -- sub-issue: end-to-end assertion in one test, fires one upload and verifies the same event hits both surfaces with the same delivery_id / event_id.

gRPC services (port 9090)

These need their own test directory (tests/grpc/) and a grpcurl helper in tests/lib/common.sh. Likely one sub-issue per service:

• 7.17 gRPC SbomService -- 7 RPCs (GenerateSbom, GetSbom, ListSbomsForArtifact, ConvertSbom, DeleteSbom, RegenerateSbom, CheckLicenseCompliance).
• 7.18 gRPC CveHistoryService -- 4 RPCs (GetCveHistory, UpdateCveStatus, GetCveTrends, TriggerRetroactiveScan).
• 7.19 gRPC SecurityPolicyService -- license policy CRUD.

Suggested ordering: gRPC harness work (the grpcurl helper) first, then the three gRPC sub-issues in parallel. The webhook and event sub-issues are independent and can be picked up in any order.