feat(webhooks-oss): WAIT_FOR_WEBHOOK end-to-end — port + persistence across 5 backings

[nthmost-orkes]

Summary

Ships WAIT_FOR_WEBHOOK end-to-end on OSS conductor: a faithful port of webhooks-oss/ from orkes-io/orkes-conductor#3612, plus the in-memory + 5 persistent backings needed to actually run it in any OSS deployment shape, plus retention of incoming_webhook_event rows. Closes #888.

Originally landed as two PRs (#1106 port + #1110 persistence); per leadership alignment they were folded into this one. Same commits, single review thread. #1110 auto-closed when its branch was fast-forwarded into this one.

User impact: First-time users can configure WAIT_FOR_WEBHOOK workflows, receive verified inbound events (HMAC/Slack/Stripe/Twitter/SendGrid/SignatureBased/HeaderBased), and have them dispatch into running workflows — on any supported OSS persistence shape (in-memory single-node, postgres, mysql, sqlite, redis, cassandra), not just dev mode.

Review-round changes

Six commits added after initial filing, addressing Viren's review:

Commit	Change	Why
893a990	Delete Tag.java from common	Tags are enterprise RBAC — TagsService/AutomaticTagService don't exist in OSS; the class had no callers other than WebhookConfig.tags
b986d65	Drop List<Tag> tags from WebhookConfig	Field was always null in OSS (nothing ever sets it); dead API surface that would mislead users
762e4c2	Delete EventMessage.java from common	Ported but never wired — IncomingWebhookService has no addEventMessage calls and OSS ExecutionDAO has no such method. Decision (per Viren): DLQ doesn't fit Conductor's design; timeoutSeconds on WAIT_FOR_WEBHOOK tasks is the correct negative-path mechanism
92e122d	Delete WebhookExecutionHistory + recordHistory() + lastRunWorkflowIdSize	Both Orkes and OSS carried // TODO Remove this on this field. Enterprise-only execution log embedded in config (anti-pattern); only implemented in webhooks-enterprise, not webhooks-oss
237701e	Drop WebhookConfig.getWorkflowNames()	Zero callers anywhere in OSS — duplicated getReceiverWorkflowNamesToVersions().keySet() with @JsonIgnore
a05447c	Map NonTransientException → 400 in ApplicationExceptionMapper	Signature verification failures were returning 500, which tells external senders (Stripe, GitHub, Slack) to retry. NonTransientException semantically means "this won't work no matter how many retries" — correct HTTP mapping is 400 Bad Request
5cdd389	Add Webhook.cancel() + remove(TaskModel, int) on all 6 backings	Ports orkes-io/orkes-conductor#3663: Webhook.start() puts a taskId into the correlation hash set but there was no symmetric removal when a workflow was terminated before a matching event arrived — set grew without bound. cancel() calls webhookTaskService.remove(TaskModel, int) (new overload on all 6 backings); WebhookTaskHashing.computeHashIfPresent() handles tasks cancelled before IN_PROGRESS as a safe no-op.

Orkes parity

Mirrors the layout from orkes-io/orkes-conductor#3612, which performed the webhooks/ → webhooks-oss/ + webhooks-enterprise/ split per the precedent set by scheduler in orkes-io/orkes-conductor#3529 and #1064.

Source files mirrored:

• orkes-conductor/webhooks-oss/src/main/java/io/orkes/conductor/webhook/**
• 9 supporting classes in orkes-conductor/common/ and orkes-conductor/core/
• 5 backend DAO + TaskService impls under orkes-conductor/{postgres,mysql,redis,cassandra}-persistence/

All io.orkes.conductor.* packages translated to org.conductoross.conductor.* per repo convention.

OSS-side additions (no Orkes parallel)

Required because Orkes' equivalents are tenant-aware or rely on infrastructure that doesn't exist in OSS:

• InMemoryWebhookDAO + InMemoryWebhookTaskService — @ConditionalOnMissingBean defaults for the single-node case. No Orkes parallel because Orkes always runs against persistent storage.
• WebhookConfigService + WebhookConfigResource — tenant-free CRUD at POST/PUT/GET/DELETE /api/metadata/webhook. OSS equivalent of the enterprise resource that uses AuditUtils + OrkesAuthentication.
• IncomingWebhookResource — tenant-free port at POST/GET /api/webhook/{id}. Drops orgId switching.
• WebhookWorker — OSS port of the enterprise worker. Plain @Component with @PostConstruct/@PreDestroy, no LifecycleAwareComponent, no MetricsCollector/Monitors, no OrkesRequestContext, no ExtendedEventExecution audit bookkeeping. Failures log instead of persist.
• WebhookTaskHashing (in core) — hash computation shared across all 6 WebhookTaskService impls. Orkes carries the body inline in every impl; OSS lifts it once. Behavioral parity across backings is enforced by call, not by 5 hand-maintained copies. Also fixes a latent NPE (matches validated before refname iteration removal).
• WebhookMatcherComputer (in core) — same pattern for the WorkflowDef → matchers transformation. All 6 DAO impls call it; each getMatchers() is a one-liner. Net –166 LoC vs. each impl carrying its own copy.
• Replay-protection SPI — WebhookVerifier.dedupKey(...) (default no-op) + WebhookDAO.tryRecordSignature(webhookId, signature, ttl) (default no-op). IncomingWebhookService rejects signature replays inside a 5-minute window. Backings that don't implement dedup retain prior behavior. No Orkes parallel; Orkes has no replay protection.

Backings landed

Backend	DAO + service	Schema	Retention
In-memory	InMemoryWebhookDAO + InMemoryWebhookTaskService	ConcurrentHashMap (default; single-node)	n/a
Postgres	PostgresWebhookDAO + PostgresWebhookTaskService	V16__webhook.sql (4 tables)	PostgresWebhookCleanupJob — WITH ... DELETE ... RETURNING
MySQL	MySQLWebhookDAO + MySQLWebhookTaskService	V10__webhook.sql (4 tables, InnoDB/utf8mb4)	MySQLWebhookCleanupJob — DELETE ... LIMIT N
SQLite	SqliteWebhookDAO + SqliteWebhookTaskService	V4__webhook.sql (4 tables)	SqliteWebhookCleanupJob — DELETE WHERE rowid IN (...)
Redis	RedisWebhookDAO + RedisWebhookTaskService	hash maps + sets via JedisProxy	RedisWebhookCleanupJob — HGETALL + parse + HDEL walker
Cassandra	CassandraWebhookDAO + CassandraWebhookTaskService	ensureTables() on construction	Table-level default_time_to_live (no separate job)

Shared design across backings

• Matcher recomputation on read. Every backing persists only the target-workflow versions snapshot (taken at config-create time so any expression evaluation is preserved). The actual matches criteria are recomputed from MetadataDAO on every getMatchers() call. WorkflowDef updates take effect without re-registering the webhook. Regression covered per backing by getMatchers_recomputesFromMetadataDAO_onEachCall_noStaleCache.
• Bean naming. All WebhookDAO beans register as webhookDAO and all WebhookTaskService beans as webhookTaskService. The in-memory impls use @ConditionalOnMissingBean(name="webhookDAO"/"webhookTaskService") — whichever persistence module is on the classpath wins; with none, in-memory provides the default.
• Cleanup property surface (cross-backing):
  • conductor.webhooks.cleanup.enabled (default true)
  • conductor.webhooks.cleanup.cron (default hourly "0 0 * * * *")
  • conductor.webhooks.cleanup.retention-duration (default PT168H / 7 days)
  • conductor.webhooks.cleanup.batch-size (default 1000) — SQL backings only
  • conductor.webhooks.cleanup.max-runtime (default PT60S) — SQL backings only
• Operational worker properties. conductor.webhook.worker.{threadCount,pollingInterval,pollBatchSize} via WebhookWorkerProperties.

Design departures from Orkes' source impls

1. No org_id anywhere — tables, queries, code paths. Orkes' schema bleeds org_id into 4 tables and dozens of WHERE clauses; OSS is tenant-free.
2. Matcher recomputation on read — Orkes' PostgresWebhookDAO persists pre-computed matchers and runs a scheduled refreshMatchers executor to handle WorkflowDef updates. OSS drops the cache, recomputes on every read.
3. No ConductorLimitsDAO quota enforcement — Orkes enterprise-only.
4. NotFoundException instead of Preconditions.checkNotNull for missing-id errors.
5. Schema rename — webhook_matchers (Orkes) → webhook_target_workflows (OSS), since OSS persists target workflows, not pre-computed matchers.
6. Cleanup pattern. Orkes' WebhookCleanupJob implements an Orkes-internal DataCleaner interface. OSS uses Spring @Scheduled directly with the property surface above.

Code rewrites beyond mechanical namespace translation

Port-side (webhooks-oss)

#	Where	Change	Why
1	IncomingWebhookService, TimeBasedUUIDGenerator	ApplicationException → NonTransientException	OSS retired ApplicationException
2	TimeBasedUUIDGenerator	Dropped generate(long), getOrgId(String), OrkesRequestContext lookups	OrkesRequestContext is Orkes-only tenant context
3	IncomingWebhookService.handleWebhook	Throws NotFoundException on unknown webhook id instead of returning null. Dropped ExecutionDAOFacade dep.	4xx contract correctness + dead code
4	WebhookConfigService.updateWebhook	Throws NotFoundException if existing config not found	Was NPE/500
5	WebhookConfigService.getWebhooks + WebhookConfigResource.getWebhook	toBuilder().secretValue(SECRET).build() instead of in-place setSecretValue("***")	In-memory DAO returns shared refs — in-place mutation corrupted persisted secret. Found via smoke.
6	IncomingWebhookResource, WebhookConfigResource	/webhook → /api/webhook, /metadata/webhook → /api/metadata/webhook. All @PathVariable use explicit ("id").	OSS convention + Spring requires explicit names without -parameters. Found via smoke.
7	WebhookTaskMapper	Package → org.conductoross.conductor.webhook.tasks.mapper	AI module precedent
8	WebhookConfigResource, WebhookConfigService	Drop AuditUtils, FeatureFlags, all @PreAuthorize/@PostFilter, OrkesAuthentication.getAuthenticatedUser. Tags (Tag.java, WebhookConfig.tags) fully removed — enterprise RBAC with no OSS backing.	Enterprise-only
9	webhooks-oss/build.gradle	spring-boot-starter-web + spring-boot-autoconfigure upgraded compileOnly → implementation. Stripe/SendGrid pinned via revStripe/revSendgrid.	Spring web needed at runtime; AGENTS.md pinning convention
10	StripeVerifier.verify	Null-guard on event.getApiVersion() before .split(...)	NPE on Stripe-shaped bodies that omit api_version. Orkes has identical code — needs mirror fix to keep parity.
11	SlackVerifier	Full rewrite: implements Slack's X-Slack-Signature: v0=<hmac_sha256(timestamp:body)> + X-Slack-Request-Timestamp with 5-minute replay window and constant-time compare.	Orkes' impl returns ErrorList.empty() once urlVerified=true — effectively unauthenticated post-handshake. OSS does real signing-secret verification. Orkes is genuinely vulnerable here; mirror-fix candidate.
12	ApplicationExceptionMapper	NonTransientException → 400 Bad Request	Was falling through to 500, causing external senders (Stripe, GitHub, Slack) to retry rejected events. "Non-transient" semantically is a client error.

Persistence-side

• core/WebhookTaskHashing.java validates matches before computing the hash. Fixes a latent NPE where removeIterationFromTaskRefName(null) would mask the missing-matches NonTransientException. Regression: InMemoryWebhookTaskServiceTest.put_missingMatchesField_throws.
• core/WebhookMatcherComputer.java centralizes the WorkflowDef → matchers loop. All 6 DAO impls + InMemory call it. Behavior unchanged from inline; eliminates 5 hand-maintained copies.
• PostgresWebhookDAOTest + siblings set spring.flyway.ignore-migration-patterns=*:missing to tolerate an orphan V10.1 in CI's shared testcontainers postgres (another test class applies V10.1 via experimental-queue-notify).
• CassandraWebhookDAO keeps a local objectMapper + toJson/readValue because CassandraBaseDAO's are package-private. Same workaround CassandraSchedulerDAO uses.

Tests

146 new test methods across 24 test classes + 1 property-style class (200 random reps per run), 0 failures.

Port (webhooks-oss): 78 tests

• Verifiers (HMAC/Slack/Stripe/Twitter/SendGrid/SignatureBased/HeaderBased) — lifted from Orkes
• TargetWorkflowCollector, idempotency substitution
• WebhookConfigService (9) — CRUD, conflict, secret sanitization (mutation regression), NotFound on update
• WebhookConfigResource (8) — validation, 404s, secret sanitization
• InMemoryWebhookDAO (9) — CRUD + matcher computation against mocked MetadataDAO
• InMemoryWebhookTaskService (6) — hash semantics, bucketing, missing-matches throws
• IncomingWebhookResource (2) — delegation
• Webhook system task (1) — put() registration
• WebhookWorker (9) — handleMessage paths, pollAndExecute
• StripeVerifier (3) — null api_version accepted, bad signature rejected, missing header rejected
• WebhooksOssEndToEndTest (4) — full bean graph: register → receive → enqueue → worker dispatch → workflow start / WAIT_FOR_WEBHOOK completion
• WebhookHashContractPropertyTest (2 × 100 reps) — property-style: hash-site agreement across InMemoryWebhookTaskService and WebhookHashingService.computeJsonHash

Persistence: 68 tests (most testcontainer-based)

Test class	Count	Local pass
PostgresWebhookDAOTest	8	(needs Docker)
PostgresWebhookTaskServiceTest	6	(needs Docker)
PostgresWebhookCleanupJobTest	4	(needs Docker)
MySQLWebhookDAOTest	6	(needs Docker)
MySQLWebhookTaskServiceTest	6	(needs Docker)
MySQLWebhookCleanupJobTest	3	(needs Docker)
SqliteWebhookDAOTest	7	✅
SqliteWebhookTaskServiceTest	6	✅
SqliteWebhookCleanupJobTest	3	✅
RedisWebhookDAOTest	7	(needs Docker)
RedisWebhookTaskServiceTest	6	(needs Docker)
RedisWebhookCleanupJobTest	4	(needs Docker)
CassandraWebhookDAOTest	7	(needs Docker)
CassandraWebhookTaskServiceTest	6	(needs Docker)

Smoke validation

Single-node (against :conductor-server-lite:bootRun)

The reproducible harness at nthmost-orkes/webhook-emitter/conductor-smoke registers a workflow, registers a webhook, starts the workflow, fires a signed event, polls until COMPLETED. Result:

  → HMAC_BASED ... PASS (2.23s)
  → SIGNATURE_BASED ... PASS (2.23s)
  → HEADER_BASED ... PASS (1.96s)
  → SLACK_BASED ... PASS (1.95s)
  → STRIPE ... PASS (1.97s)
  → TWITTER ... PASS (2.24s)
summary: 6/6 verifiers passed

Per-backend matrix (docker-compose + smoke)

webhook-emitter/conductor-smoke/matrix.sh brings up each backing's docker-compose stack against this branch's conductor:server image, then runs the 6-verifier smoke against it.

Backing	Result	Notes
Postgres	✅ 6/6 PASS	full webhook flow through real Postgres DAO
Redis	✅ 6/6 PASS	full webhook flow through real Redis DAO
SQLite	✅ 6/6 PASS	server's intrinsic default — local :conductor-server-lite:bootRun resolves conductor.db.type=sqlite from application.properties, so the local bootRun smoke is the sqlite functional pass. Also covered by :conductor-sqlite-persistence:test (16/16 green locally).
MySQL	⚠️ docker bring-up fails	pre-existing flyway bean wiring issue at Spring startup — fails before webhook code loads; unrelated to this PR. Testcontainer tests will validate via MySQLWebhookDAOTest etc.
Cassandra	⚠️ docker bring-up fails	missing com.codahale.metrics.JmxReporter transitive dep in server classpath; fails on existing cassandraMetadataDAO bean, not webhook code. Testcontainer tests will validate via CassandraWebhookDAOTest etc.

The mysql/cassandra docker failures are upstream of any WebhookDAO code path — both fail at Spring bootstrap of pre-existing beans (mySqlMetadataDAO / cassandraMetadataDAO). Worth filing separately against conductor-oss/conductor's docker image setup, but they don't gate this PR.

Full battery (conductor-server-lite SQLite, loki)

8/8 harnesses passed against this branch's JAR on a remote host (loki, Ubuntu 24.04, Java 21, SQLite persistence):

Harness	Result	Key metric
smoke.py — verifier matrix	✅ 6/6	HMAC/Sig/Header/Slack/Stripe/Twitter all COMPLETED
negative_smoke.py — failure modes	✅ 5/5	timeout→TIMED_OUT, bad_sig→400, unknown→404, no_match→RUNNING, replay→400 second fire
gap_smoke.py — coverage gaps	✅ 3/3	workflowsToStart auto-starts workflow, HTTP task returns 200, SENDGRID ECDSA verified
composite_smoke.py — all system tasks	✅ 14/14 checks	SET_VARIABLE, JSON_JQ_TRANSFORM, SWITCH, FORK_JOIN, DO_WHILE, SUB_WORKFLOW, FORK_JOIN_DYNAMIC, JOIN, INLINE, TERMINATE all COMPLETED
multi_verifier_smoke.py — 3 configs, 1 wf	✅ 3/3	HMAC + Signature + Header, each fires fresh instance
bucket_smoke.py — N=20 fan-out	✅ 20/20	all completed from single event, median 0.47s
replay_smoke.py — observation	✅ exits 0	second fire rejected (400), Scenario B behavior documented
load_smoke.py — 100 wf, concurrency=10	✅ 100/100	8.56/s throughput, p50=1019ms, p95=1276ms, 0 lost

Verified

• ./gradlew :conductor-webhooks-oss:test — 78/78 unit + 200/200 property reps green
• ./gradlew :conductor-sqlite-persistence:test --tests '*Webhook*' — 16/16 green
• ./gradlew :conductor-{core,webhooks-oss,postgres,mysql,sqlite,redis,cassandra}-persistence:compileJava — clean
• ./gradlew :conductor-server:compileJava — clean (full server tree)
• ./gradlew :conductor-*:spotlessCheck (all touched modules) — clean
• Local bootRun smoke against :conductor-server-lite — full webhook → workflow chain succeeded
• Per-backend matrix smoke: postgres + redis 6/6
• Full 8/8 battery on remote host (loki, SQLite) — see table above

What's NOT here (and why)

• DLQ / EventMessage: decided against. Orkes ships EventMessage/EventMessageService as an enterprise-only observability layer (Redis-backed, org-scoped). It doesn't belong in OSS, and the decision from review is that timeoutSeconds on WAIT_FOR_WEBHOOK tasks is the correct negative-path mechanism — consistent with Conductor's design philosophy. EventMessage.java was removed.

Known follow-ups (out of scope, file as separate PRs)

• Fix mysql-persistence docker image: flyway bean wiring at startup (existing issue, surfaces under matrix harness)
• Fix cassandra-persistence docker image: missing com.codahale.metrics.dropwizard-metrics-jmx transitive dep
• Docs page with verified curl examples for /api/metadata/webhook and /api/webhook/{id}

Test plan

• CI green
• CI green on all 5 testcontainer-based persistence test suites
• Smoke: POST /api/webhook/{id} → verifier runs → WebhookTaskMapper dispatches → workflow completes
• Verify conductor.webhook.worker.* properties bind via WebhookWorkerProperties
• Verify conductor.webhooks.cleanup.* properties bind via Spring environment

Stats

91 files changed, ~9,600 LoC added net, 23 commits. Originally split as #1106 (port, ~5.5K LoC, 54 files) + #1110 (persistence, ~4.3K LoC, 39 files); folded per leadership alignment. 6 additional cleanup commits from review round.

[nthmost-orkes]

Re-validated end-to-end: built conductor at 84ea71859 and ran the smoke harness at nthmost-orkes/webhook-emitter@16fd7e7 against it — 6/6 verifiers PASS (HMAC, signature, header, slack, stripe, twitter). Each fires a real signed event → conductor verifier → WAIT_FOR_WEBHOOK task completes → workflow reaches COMPLETED.

[v1r3n]

see comments

changes were made

[chrishagglund-ship-it]

I can't provide meaningful review of the code quality. Looks great to me on a quick scan over 11k lines. Claude thinks it's pretty good too, noting a few risky things but mentioning that they're documented.

I ran through an example scenario with the emitter as well as some manual testing with it and a manually defined wf, stuff worked.

[nthmost-orkes]

I can't provide meaningful review of the code quality. Looks great to me on a quick scan over 11k lines. Claude thinks it's pretty good too, noting a few risky things but mentioning that they're documented.

I ran through an example scenario with the emitter as well as some manual testing with it and a manually defined wf, stuff worked.

Thanks, I appreciate you just trying the stuff out, for sure!