fix(backend): drain BullMQ workers on graceful shutdown (#1347)

[Kenlachy]

Problem

Kubernetes sends SIGTERM during rolling deploys, but neither the backend nor the oracle called app.enableShutdownHooks(). Without it, NestJS never fires onApplicationShutdown on any provider, so in-flight webhook deliveries and randomness jobs are killed mid-execution rather than finishing or being returned to the queue.

Solution

Backend (backend/src/main.ts)

• Added app.enableShutdownHooks() before app.listen() so NestJS reacts to SIGTERM

Backend worker (webhook-delivery.worker.ts)

• Implements OnApplicationShutdown — calls (this as any).worker.close() which tells BullMQ to stop picking up new jobs and wait for in-flight deliveries to finish (or fail) before closing the Redis connection

Oracle (oracle/src/main.ts)

• Added app.enableShutdownHooks() before app.listen()

Oracle worker (randomness.worker.ts)

• Implements OnApplicationShutdown — injects the Bull queue via @InjectQueue and calls this.queue.close() on shutdown to drain in-flight randomness jobs

Kubernetes (k8s/deployment.yaml — both backend & oracle)

• Increased terminationGracePeriodSeconds from 30 → 45 to comfortably exceed the longest job (10s webhook HTTP timeout + retries for backend; on-chain VRF submission time for oracle)

Tests

Test	File	What it verifies
Unit	webhook-delivery.worker.spec.ts	onApplicationShutdown() calls worker.close(); doesn't throw if worker is undefined
Unit	randomness.worker.spec.ts	onApplicationShutdown() calls queue.close()
E2E	graceful-shutdown.e2e-spec.ts	Spins up a real BullMQ worker against Redis, enqueues slow jobs, triggers app.close(), asserts all in-flight jobs completed before shutdown resolved

How it works end-to-end

1. Kubernetes sends SIGTERM to the pod
2. NestJS lifecycle sees SIGTERM (because enableShutdownHooks() is enabled)
3. NestJS calls onApplicationShutdown() on every provider that implements it
4. Workers call queue.close() / worker.close() which:
   • Stops picking up new jobs
   • Waits for in-flight jobs to complete
   • Closes the Redis connection
5. NestJS finishes its own teardown
6. Process exits — all within the 45-second terminationGracePeriodSeconds

Files changed

File	Change
backend/src/main.ts	+app.enableShutdownHooks()
backend/src/queues/webhook-delivery.worker.ts	+OnApplicationShutdown interface + method
backend/src/queues/webhook-delivery.worker.spec.ts	+2 shutdown tests
backend/k8s/deployment.yaml	terminationGracePeriodSeconds: 30 → 45
backend/test/graceful-shutdown.e2e-spec.ts	New e2e test for SIGTERM during active jobs
oracle/src/main.ts	+app.enableShutdownHooks()
oracle/src/queue/randomness.worker.ts	+OnApplicationShutdown + @InjectQueue
oracle/src/queue/randomness.worker.spec.ts	+mock queue provider + shutdown test
oracle/k8s/deployment.yaml	terminationGracePeriodSeconds: 30 → 45
Closes #1347

[vercel]

@github-actions[bot] is attempting to deploy a commit to the otaiki1's projects Team on Vercel.

A member of the Team first needs to authorize it.

[Otaiki1]

FIX CONFLICTS

[Kenlachy]

fixed