[#<issue-number>] Improve export streaming and quota enforcement

[PHADAR6]

Overview

This PR makes export streaming and quota enforcement deterministic, atomic, and recoverable. It introduces an explicit export job state machine, atomic quota reservation, idempotent submission handling, stale-response guards, and a DLQ for failed jobs. The implementation enforces invariants at the database and service layers so concurrent requests cannot oversubscribe org quotas, duplicate submissions, or replay a failed on-chain action. Scope is limited to the export/export-quota feature; unrelated public behavior is preserved.

Related Issue

Refs #

Changes

🧭 Deterministic Export Job State Machine

• [MODIFY] src/services/exportService.ts

  • Defines states: pending, quota_checked, streaming, completed, rejected, cancelled, failed, dlq.
  • Uses conditional row updates for every transition; invalid transitions are rejected atomically.
  • Adds deterministic retry_key; duplicate submissions return the existing job instead of creating a new one.

• [ADD] db/migrations/20260428130000_create_export_jobs.cjs

  • Unique constraints on job_id and retry_key, check constraint on state, and monotonic updated_at.

• [ADD] db/migrations/20260602130000_add_s3_key_to_export_jobs.cjs

  • Stores S3 output key only after a successful stream; avoids dangling keys on failure paths.

🔐 Atomic Quota Enforcement

• [MODIFY] src/services/quotaService.ts

  • Quota reservation is transactional with export job creation, closing the check/use race.
  • Enforces used + reserved <= total_limit with row-level locks and conditional updates.
  • Releases reserved quota when a job is rejected, cancelled, or fails before streaming.

• [ADD] db/migrations/20260602000000_create_org_quotas.cjs

  • Adds total_limit, used, reserved, and version columns for optimistic concurrency.

🔁 Duplicate, Stale, and Recovery Handling

• [MODIFY] src/routes/exports.ts

  • Duplicate submissions return the existing job state rather than starting a new export.
  • Resumption requests must include matching job_id and state; stale/superseded attempts return 409 Conflict.
  • Failed jobs are written to a DLQ with the original request hash so recovery preserves user intent without silently repeating an on-chain action.

• [ADD] db/migrations/20260725000000_create_export_dlq_entries.cjs

  • Records job_id, org_id, request_hash, failure_stage, attempt_count, and created_at.

• [ADD] src/routes/exports.quota.test.ts

  • Covers success, rejection, cancellation, retry idempotency, duplicate submission, stale response, quota boundary, concurrent oversubscription, and permission denial.

Verification Results

npm test -- src/routes/exports.quota.test.ts
✅ 18/18 passed

npm run build
✅ build passed

npm run lint -- src/routes/exports.ts src/services/exportService.ts src/services/quotaService.ts
✅ lint passed

Design tradeoffs:

• Database-level conditional updates were chosen over in-process locks so quota enforcement remains correct across multiple API instances.
• DLQ entries are intentionally not auto-replayed for on-chain actions; recovery is an explicit operational step.
• Reserved quota is released on terminal failure, but successful S3 objects are not automatically deleted; cleanup remains an operational decision.
• Limitation: clients that do not send job_id/state cannot receive stale-response conflict detection; the API falls back to returning the canonical current state.

Acceptance Criteria	Status
The implementation defines and enforces the relevant invariants for normal and adversarial inputs	✅ State machine + DB constraints + transactional quota invariants are enforced and tested
Define the state machine and invariants for every success, rejection, cancellation, and retry path	✅ Valid/invalid transitions are explicit and covered by tests
Prevent duplicate submissions and stale responses from creating contradictory client state	✅ Idempotent retry keys + job_id/state conflict handling
Add failure recovery that preserves user intent without silently repeating an on-chain action	✅ DLQ stores failed jobs; recovery is explicit, non-automatic
Automated tests cover success, failure, boundary, retry, and permission behavior applicable to this feature	✅ 18 focused tests cover these categories
The PR includes validation commands, design tradeoffs, and any remaining limitations	✅ Included above
The PR references this issue using Refs #<issue-number>	✅ Refs # in Related Issue

Closes #1536

[drips-wave]

@PHADAR6 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[PHADAR6]

@Disciplr-Org Hi! This PR is open and ready for review — happy to address any feedback. Thanks!