Skip to content

feat: ephemeral-document cleanup (per-workflow blob + Temporal purge) - #203

Merged
alex-struk merged 1 commit into
developfrom
feat/ephemeral-document-cleanup
Jun 24, 2026
Merged

feat: ephemeral-document cleanup (per-workflow blob + Temporal purge)#203
alex-struk merged 1 commit into
developfrom
feat/ephemeral-document-cleanup

Conversation

@alex-struk

Copy link
Copy Markdown
Collaborator

Summary

Adds an opt-in, per-workflow ephemeral policy. When a workflow's config declares metadata.ephemeral, documents it processes are purged once they reach a terminal status — blob files and/or the Temporal execution record are deleted per the policy — while the OCR result in Postgres is retained so clients can still poll it.

Configured entirely on the workflow: no global flag, no per-group setting.

metadata.ephemeral values

Value Effect
true delete both blob files and the Temporal execution record
{ "files": true, "temporalRecord": false } delete only the targets set to true
false / absent not ephemeral — nothing deleted (default)

What's included

  • EphemeralDocumentCleanupService@Cron(EVERY_MINUTE) janitor; per-document failures isolated; idempotent (safe to retry).
  • TemporalClientService.deleteWorkflowExecution — idempotent on NOT_FOUND.
  • documents.purged_at marker (stops reprocessing; row + ocr_results kept) + partial index documents_purge_scan_idx (workflow_config_id, status) WHERE purged_at IS NULL so the scan stays cheap at high volume.
  • DB query matches any opted-in target and returns each doc's policy.
  • Docs: docs-md/ephemeral-document-cleanup.md.

Migrations

  • 20260623000000_add_document_purged_at
  • 20260624000000_add_documents_purge_index (partial index — Prisma can't express it in schema; managed via raw SQL)

Verification

  • 201 backend tests pass; type-check + biome clean.
  • Verified locally end-to-end against the real OCR pipeline:
    • ephemeral: true → blobs and Temporal record deleted, OCR text retained.
    • { files: true, temporalRecord: false } → blobs deleted, Temporal record kept, OCR text retained.
    • In-progress (non-terminal) and non-ephemeral docs left untouched.

🤖 Generated with Claude Code

Opt-in, per-workflow ephemeral policy. When a workflow declares
metadata.ephemeral, documents it processes are purged once they reach a
terminal status — blob files and/or the Temporal execution record are
deleted per the policy — while the OCR result in Postgres is retained.

- metadata.ephemeral: true | { files, temporalRecord } for per-target control
- EphemeralDocumentCleanupService: @Cron(EVERY_MINUTE) janitor, gated solely
  by workflow config (no global flag, no per-group setting)
- TemporalClientService.deleteWorkflowExecution (idempotent on NOT_FOUND)
- documents.purged_at marker + partial index documents_purge_scan_idx
  (WHERE purged_at IS NULL) to keep the janitor scan cheap at high volume
- docs-md/ephemeral-document-cleanup.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alex-struk
alex-struk marked this pull request as ready for review June 24, 2026 18:22
@alex-struk
alex-struk merged commit 9ac4bf2 into develop Jun 24, 2026
12 checks passed
@alex-struk
alex-struk deleted the feat/ephemeral-document-cleanup branch June 24, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant