Skip to content

Streaming workflow design (catch-up, live ingestion, freeze & prune) #722

Description

@karthikiyer56

TL;DR

Design the full-history daemon's streaming workflow — one daemon, one mode that catches up from backfill on startup, ingests live ledgers, and maintains the retention window by freezing completed data and pruning what's aged out. Query serving is the only thing out of scope (#770).

What the design covers

The daemon's lifecycle:

  • Catch up on startup by running backfill as a subroutine, up to the live tip.
  • Ingest live ledgers into hot storage.
  • Freeze completed chunks/indexes to immutable files and prune what's past the retention window.

…and staying correct across crashes and restarts.

Out of scope

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

Status
Needs Review

Relationships

None yet

Development

No branches or pull requests

Issue actions