Streaming daemon: Phase 2 layer 2 — live ingestion (closes #816, #808)#821
Draft
chowbao wants to merge 4 commits into
Draft
Streaming daemon: Phase 2 layer 2 — live ingestion (closes #816, #808)#821chowbao wants to merge 4 commits into
chowbao wants to merge 4 commits into
Conversation
This was referenced Jun 24, 2026
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
730ba41 to
df8ec80
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
380bae2 to
c2398ad
Compare
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
df8ec80 to
17b5c39
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
c2398ad to
21ae7ea
Compare
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
17b5c39 to
145c1cc
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
21ae7ea to
1adb234
Compare
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
e15575a to
bc56b0a
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
3fac870 to
da43f9f
Compare
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
bc56b0a to
440443b
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
da43f9f to
b89d910
Compare
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
440443b to
c4944fb
Compare
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
b89d910 to
07ea19f
Compare
chowbao
added a commit
that referenced
this pull request
Jun 25, 2026
…hanges Rebased onto the updated #820 and propagated #817's API changes into the Phase 2 live-ingestion/daemon layer: - window -> tx-hash index rename + key prefix index: -> txhash_index: (TxHashIndexCoverage.Index, Catalog.txhashIndex), Catalog.Get/Has -> get/has, config sections regrouped (cfg.Retention/Layout/Storage/Ingestion), pins via PinLayout. - daemon.go merge: kept #821's live-ingestion wiring (LifecycleConfig + Core) and deduped the HotProbe line (#821's Phase-2 wiring already set it, so #820's HotProbe fix is redundant here). - removed the #819 cold-only catch-up E2E (TestRunDaemon_CatchUpMaterializes...) + its someTxBackend/oneTxLCMBytes helpers: #821's daemon now requires Boundaries.Core and runs a continuous live loop, so a cold-only "catch up then return" test can't fit — and TestE2E_DaemonLifecycle covers it end to end. Mechanical propagation only; build/vet/test -short green (the heavy lifecycle E2E stays -short-gated).
…ing (closes #816, closes #808) Wires the hot tier into the daemon: the full startStreaming (open the resume hot DB, launch the lifecycle goroutine, run the live ingestion loop following captive core), the production boundaries (captive-core opener, bulk-backend tip adapter), the lifecycle-config/HotProbe assembly, and the hot watermark refinement (startup passes the real probe so resume re-derives from the last synced batch). Completes the full-history streaming daemon — backfill + live ingest + freeze/discard/prune. Verified by the lifecycle E2E (first-start -> ingest -> freeze -> lookup -> restart-resume re-derivation -> prune). Closes #816. Closes #808.
Doc/comment fixes and dead-code removal in the Phase 2 layer 2 wiring; no behavior change (daemon.go/startup.go are comment-only apart from one unused parameter rename). - daemon.go buildProductionBoundaries: the "Core:" doc bullet described the old NewCaptiveCoreStream/OpenLedgerStream/RawLedgers model; rewrite it to the actual CoreOpener.OpenCore (PrepareRange + indexed GetLedger), fold the captive-core config deferral into the existing #772 TODO, and drop the now-overstated "fully wired". Rename the unused ctx parameter to _. - startup.go: condense the two lifecycle-goroutine comment blocks (the per-iteration-ctx rationale and the cancel+join defer) to the load-bearing "why" without the duplicated invariant restatement; fix a grammar slip. - e2e_test.go: the header named a non-existent CoreStreamOpener interface (it is CoreOpener); remove the dead e2eGetter.maxSeq field (set in OpenCore, never read — GetLedger keys off the frames map); fix the trailing helper comment that claimed snapshotFreezeCount was added here (it lives in the base). - daemon_test.go: the capturedBuild doc opened with a non-existent "fakeBoundaries" symbol and stale "stream" vocabulary; name the type and say "getter".
The serve+ingest rework dropped TestStartStreaming_ServeReadsErrorSurfaces, leaving the ServeReads-error return path untested — and that path now does more than before: it must close the already-opened resume hot DB so a restart can reacquire the rocksdb LOCK. Restore the test adapted to the new flow: a fake core with a blocking getter so ServeReads is reached, then assert the wrapped "serve reads" error surfaces (not classified clean), core was started, and the resume hot DB is reopenable afterward (its LOCK was released).
chowbao
force-pushed
the
streaming-phase2-lifecycle
branch
from
c4944fb to
aeca6a0
Compare
Rebased the live-ingestion capstone onto the reorganized #820 and propagated: - qualify moved symbols (geometry./catalog.) in daemon.go, startup.go, e2e_test.go - window->tx-hash-index + RetentionGate->RetentionFloor renames; cat.layout->Layout(), cat.Has->public HotState shim, .IndexFilePath->.TxHashIndexFilePath - config regroup: cfg.Streaming.CaptiveCoreConfig -> cfg.Ingestion.CaptiveCoreConfig - restored #821's daemon_test.go (drops the cold-only catch-up test the full daemon supersedes; adds the supervise/backend-tip/boundaries tests) + the HotProbe/Core wiring - avoided the txhash_txhash_index find-replace corruption (was only in the dropped restack) build + vet + go test -short green EXCEPT the lifecycle E2E, whose generated TOML still uses the pre-regroup [streaming]/[backfill] schema (follow-up; per maintainer the stack will be re-rebased).
chowbao
force-pushed
the
streaming-phase2-ingestion
branch
from
07ea19f to
925993c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #816. Closes #808 — Phase 2 (Live ingestion + lifecycle), layer 2 of 2 (capstone). Stacked on #PR2A (
streaming-phase2-lifecycle).Wires the hot tier into the daemon — completing the full-history streaming daemon:
startStreaming: open the resume hot DB, launch the lifecycle goroutine, run the live ingestion loop following captive core (one atomic syncedWriteBatchper ledger across all three types)HotProbeassemblyCompleting this yields the whole daemon: backfill + live ingest + freeze/discard/prune.
Verified by the lifecycle E2E (first-start → ingest → freeze → tx-hash lookup → restart-resume re-derivation → prune), passing non-short (~101s), plus
go test -shortacross the fullfullhistorytree.Verification:
go build+go vet+go test -shortgreen on./cmd/stellar-rpc/internal/fullhistory/...(cgo RocksDB toolchain). Note: the fullcmd/stellar-rpcbinary link requires the Rustlibpreflight/libxdr2json(CImake build-libs);go vet ./cmd/stellar-rpc/type-checks the entrypoint locally.golangci-lintruns in CI.Stack:
streaming-phase2-ingestion→streaming-phase2-lifecycle