feat: launch ingesters per enabled network in the bootstrap (#117) - #155
Conversation
…56#117) src/index.ts previously started exactly one instance of each venue ingester (SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network was active at startup. To index more than one network in the same process, each ingester now runs once per enabled network. - src/network/enabledNetworks.ts (new): getEnabledNetworks() reads the comma-separated ENABLED_NETWORKS env var (e.g. "testnet,mainnet"), trims, lowercases, dedupes and filters to known network names. When unset — or nothing valid remains — it falls back to [activeNetwork], so a single-network deployment starts exactly one instance of each ingester. - src/index.ts: loops over the enabled networks and starts each ingester with its network context. restartIngester now captures the specific (venue, network) pair, so a crash in one instance restarts only that instance and cannot affect the others. - src/ingesters/snapshot.ts: appendSnapshots() / startSnapshotIngester() take a network arg. price_snapshots rows are network-scoped via the (network, pair, ts) PK, so a single instance could only ever snapshot the active network — each enabled network now gets its own loop. - src/ingest/venues/aquarius.ts: startAquariusIngester() / ingestAquariusPair() take a network arg and resolve the per-network Aquarius block via getNetworkConfig(network). - .env.example: documents ENABLED_NETWORKS. - src/__tests__/enabledNetworks.test.ts (new): covers the fallback, explicit multi-network lists, whitespace/casing normalization, dedupe and filtering of unrecognized names. closes Miracle656#117 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ky4A59maw6dB2CptXyhdW
Miracle656
left a comment
There was a problem hiding this comment.
Approved and merging.
This is the right remaining scope for #117. Three of the five ingesters already took a network argument after #139; this adds the two that didn't (Snapshot, Aquarius) and, most importantly, the bootstrap loop that actually uses them. Without that loop the per-network plumbing underneath was inert.
This also closes a live bug, not a hypothetical one. price_snapshots is keyed (network, pair, ts), but appendSnapshots read activeNetwork directly — so once a second network was enabled, one of the two chains would silently record no snapshots at all. Not an error, not an empty table: just a chart that quietly flatlines for one network while the other looks fine. Threading network through appendSnapshots/startSnapshotIngester is exactly the fix.
Fault isolation keyed by the pair, not the venue — restartIngester(name, network, fn) — is the detail that makes the acceptance criterion real. Keying on venue alone would have meant a Soroswap crash on mainnet restarting the testnet Soroswap loop too, or worse, both instances racing after one restart. The log line carries ${name}/${network} for the same reason: 'Soroswap ingester crashed' is not actionable when two of them are running.
The getEnabledNetworks fallback chain is careful in the way that matters: unset, blank, all-whitespace, and all-unrecognised all collapse to [activeNetwork], so a single-network deployment starts five loops and not ten — the third acceptance criterion — and a typo in ENABLED_NETWORKS degrades to today's behaviour instead of starting zero ingesters. Seven tests covering exactly those edges.
Verified locally: tsc --noEmit clean; full suite 312 passed / 1 skipped on two consecutive runs. One earlier run had networkVenueConfig.test.ts time out, but it passes 3/3 in isolation and main measures the same flake — that file has a 5s timeout and the cold suite reports ~90s of import time. Not from this PR.
One cross-repo nit, nothing to change here: wraith calls the same concept NETWORKS while this uses ENABLED_NETWORKS. Yours is the clearer name; I'll note the divergence in the deploy docs rather than churn either.
Thanks — this was the keystone of the dual-network work on Lens.
Summary
src/index.tspreviously started exactly one instance of each of the five venue ingesters (SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network was active at startup. To index more than one network in the same process, each ingester now runs once per enabled network.The per-network client factories (#139) and per-network venue config (#138) this builds on are already on
main, so this diff is just the orchestration change.Changes
src/network/enabledNetworks.ts(new):getEnabledNetworks()reads the comma-separatedENABLED_NETWORKSenv var (e.g.testnet,mainnet), trims, lowercases, dedupes, and filters to known network names. When unset — or when nothing valid remains after filtering — it falls back to just the currently active network (STELLAR_NETWORK), so a single-network deployment starts exactly one instance of each ingester rather than one per network.src/index.ts: loops over the enabled networks and starts SDEX, AMM, Soroswap, Snapshot, and Aquarius once per network.restartIngester's retry closure now captures the specific(venue, network)pair, so a crash in, say, Soroswap on mainnet only restarts that one instance and cannot affect the ingesters running on testnet.src/ingesters/snapshot.ts:appendSnapshots()/startSnapshotIngester()now take anetworkargument.price_snapshotsrows are network-scoped via the(network, pair, ts)primary key, so a single shared instance could only ever snapshot the active network — each enabled network now gets its own loop.src/ingest/venues/aquarius.ts:startAquariusIngester()/ingestAquariusPair()take anetworkargument and resolve the per-network Aquarius block viagetNetworkConfig(network)(enabled flag + API URL)..env.example: documentsENABLED_NETWORKS.src/__tests__/enabledNetworks.test.ts(new): covers the default fallback, explicit multi-network lists, whitespace/casing normalization, deduplication, and filtering out unrecognized network names.Acceptance criteria (#117)
restartIngesterkeyed on(venue, network))ENABLED_NETWORKSunset →[activeNetwork])Testing
npx tsc --noEmitpassesnpx vitest runpasses — 312 passed, 1 skipped (7 new tests inenabledNetworks.test.ts)closes #117
🤖 Generated with Claude Code