Skip to content

feat: launch ingesters per enabled network in the bootstrap (#117) - #155

Merged
Miracle656 merged 1 commit into
Miracle656:mainfrom
gideonpius7-design:feat/issue-117-per-network-ingesters
Sep 1, 2026
Merged

feat: launch ingesters per enabled network in the bootstrap (#117)#155
Miracle656 merged 1 commit into
Miracle656:mainfrom
gideonpius7-design:feat/issue-117-per-network-ingesters

Conversation

@gideonpius7-design

Copy link
Copy Markdown
Contributor

Summary

src/index.ts previously started exactly one instance of each of the five venue ingesters (SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network was active at startup. To index more than one network in the same process, each ingester now runs once per enabled network.

The per-network client factories (#139) and per-network venue config (#138) this builds on are already on main, so this diff is just the orchestration change.

Changes

  • src/network/enabledNetworks.ts (new): getEnabledNetworks() reads the comma-separated ENABLED_NETWORKS env var (e.g. testnet,mainnet), trims, lowercases, dedupes, and filters to known network names. When unset — or when nothing valid remains after filtering — it falls back to just the currently active network (STELLAR_NETWORK), so a single-network deployment starts exactly one instance of each ingester rather than one per network.
  • src/index.ts: loops over the enabled networks and starts SDEX, AMM, Soroswap, Snapshot, and Aquarius once per network. restartIngester's retry closure now captures the specific (venue, network) pair, so a crash in, say, Soroswap on mainnet only restarts that one instance and cannot affect the ingesters running on testnet.
  • src/ingesters/snapshot.ts: appendSnapshots() / startSnapshotIngester() now take a network argument. price_snapshots rows are network-scoped via the (network, pair, ts) primary key, so a single shared instance could only ever snapshot the active network — each enabled network now gets its own loop.
  • src/ingest/venues/aquarius.ts: startAquariusIngester() / ingestAquariusPair() take a network argument and resolve the per-network Aquarius block via getNetworkConfig(network) (enabled flag + API URL).
  • .env.example: documents ENABLED_NETWORKS.
  • src/__tests__/enabledNetworks.test.ts (new): covers the default fallback, explicit multi-network lists, whitespace/casing normalization, deduplication, and filtering out unrecognized network names.

Acceptance criteria (#117)

  • Each ingester runs once per enabled network
  • Restart logic isolates per-network instances (restartIngester keyed on (venue, network))
  • Single-network mode still starts five loops, not ten (ENABLED_NETWORKS unset → [activeNetwork])

Testing

  • npx tsc --noEmit passes
  • npx vitest run passes — 312 passed, 1 skipped (7 new tests in enabledNetworks.test.ts)

closes #117

🤖 Generated with Claude Code

…56#117)

src/index.ts previously started exactly one instance of each venue ingester
(SDEX, AMM, Soroswap, Snapshot, Aquarius), always bound to whatever network
was active at startup. To index more than one network in the same process,
each ingester now runs once per enabled network.

- src/network/enabledNetworks.ts (new): getEnabledNetworks() reads the
  comma-separated ENABLED_NETWORKS env var (e.g. "testnet,mainnet"), trims,
  lowercases, dedupes and filters to known network names. When unset — or
  nothing valid remains — it falls back to [activeNetwork], so a
  single-network deployment starts exactly one instance of each ingester.
- src/index.ts: loops over the enabled networks and starts each ingester
  with its network context. restartIngester now captures the specific
  (venue, network) pair, so a crash in one instance restarts only that
  instance and cannot affect the others.
- src/ingesters/snapshot.ts: appendSnapshots() / startSnapshotIngester()
  take a network arg. price_snapshots rows are network-scoped via the
  (network, pair, ts) PK, so a single instance could only ever snapshot the
  active network — each enabled network now gets its own loop.
- src/ingest/venues/aquarius.ts: startAquariusIngester() / ingestAquariusPair()
  take a network arg and resolve the per-network Aquarius block via
  getNetworkConfig(network).
- .env.example: documents ENABLED_NETWORKS.
- src/__tests__/enabledNetworks.test.ts (new): covers the fallback, explicit
  multi-network lists, whitespace/casing normalization, dedupe and filtering
  of unrecognized names.

closes Miracle656#117

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ky4A59maw6dB2CptXyhdW

@Miracle656 Miracle656 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved and merging.

This is the right remaining scope for #117. Three of the five ingesters already took a network argument after #139; this adds the two that didn't (Snapshot, Aquarius) and, most importantly, the bootstrap loop that actually uses them. Without that loop the per-network plumbing underneath was inert.

This also closes a live bug, not a hypothetical one. price_snapshots is keyed (network, pair, ts), but appendSnapshots read activeNetwork directly — so once a second network was enabled, one of the two chains would silently record no snapshots at all. Not an error, not an empty table: just a chart that quietly flatlines for one network while the other looks fine. Threading network through appendSnapshots/startSnapshotIngester is exactly the fix.

Fault isolation keyed by the pair, not the venuerestartIngester(name, network, fn) — is the detail that makes the acceptance criterion real. Keying on venue alone would have meant a Soroswap crash on mainnet restarting the testnet Soroswap loop too, or worse, both instances racing after one restart. The log line carries ${name}/${network} for the same reason: 'Soroswap ingester crashed' is not actionable when two of them are running.

The getEnabledNetworks fallback chain is careful in the way that matters: unset, blank, all-whitespace, and all-unrecognised all collapse to [activeNetwork], so a single-network deployment starts five loops and not ten — the third acceptance criterion — and a typo in ENABLED_NETWORKS degrades to today's behaviour instead of starting zero ingesters. Seven tests covering exactly those edges.

Verified locally: tsc --noEmit clean; full suite 312 passed / 1 skipped on two consecutive runs. One earlier run had networkVenueConfig.test.ts time out, but it passes 3/3 in isolation and main measures the same flake — that file has a 5s timeout and the cold suite reports ~90s of import time. Not from this PR.

One cross-repo nit, nothing to change here: wraith calls the same concept NETWORKS while this uses ENABLED_NETWORKS. Yours is the clearer name; I'll note the divergence in the deploy docs rather than churn either.

Thanks — this was the keystone of the dual-network work on Lens.

@Miracle656
Miracle656 merged commit 84111e6 into Miracle656:main Sep 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Launch ingesters per network in the bootstrap

2 participants