Lp share token tracking - #156
Conversation
…hema and indexer utilities
|
@githoboman Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
There was a problem hiding this comment.
The schema, indexing wiring, reorg-aware rollbackToLedger cleanup, and idempotent eventId-keyed upsert here are all well done, and modelling deposits as a mint-to-provider / withdrawals as a burn-from-provider mirrors decoder.ts nicely. One correctness issue needs fixing before merge:
mint/burn are treated as LP-share events for every contract, so the table over-captures.
const DEPOSIT_EVENTS = new Set(["deposit", "mint"]);
const WITHDRAW_EVENTS = new Set(["withdraw", "burn"]);pollOnce feeds the full event batch to parseLpShareEvents(events), and there's no filter narrowing to pool contracts. deposit/withdraw are pool-specific and safe — but mint/burn are generic SEP-41 events emitted by every token (stablecoin issuers, NFT-ish tokens, ordinary SACs). As written, every USDC mint, every token burn, etc. gets written into LpShareTransfer with poolId = contractId and double-stored alongside its normal token-transfer row. That defeats the point of a table meant to surface liquidity-pool shares specifically.
Options to fix:
- Simplest & safe: drop
mint/burnfrom the recognized set and rely only on the explicitdeposit/withdrawdialect (Soroswap/Phoenix/etc.). You lose pools that only emit bare share-token mint/burn, but you stop mislabelling all token mint/burn as LP shares. - Keep
mint/burnbut gate them behind a positive pool signal — e.g. only accept them from contracts already known to be pools (a pool registry / a priordeposit/withdrawseen from thatcontractId), not unconditionally.
A test asserting that a plain SEP-41 mint from a non-pool token is not recorded would lock this down. Everything else looks ready — happy to merge once the over-capture is closed.
…s by network
Closes the over-capture: `mint` and `burn` were accepted as LP-share events
from every contract. Those are generic SEP-41 events that every token emits,
so every USDC mint and every ordinary token burn was being written into
LpShareTransfer with poolId = that token's contract, double-stored beside its
own token-transfer row. `deposit`/`withdraw` are pool-specific and stay
unconditional — emitting them is itself a claim to be a pool.
Bare mint/burn is now only honoured from a contract already established as a
pool. A contract joins that set by:
- emitting an explicit deposit/withdraw, in this batch or a previous one
(parseLpShareEvents scans the batch for the explicit dialect before
decoding anything, so a pool's first deposit and the mint alongside it are
not split by their order within one batch);
- already appearing as a poolId in the table, loaded at loop start so a
restart does not forget and silently stop recording bare events;
- being named in LP_POOL_CONTRACT_IDS, for a pool that only ever emits the
bare dialect.
Tests: a plain token mint from an unknown contract is not recorded, the
explicit dialect is accepted from anyone, a same-batch deposit promotes the
contract, and a carried-over pool keeps working.
Network scoping, which this branch predates:
- LpShareTransfer gains a `network` column with @@unique([network, eventId])
in place of the global unique, and network-leading indexes. A global unique
on eventId would let a testnet event suppress its mainnet namesake, and
createMany's skipDuplicates would swallow it without a trace.
- The migration is renumbered to 20260901130000, after add_network, for the
same reason as the tombstone one: add_network back-filled the tables that
existed when it was written, and this was not one of them.
- upsertLpShareTransfers takes the network; rollbackToLedger deletes LP rows
network-scoped alongside the others.
Merge fixes: dropped the duplicate pre-network ContractTombstone model and the
superseded June tombstone migration (both arrived via the Miracle656#155 stack, which is
now on main), and repaired a stale `totalIndexed` reference that survived the
merge as a bare identifier.
tsc clean; full suite 382 passed.
Miracle656
left a comment
There was a problem hiding this comment.
Approved and merging — I closed the over-capture myself (027b1d5) and rebased onto main, since the wave has closed.
The parts I praised first time still hold up: the eventId-keyed idempotent upsert, the reorg-aware rollbackToLedger cleanup, and modelling a deposit as a mint-to-provider / a withdrawal as a burn-from-provider so it mirrors decoder.ts. Handling both event dialects was also the right instinct — pools genuinely do emit both.
The fix takes your option 2 rather than option 1, because dropping mint/burn entirely would have lost the pools that only ever emit the bare share-token dialect, and that is a real category rather than a hypothetical one.
deposit/withdraw stay unconditional: they are pool-specific, so emitting one is the claim to be a pool. mint/burn are only honoured from a contract already established as a pool, and a contract joins that set three ways:
- It emitted an explicit
deposit/withdraw— in a previous batch, or in this one.parseLpShareEventsscans the batch for the explicit dialect before decoding anything, which matters more than it first appears: a pool's first-ever deposit and the share mint emitted alongside it land in the same batch, and without the pre-scan whether both are recorded would depend on their order within it. - It already appears as a
poolIdin the table, loaded when the loop starts. Without this a restart forgets every pool it had learned and silently stops recording their bare events until the next explicit deposit — the kind of regression that only shows up as a gap in a chart weeks later. - It is named in
LP_POOL_CONTRACT_IDS, for a pool that emits nothing but the bare dialect.
The test you asked about is there: a plain SEP-41 mint from an unknown contract produces no record. Plus the explicit dialect from any contract, same-batch promotion, and a carried-over pool.
Network scoping, which this branch predates. Cut before #159, so:
LpShareTransfergainsnetworkwith@@unique([network, eventId])replacing the global unique, and network-leading indexes. A global unique would let a testnet event id suppress its mainnet namesake — and because the insert usescreateManywithskipDuplicates, it would do so without an error. Silent data loss is the worst version of this bug.- The migration is renumbered after
add_network, same reasoning as the tombstone one:add_networkback-filled the tables that existed when it was written, and this was not one of them. rollbackToLedgerdeletes LP rows network-scoped alongside the rest — that was a genuine union with main, since your branch added the LP delete and main added the network predicate to the others.
Two merge artefacts I repaired, both from the #155 → #156 stack now that #155 is on main: a duplicate pre-network ContractTombstone model (Prisma refuses to generate on that), and a stale totalIndexed that survived the merge as a bare identifier where main had moved to loop.totalIndexed. Also added lpShareTransfer to the prisma mock in network.test.ts so the reorg test covers it.
Verified: tsc --noEmit clean, full suite 382 passed.
Good work — the over-capture was the only thing wrong with it, and the dialect table in the module header is what made it easy to reason about the fix.
Miracle656#155 and Miracle656#156 have landed, so every code file this branch carried from that stack now exists on main in a later form. Took main's version of db.ts, indexer.ts, lp-shares.ts, tombstones.ts, schema.prisma and both test files, and dropped the two superseded June migrations. What remains is docs/event-reference.md plus the .gitignore change that makes it visible: docs/ was ignored wholesale, which is why every doc in this repo had to be force-added. Replaced that with a targeted ignore for docs/openapi.json, which is generated by npm run docs:openapi alongside the tracked root copy. Verified the doc against the code rather than reading it: KNOWN_EVENT_TYPES matches decoder.ts:7 exactly, and every base64 ScVal in the worked examples decodes to the value the doc claims — topic0 'transfer', the two G-addresses, and 1000000000 stroops. tsc clean; full suite 382 passed.
Implemented LP-share transfer tracking, surfacing liquidity-pool deposits/withdrawals as first-class LP-share transfers tagged with the pool ID.
Closes #142
New module — src/indexer/lp-shares.ts
A self-contained, pure decoder mirroring the existing NFT ingester / decoder style:
isLpShareEvent / parseLpShareEvent / parseLpShareEvents — detect and decode LP events, never throwing (a bad event yields null so it can't stall ingest).
Handles both event dialects seen in Soroban AMMs:
Explicit deposit/withdraw (Soroswap, Phoenix, …) — provider in topics[1].
Bare SEP-41 mint/burn of the pool's own share token (native liquidity_pool SAC) — recipient in topics[2] for mint, holder in topics[1] for burn.
A deposit is modelled as shares minted to the provider (toAddress, no from); a withdrawal as shares burned from the provider (fromAddress, no to) — consistent with how decoder.ts treats mint/burn.
extractShares reads the share amount from a bare i128 or a map-wrapped value (share_amount/shares/amount/…), since AMMs bundle several figures into the value; returns the absolute value as a decimal string, preserving full i128 precision.
upsertLpShareTransfers — idempotent bulk insert (skipDuplicates on eventId).
Schema & migration
New LpShareTransfer model in prisma/schema.prisma with a poolId column (the emitting pool contract — always available on the row), action, from/to, shares, ledger fields, and a unique eventId. Indexed on poolId, poolId+action, addresses, ledger, txHash.
Migration 20260629130000_add_lp_share_transfers, following the tombstones migration's exact SQL conventions.
Wiring
src/indexer.ts — added an LP-share path to pollOnce, best-effort and additive (deposit/withdraw aren't touched by the fungible path; a pool's own share mint/burn is recorded here in addition to its token row), plus an updated processed-events log line.
src/db.ts — rollbackToLedger now also prunes LP-share rows above the target ledger during reorgs.
Verification
tsc --noEmit passes clean across the whole project.
New suite src/tests/lpShares.test.ts — 25 tests, all passing (both dialects, share extraction from bare/map/large/negative values, pool-ID tagging, batch filtering, idempotent insert).
Full jest run: my change adds zero new failures. The 3 failing suites are pre-existing tests/integration/ vitest files (run via the se