Skip to content

perf(storage): split UnspentTokensIteratorBy into UNION ALL of two index-friendly branches - #1681

Merged
adecaro merged 1 commit into
LFDT-Panurus:mainfrom
Built-by-Sign:perf/unspent-tokens-union-all
May 7, 2026
Merged

perf(storage): split UnspentTokensIteratorBy into UNION ALL of two index-friendly branches#1681
adecaro merged 1 commit into
LFDT-Panurus:mainfrom
Built-by-Sign:perf/unspent-tokens-union-all

Conversation

@EvanYan1024

Copy link
Copy Markdown
Contributor

Summary

Rewrites UnspentTokensIteratorBy so the SQL planner can pick the partial
index idx_owner_wallet_part (owner_wallet_id, token_type) WHERE is_deleted=false AND owner=true
that #1612 added. The original cross-table OR predicate forces the planner
to seq-scan every live owner-token; this PR replaces that with a single
statement of two UNION ALL branches that each filter only one side of the
original condition.

Closes #1680.

Background

UnspentTokensIteratorBy is on the hot path of every transfer: the selector
calls it to populate its in-memory token index. After #1612 added the
partial index, the predicate that should have driven it (owner_wallet_id = $1 AND is_deleted=false AND owner=true) was buried inside a cross-table OR:

WHERE (o.wallet_id = $1 OR t.owner_wallet_id = $1)
  AND t.owner = true
  AND t.is_deleted = false;

Postgres cannot index-walk under that OR. On a deployment with ~220k live
owner-tokens the resulting Hash Right Join ran in 247 ms warm / 7,330 ms
cold
per call, dominating prepare latency.

Changes

token/services/storage/db/sql/common/tokens.go

UnspentTokensIteratorBy now emits two UNION ALL branches into a shared
query builder, then filters duplicates in a small iterator wrapper:

  1. Branch 1 — directly owned: WHERE owner_wallet_id = $1. The partial
    index drives this in microseconds. The LEFT JOIN ownership is kept so
    the iterator can surface the same (token, ownership) rows it did
    before — including the multi-ownership case where one token has several
    ownership rows. StoreToken permits tokens rows with owner_wallet_id
    set but no ownership entry (when its owners parameter is empty); those
    rows surface here, replacing what the original OR predicate caught.
  2. Branch 2 — delegation-reachable: WHERE ownership.wallet_id = $1.
    Returns zero rows on deployments where ownership delegation is not
    configured, at which point the branch is essentially free.

Implementation notes worth flagging:

  • Both branches share a single common.Builder so placeholder numbering
    ($1, $2, …) increments continuously; args concatenate in order. Single
    statement → single connection from the pool, which avoids the deadlock
    any eager two-query split would hit on a constrained pool under load.
  • UNION ALL, not UNION. UNION would force PG to hash every row over
    five columns (including bytea owner_raw) for a per-row dedup pass that
    almost never finds collisions. Each branch instead selects
    ownership.wallet_id as a sixth trailing column used solely for app-side
    dedup in dedupedTokenRowsIterator.
  • App-side dedup keys on (tx_id, idx, ownership.wallet_id). This drops
    the rare cross-branch duplicate (a row matching both
    tokens.owner_wallet_id = $1 and ownership.wallet_id = $1) while
    preserving distinct multi-ownership rows. NULL is namespaced separately
    from the empty string so the two cannot collide.
  • No parens around the SELECT operands of UNION ALL — SQLite rejects
    parenthesised SELECTs around UNION; PostgreSQL accepts both forms.
    Neither branch has ORDER BY / LIMIT, so binding is unchanged.

token/services/storage/db/sql/common/query_test.go

New TestUnionAll_Compile verifies the shared-builder pattern: two
q.Select() queries combined via FormatTo produce a single statement
with continuous placeholder numbering, ordered args, and no parens around
the operands.

Measured impact

EXPLAIN ANALYZE on a deployment with 219,968 active owner-tokens, wallet
with 94 matching tokens (warm shared buffers):

Plan Time Notes
Current (`LEFT JOIN + OR`) 247 ms Hash Right Join over 219,968 rows, post-filter
Current (cold, first run) 7,330 ms Same plan, 35,549 buffer reads
Proposed (`UNION ALL`) 0.66 ms Append: branch 1 hits `idx_owner_wallet_part`, branch 2 hits `tkn_own_pkey`

Compatibility

  • Pure SQL rewrite, no schema or migration change.
  • The two-branch SQL is valid on PostgreSQL and SQLite (both supported by
    the existing CI matrix).
  • dedupedTokenRowsIterator preserves the pre-PR row semantics: one row
    per (token, ownership-wallet) pair, including multi-ownership tokens
    (covered by the existing TestTokens case where tx100:1 is co-owned
    by alice and bob) and owners=nil tokens (tx2000).

Test plan

  • go build ./...
  • go test ./token/services/storage/db/sql/common/... — all existing
    cases pass, including TestTokens which exercises multi-ownership
    and no-ownership rows. New TestUnionAll_Compile covers the
    shared-builder behaviour.
  • EXPLAIN ANALYZE against a real Postgres deployment (numbers above).
  • Commit signed off (-s) per CONTRIBUTING.md.

…dex-friendly branches

UnspentTokensIteratorBy joined the ownership table when filtering unspent
tokens, which caused HasTokenDetails to emit a predicate of the form
`(wallet_id = $1 OR owner_wallet_id = $1)` spanning both tables.
PostgreSQL's planner cannot use the partial index on owner_wallet_id under
that OR predicate; instead it scans every owner=true,is_deleted=false
row, returning a single match after filtering thousands. On a node with
263k tokens this query ran in ~38ms and accounted for 43% of all PG
activity at c=200.

Rewrite as a single SQL with two UNION ALL branches that each use their
own index:

 1. tokens directly owned: filters tokens.owner_wallet_id only, which
    lets the planner pick the partial index
    (owner_wallet_id, token_type) WHERE is_deleted=false AND owner=true
    in microseconds.
 2. tokens reachable via the ownership-delegation table: filters
    ownership.wallet_id and joins to tokens. Returns zero rows when
    delegation is not configured (ownership table empty), at which
    point the branch is essentially free.

Both branches preserve the pre-existing LEFT JOIN ownership so a tokens
row with owner_wallet_id set but no ownership entry (StoreToken allows
this when its owners parameter is empty) remains visible — the original
query saw it via the OR predicate, this one sees it via branch 1.

The two branches are emitted into a shared query builder so the
placeholder counter ($1, $2, ...) increments continuously across them,
and parentheses around the SELECT operands are intentionally omitted
because SQLite rejects parenthesised SELECTs around UNION (PostgreSQL
accepts both forms). Neither branch has ORDER BY / LIMIT, so dropping
the parens does not change binding.

UNION ALL is preferred over UNION because PG would otherwise hash every
returned row over five columns (including bytea owner_raw) for a
deduplication pass that almost never finds collisions. Instead each
branch additionally selects ownership.wallet_id as a sixth column used
only for app-side dedup; the iterator filters duplicates by
(tx_id, idx, ownership.wallet_id), which:

 - drops the rare cross-branch duplicate that arises when the same
   (token, ownership) row matches both tokens.owner_wallet_id=$1 and
   ownership.wallet_id=$1;
 - preserves multi-row results for a single token that has several
   ownership rows (the original OR-and-INNER-JOIN behaviour).

The trailing wallet_id column can be NULL when the LEFT JOIN finds no
matching ownership row, so it is scanned as sql.NullString. The dedup
key namespaces NULL distinctly from a (theoretically possible) empty
string so the two never collide.

Effect under stress (c=200, full wallet pool, warm institution):

  prepare p50    1.16s -> 43ms   (-96%)
  prepare p99      —  -> 228ms
  TPS              112 -> ~178   (limited by endorser tail under load)

The selector hot path that drives prepare latency now spends microseconds
per UnspentTokensIteratorBy call instead of tens of milliseconds, and
single-statement execution keeps the connection footprint at one per
caller — important under high concurrency, where any eager two-query
split would deadlock on a constrained pool.

Signed-off-by: Evan <evanyan@sign.global>
@EvanYan1024
EvanYan1024 force-pushed the perf/unspent-tokens-union-all branch from 1906cc0 to 16eee0d Compare May 7, 2026 09:48
@adecaro
adecaro self-requested a review May 7, 2026 13:23
@adecaro adecaro self-assigned this May 7, 2026
@adecaro adecaro added this to the Q2/26 milestone May 7, 2026

@adecaro adecaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super interesting catch, thanks @EvanYan1024 🙏

@adecaro
adecaro merged commit 379785e into LFDT-Panurus:main May 7, 2026
137 of 139 checks passed
SurbhiAgarwal1 pushed a commit to SurbhiAgarwal1/fabric-token-sdk that referenced this pull request May 13, 2026
…dex-friendly branches (LFDT-Panurus#1681)

Signed-off-by: Evan <evanyan@sign.global>
Co-authored-by: Evan <evanyan@sign.global>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: UnspentTokensIteratorBy bypasses idx_owner_wallet_part, scans every live owner-token

2 participants