perf(storage): split UnspentTokensIteratorBy into UNION ALL of two index-friendly branches - #1681
Merged
adecaro merged 1 commit intoMay 7, 2026
Conversation
…dex-friendly branches
UnspentTokensIteratorBy joined the ownership table when filtering unspent
tokens, which caused HasTokenDetails to emit a predicate of the form
`(wallet_id = $1 OR owner_wallet_id = $1)` spanning both tables.
PostgreSQL's planner cannot use the partial index on owner_wallet_id under
that OR predicate; instead it scans every owner=true,is_deleted=false
row, returning a single match after filtering thousands. On a node with
263k tokens this query ran in ~38ms and accounted for 43% of all PG
activity at c=200.
Rewrite as a single SQL with two UNION ALL branches that each use their
own index:
1. tokens directly owned: filters tokens.owner_wallet_id only, which
lets the planner pick the partial index
(owner_wallet_id, token_type) WHERE is_deleted=false AND owner=true
in microseconds.
2. tokens reachable via the ownership-delegation table: filters
ownership.wallet_id and joins to tokens. Returns zero rows when
delegation is not configured (ownership table empty), at which
point the branch is essentially free.
Both branches preserve the pre-existing LEFT JOIN ownership so a tokens
row with owner_wallet_id set but no ownership entry (StoreToken allows
this when its owners parameter is empty) remains visible — the original
query saw it via the OR predicate, this one sees it via branch 1.
The two branches are emitted into a shared query builder so the
placeholder counter ($1, $2, ...) increments continuously across them,
and parentheses around the SELECT operands are intentionally omitted
because SQLite rejects parenthesised SELECTs around UNION (PostgreSQL
accepts both forms). Neither branch has ORDER BY / LIMIT, so dropping
the parens does not change binding.
UNION ALL is preferred over UNION because PG would otherwise hash every
returned row over five columns (including bytea owner_raw) for a
deduplication pass that almost never finds collisions. Instead each
branch additionally selects ownership.wallet_id as a sixth column used
only for app-side dedup; the iterator filters duplicates by
(tx_id, idx, ownership.wallet_id), which:
- drops the rare cross-branch duplicate that arises when the same
(token, ownership) row matches both tokens.owner_wallet_id=$1 and
ownership.wallet_id=$1;
- preserves multi-row results for a single token that has several
ownership rows (the original OR-and-INNER-JOIN behaviour).
The trailing wallet_id column can be NULL when the LEFT JOIN finds no
matching ownership row, so it is scanned as sql.NullString. The dedup
key namespaces NULL distinctly from a (theoretically possible) empty
string so the two never collide.
Effect under stress (c=200, full wallet pool, warm institution):
prepare p50 1.16s -> 43ms (-96%)
prepare p99 — -> 228ms
TPS 112 -> ~178 (limited by endorser tail under load)
The selector hot path that drives prepare latency now spends microseconds
per UnspentTokensIteratorBy call instead of tens of milliseconds, and
single-statement execution keeps the connection footprint at one per
caller — important under high concurrency, where any eager two-query
split would deadlock on a constrained pool.
Signed-off-by: Evan <evanyan@sign.global>
EvanYan1024
force-pushed
the
perf/unspent-tokens-union-all
branch
from
May 7, 2026 09:48
1906cc0 to
16eee0d
Compare
adecaro
self-requested a review
May 7, 2026 13:23
adecaro
approved these changes
May 7, 2026
adecaro
left a comment
Contributor
There was a problem hiding this comment.
Super interesting catch, thanks @EvanYan1024 🙏
SurbhiAgarwal1
pushed a commit
to SurbhiAgarwal1/fabric-token-sdk
that referenced
this pull request
May 13, 2026
…dex-friendly branches (LFDT-Panurus#1681) Signed-off-by: Evan <evanyan@sign.global> Co-authored-by: Evan <evanyan@sign.global>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites
UnspentTokensIteratorByso the SQL planner can pick the partialindex
idx_owner_wallet_part (owner_wallet_id, token_type) WHERE is_deleted=false AND owner=truethat #1612 added. The original cross-table
ORpredicate forces the plannerto seq-scan every live owner-token; this PR replaces that with a single
statement of two
UNION ALLbranches that each filter only one side of theoriginal condition.
Closes #1680.
Background
UnspentTokensIteratorByis on the hot path of every transfer: the selectorcalls it to populate its in-memory token index. After #1612 added the
partial index, the predicate that should have driven it (
owner_wallet_id = $1 AND is_deleted=false AND owner=true) was buried inside a cross-table OR:Postgres cannot index-walk under that OR. On a deployment with ~220k live
owner-tokens the resulting
Hash Right Joinran in 247 ms warm / 7,330 mscold per call, dominating prepare latency.
Changes
token/services/storage/db/sql/common/tokens.goUnspentTokensIteratorBynow emits twoUNION ALLbranches into a sharedquery builder, then filters duplicates in a small iterator wrapper:
WHERE owner_wallet_id = $1. The partialindex drives this in microseconds. The
LEFT JOIN ownershipis kept sothe iterator can surface the same
(token, ownership)rows it didbefore — including the multi-ownership case where one token has several
ownership rows.
StoreTokenpermitstokensrows withowner_wallet_idset but no ownership entry (when its
ownersparameter is empty); thoserows surface here, replacing what the original OR predicate caught.
WHERE ownership.wallet_id = $1.Returns zero rows on deployments where ownership delegation is not
configured, at which point the branch is essentially free.
Implementation notes worth flagging:
common.Builderso placeholder numbering(
$1,$2, …) increments continuously; args concatenate in order. Singlestatement → single connection from the pool, which avoids the deadlock
any eager two-query split would hit on a constrained pool under load.
UNION ALL, notUNION.UNIONwould force PG to hash every row overfive columns (including
bytea owner_raw) for a per-row dedup pass thatalmost never finds collisions. Each branch instead selects
ownership.wallet_idas a sixth trailing column used solely for app-sidededup in
dedupedTokenRowsIterator.(tx_id, idx, ownership.wallet_id). This dropsthe rare cross-branch duplicate (a row matching both
tokens.owner_wallet_id = $1andownership.wallet_id = $1) whilepreserving distinct multi-ownership rows. NULL is namespaced separately
from the empty string so the two cannot collide.
UNION ALL— SQLite rejectsparenthesised SELECTs around
UNION; PostgreSQL accepts both forms.Neither branch has
ORDER BY/LIMIT, so binding is unchanged.token/services/storage/db/sql/common/query_test.goNew
TestUnionAll_Compileverifies the shared-builder pattern: twoq.Select()queries combined viaFormatToproduce a single statementwith continuous placeholder numbering, ordered args, and no parens around
the operands.
Measured impact
EXPLAIN ANALYZEon a deployment with 219,968 active owner-tokens, walletwith 94 matching tokens (warm shared buffers):
Compatibility
the existing CI matrix).
dedupedTokenRowsIteratorpreserves the pre-PR row semantics: one rowper
(token, ownership-wallet)pair, including multi-ownership tokens(covered by the existing
TestTokenscase wheretx100:1is co-ownedby alice and bob) and
owners=niltokens (tx2000).Test plan
go build ./...go test ./token/services/storage/db/sql/common/...— all existingcases pass, including
TestTokenswhich exercises multi-ownershipand no-ownership rows. New
TestUnionAll_Compilecovers theshared-builder behaviour.
EXPLAIN ANALYZEagainst a real Postgres deployment (numbers above).-s) per CONTRIBUTING.md.