Repository navigation
fix(sync): keep identical partial fills from broker sync - #54
Merged
Merged
Conversation
Broker sync dedupes executions by a content hash of symbol, side, quantity, price and time. broker-sdk trades carry no fill ID, so one order that fills in identical pieces collapsed into a single execution: volume, fees and P&L were under-counted, and a position closed by those pieces could stay open (#50: a closed Hyperliquid short merged into the next short on the same coin). brokerTradesToExecutions now numbers the later copies of an identical fill within a batch, and the content hash includes that number. The first copy keeps its existing hash, so fills synced before this change still deduplicate and nothing is imported twice; the next sync adds the missing copies that are still in the broker's history. Copies are counted rather than indexed, so a reordered batch, or a later batch that sees fewer copies, maps onto the same hashes. IBKR Flex XML uploads share this mapping and stay deduplicated against sync. Fixes #50 Fixes #52
broker-sdk's Flex parser reads every <Trade> row as a fill, including order, closed-lot and summary rows that repeat the executions they summarize. File uploads already strip those rows before parsing; sync did not, so a query that includes order-level rows stored each multi-execution order as an extra fill. With identical fills now numbered, the order row of a single-execution order would also be kept next to its execution. Sync now passes broker-sdk a fetch that removes non-execution <Trade> rows from the statement, using the same filter as uploads, so both paths journal the same executions.
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes broker sync dropping genuine partial fills when one order fills in identical pieces (#50, #52).
Cause
Executions are deduplicated by a content hash of symbol, side, quantity, price and time. broker-sdk trades carry no fill ID, so when an order fills in several pieces with the same size and price at the same timestamp, every piece after the first hashed the same and was dropped as a duplicate. Volume, fees and P&L were under-counted, and a position closed by those pieces could stay open. #50 is this case: a Hyperliquid short closed by identical partial fills stayed partly open, so it merged into the next short on the same coin and the closed trade never appeared.
Fix
brokerTradesToExecutionsnumbers the later copies of an identical fill within a batch (occurrenceonImportedExecution), andexecutionHashincludes that number. The first copy keeps its existing hash, so fills synced before this change still deduplicate and nothing is imported twice. The next sync adds the missing copies that are still in the broker's history. Copies are counted rather than indexed, so a reordered batch, or a later batch that sees fewer copies, maps onto the same hashes. IBKR Flex XML uploads share this mapping and stay deduplicated against sync.<Trade>row as a fill, including order, closed-lot and summary rows that repeat the executions they summarize. File uploads already strip those rows; sync did not, so a query with order-level rows stored each multi-execution order as an extra fill, and with numbering the order row of a single-execution order would also be kept. Sync now passes broker-sdk afetch(its documented option) that removes non-execution rows with the same filter uploads use.The numbering relies on each SDK
Tradebeing one fill, as the type documents; broker-sdk's owndiffSnapshotscounts identical trades the same way. I checked every adapter in broker-sdk 0.5.1 for a path that could emit one fill twice in a snapshot. Ten return no trades, and alpaca, etrade, hyperliquid, kucoin, public, questrade, schwab, tastytrade, topstep and tradier have none. IBKR Flex was the exception, handled above. robinhood-crypto appends cursor pages without deduplicating by order ID, so a cursor that shifted mid-paging could repeat an order; that is left as is.Worth raising in broker-sdk: passing each fill's native ID on
Trade(Hyperliquidtid, IBKRibExecID), so the journal can key on it instead of counting copies, and skipping non-execution<Trade>rows inparseFlexStatement.Test
apps/web/tests/sync-identical-fills.test.tsruns the Hyperliquid account sync #50 sequence throughsyncAccountand broker-sdk's real Hyperliquid adapter, withfetchstubbed. The first short shows as closed and the second as open; a re-sync adds nothing; an account synced before this fix keeps its fills and gains only the missing copy; a reordered or shrinking batch adds nothing; and the copy number is validated. Before the fix, the sequence stores 3 of its 4 fills and shows one open short.ibkr-sync-timezone.test.tssyncs two identical executions plus repeated order rows and gets 3 fills and one closed trade. Without thefetchfilter it stores 5.Checks run locally
pnpm format:check,pnpm typecheck,pnpm test(66 files, 625 tests),node scripts/check-licenses.mjs,pnpm -r --filter "./packages/*" buildandpnpm buildall pass.Fixes #50
Fixes #52