Fix: cap aggtrade pagination to prevent candle-hole-fill hang#167
Draft
Dhiyaahaq33 wants to merge 2 commits into
Draft
Fix: cap aggtrade pagination to prevent candle-hole-fill hang#167Dhiyaahaq33 wants to merge 2 commits into
Dhiyaahaq33 wants to merge 2 commits into
Conversation
_save_calculation_results wrote asset_record, unrealized_changes, scribbles, and account_state, but never virtual_state.pickle, despite a comment claiming it was intentionally skipped. _load_or_create_previous_state unconditionally tries to read all five pickle files inside one try block, so the missing virtual_state file raised FileNotFoundError on every subsequent run, which discarded the other four successfully-saved pieces of state too and reset the calculation to start from scratch. Fixes cunarist#146 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJ386AEgV8Z796N4TSJtPn
…scheduler _fetch_aggtrades_for_gap paginated Binance's /fapi/v1/aggTrades (max 1000 trades per page) in an unbounded while loop until it covered a single 10-second window. For a symbol with a very high trade volume, a single 10-second window can contain far more than 1000 trades, so this loop could make hundreds of sequential API round trips before returning. Collector._fill_candle_data_holes runs as an APScheduler cron job every 10 seconds with the default max_instances=1, so while one call was stuck deep in this pagination loop, every subsequent scheduled run got skipped with "maximum number of running instances reached" -- exactly the symptom reported in cunarist#140. Cap the pagination at MAX_AGGTRADE_PAGES_PER_GAP (50) pages per gap. The caller already treats a partially-filled gap as progress and re-queues the symbol on the next cron tick (see Collector._find_full_symbols / _fill_symbol_holes), so capping here doesn't lose data -- it just spreads filling an extreme-volume symbol's backlog across multiple cron ticks instead of blocking one tick indefinitely. Fixes cunarist#140 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJ386AEgV8Z796N4TSJtPn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Collector._fetch_aggtrades_for_gappaginates Binance's/fapi/v1/aggTrades(max 1000 trades/page) in an unboundedwhileloop until the whole 10-second window is covered._fill_candle_data_holesruns as acronjob every 10 seconds with APScheduler's defaultmax_instances=1. While one call is stuck deep in that pagination loop, every subsequent scheduled run is skipped with"maximum number of running instances reached"— exactly the log pattern reported in Filling recent 24 hours of candle data sometimes hangs #140, which happened "when there was a symbol that has [a massive] amount of aggtrade data... in the past 24 hours."MAX_AGGTRADE_PAGES_PER_GAP(50) pages per gap-fill call. The caller (_find_full_symbols/_fill_symbol_holes) already treats a partially-filled gap as progress (did_fill) and re-queues the symbol on the next cron tick, so this doesn't drop data — it just spreads an extreme-volume symbol's backlog across multiple cron ticks instead of blocking one tick indefinitely.Fixes #140
Test plan
ruff checkon changed files passespython -m py_compileon changed files passes🤖 Generated with Claude Code
https://claude.ai/code/session_01WJ386AEgV8Z796N4TSJtPn