perf(ha): fetch long history windows in chunks to bound peak memory - #4558
Merged
Conversation
Predbat fetches up to 22 days of history for several sensors on its first plan cycle. requests computes .json() as loads(self.text), and .text decodes .content, so during the parse the same payload is resident three times over - raw bytes, decoded string and parsed objects. For load_power that is 16.3MB of JSON and 54,000 states, and profiling showed those three representations dominating the peak memory of a run. Split windows longer than HISTORY_CHUNK_DAYS across several requests so only one chunk's three representations are live at a time. get_history_window() is extracted for the single request and get_history() assembles the chunks. Home Assistant opens every history window with the state in effect at start_time, synthesised rather than recorded. For chunks after the first that instant is already covered by the previous chunk, so the record is a duplicate. It cannot simply be ignored as harmless: when a boundary lands inside a gap in the recording it adds a data point a single request never returns, and minute_data then interpolates across that gap differently. Against 21 days of real history one boundary in six did exactly that, moving 12 minutes of load data. Dropping records at or before each chunk's start makes the state count match a single fetch exactly and the resulting minute data byte-identical, at every chunk size tested. Measured over three plan cycles against a live Home Assistant, 22 day window: chunk size peak RSS settled RSS cold cycle none 221.3MB 186.8MB 9.84s 3 days 188.7MB 173.1MB 9.43s 1 day 176.4MB 172.3MB 11.61s Three days is the default: it takes essentially all the steady-state saving and three quarters of the peak saving at no measurable time cost, where one day adds 1.8s of startup for 7 requests per sensor rather than 21. Warm cycles are unaffected either way - the history cache refreshes with a small delta fetch that stays under the chunk threshold. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces peak memory during Home Assistant history fetches by splitting long history windows into smaller requests and stitching the results back together, including special handling to drop HA’s synthesised “state-at-window-start” record at chunk boundaries.
Changes:
- Add chunked history fetching in
HAInterface.get_history()with a default maximum chunk size (HISTORY_CHUNK_DAYS), plus boundary record filtering. - Register a new unit-test suite for history chunking and add a dedicated test module to validate boundary handling and window tiling.
- Minor unit test harness update to include the new test suite.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| apps/predbat/ha.py | Implements chunked history retrieval, window helper, and boundary record dropping to bound peak memory. |
| apps/predbat/tests/test_history_chunking.py | Adds tests for chunk boundary deduplication and correct chunk window coverage. |
| apps/predbat/unit_test.py | Registers the new history chunking test suite in the test runner. |
Suppressed comments (2)
apps/predbat/tests/test_history_chunking.py:55
datetime.fromisoformat(start)will fail for theTIME_FORMAT_HAformat when it includes offsets like+0000(Python 3.11). Usedatetime.strptime(..., TIME_FORMAT_HA)(orutils.str2time) so parsing matches the format you generate.
def responder(start, end):
"""Return a synthesised record at the window start plus one real record."""
start_dt = datetime.fromisoformat(start)
# Home Assistant reports the state in effect at start_time, exactly on the boundary
# and with no sub-second precision, then any real changes within the window
return [[_record(start_dt, "500"), _record(start_dt + timedelta(hours=1), "600")]]
apps/predbat/tests/test_history_chunking.py:126
- Same parsing issue here:
fromisoformat()won't accept+0000offsets. Sincestart/endare produced withTIME_FORMAT_HA, parse them withstrptime(..., TIME_FORMAT_HA)to keep the tests compatible with the repo's Python 3.11 target.
interface = _interface()
calls = _capture(interface, lambda start, end: [[_record(datetime.fromisoformat(start) + timedelta(hours=1), "500")]])
interface.get_history("sensor.load_power", NOW, days=10, chunk_days=3)
if len(calls) != 4:
print("ERROR: expected 4 chunks for 10 days at 3 days each, got {}".format(len(calls)))
return failed + 1
print("✓ 10 day window split into 4 chunks of at most 3 days")
starts = [datetime.fromisoformat(start) for start, _ in calls]
ends = [datetime.fromisoformat(end) for _, end in calls]
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+924
to
+941
| history = [] | ||
| cursor = start | ||
| while cursor < end: | ||
| window_end = min(cursor + timedelta(days=chunk_days), end) | ||
| res = self.get_history_window(sensor, cursor, window_end) | ||
| if res: | ||
| for item in res[0]: | ||
| # Home Assistant opens every window with the state in effect at start_time. | ||
| # For chunks after the first that instant is already covered by the previous | ||
| # chunk, and the synthesised record can land inside a gap in the recording | ||
| # where it would add a data point a single request never returns, changing | ||
| # how minute_data interpolates across that gap. | ||
| if cursor > start and item.get("last_updated") and str2time(item["last_updated"]) <= cursor: | ||
| continue | ||
| history.append(item) | ||
| cursor = window_end | ||
|
|
||
| return [history] if history else None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Profiling a standalone run showed the first plan cycle's history fetch dominating peak memory.
requestscomputes.json()asloads(self.text), and.textdecodes.content— so during the parse the same payload is resident three times over: raw bytes, decoded string, parsed objects.At the high-water mark, with the ML forecaster disabled:
json/decoder.py raw_decoderequests.textrequests.contentload_poweralone is 16.3 MB of JSON / 54,000 states over 21 days.Change
Windows longer than
HISTORY_CHUNK_DAYSare split across several requests, so only one chunk's three representations are live at a time.get_history_window()is extracted for the single request;get_history()assembles the chunks.This sits in
HAInterface.get_history, which is the single point every path reaches — direct calls,HAHistorycache misses, and the ML component.The boundary record — the subtle part
Home Assistant opens every history window with the state in effect at
start_time, synthesised rather than recorded (note: no sub-second precision, unlike real records):For chunks after the first, that instant is already covered by the previous chunk. It is not harmless to keep: when a boundary lands inside a gap in the recording it adds a data point a single request never returns, and
minute_datathen interpolates across that gap differently.Against 21 days of real history, one boundary in six did exactly that — moving 12 minutes of load data. At 7-day chunks it happened to miss every gap and looked correct, which is precisely why this needed fixing rather than accepting as noise.
Dropping records at or before each chunk's start gives an exact state-count match against a single fetch and byte-identical minute data:
Measured impact
Three plan cycles against a live Home Assistant, 22-day window:
3 days is the default. It takes essentially all the steady-state saving and three quarters of the peak saving at no measurable time cost — it is marginally faster than no chunking, as HA serves several medium queries at least as well as one large one. One day adds 1.8s of startup for 21 requests per sensor instead of 7.
The full fetch only happens on the first cycle:
HAHistoryrefreshes every 2 minutes withfrom_time=<newest stored record>, a small delta that stays under the chunk threshold. So warm cycles are unaffected (0.15s either way). The steady-state saving persists because the allocator never has to grow the heap for the one-off spike.chunk_days=0restores single-request behaviour.Testing
tests/test_history_chunking.py, 8 assertions across 3 tests, registered ashistory_chunking: boundary-record handling, short windows still using one request, and chunks tiling the window contiguously with no gaps or overlap./run_all --quick— all tests pass, 20/20 random scenarios match baseline across 320 fields./run_pre_commit— all hooks pass🤖 Generated with Claude Code