Skip to content

feat: accept an external consumption forecast as a time-series entity - #437

Draft
johanzander wants to merge 4 commits into
mainfrom
feat/issue-428-consumption-forecast-series
Draft

feat: accept an external consumption forecast as a time-series entity#437
johanzander wants to merge 4 commits into
mainfrom
feat/issue-428-consumption-forecast-series

Conversation

@johanzander

Copy link
Copy Markdown
Owner

Summary

  • Adds ha_consumption_series: a new consumption-forecast strategy reading a user-authored HA time-series entity (raw_today/raw_tomorrow attributes of timestamped {start, value} records), mirroring the existing price-manager entity pattern rather than solar's hour-indexed parser (which can't represent sub-hourly data).
  • BESS still does not model loads itself — no EV/weather/occupancy modelling was added. The user's own HA template or automation builds the shaped series; BESS only normalizes 15- or 60-minute record spacing onto the DP's quarter-hour grid.
  • A missing, stale, malformed, or horizon-short series raises an explicit ConsumptionForecastUnavailableError — this strategy never silently falls back to a flat profile, unlike ha_statistics's fixed-value fallback on insufficient data.
  • Renames the scalar sensor strategy's misleading "Consumption Forecast" UI category to "Consumption Average (48h Rolling)" (it was never a real forecast) and adds a new "Consumption Forecast Series" category for the new entity.
  • Adds an example HA template (baseline + a scheduled EV block) to docs/USER_GUIDE.md so the seam is usable without starting from a blank YAML file.

Root cause

From the issue: every existing consumption strategy is either a constant or an in-house statistical model. ha_statistics's trimmed mean deliberately discounts one-off spikes (the right call for outlier robustness) but as a side effect is permanently blind to a known, predictable load like an EV session — nothing puts that load back into the forecast. Solar has an external-forecast seam (Solcast); consumption had none.

Fix

  • HomeAssistantAPIController.get_consumption_forecast_series() / get_consumption_forecast_series_tomorrow() (core/bess/ha_api_controller.py) — fetch and normalize the entity, mirroring price_manager.py's raw_today/raw_tomorrow parsing.
  • BatterySystemManager._get_consumption_forecast dispatches to it for the new strategy; _extend_consumption_predictions (extracted from the existing tomorrow-horizon-extension logic) uses the entity's real raw_tomorrow data instead of repeating today's pattern when the DP horizon spans midnight — the same treatment solar's _fetch_tomorrow_solar_forecast already gets.
  • Health-check coverage (sensor_collector.check_prediction_health) and the strategy-comparison endpoint (get_consumption_forecast_comparison) both cover the new strategy.
  • Frontend: new radio option + description in HomeFormSection.tsx, sensor category rename + addition in sensorDefinitions.ts, updated SensorConfigSection.tsx detection logic, ConsumptionForecastComparison.tsx chart labels/colors.

Test plan

  • ./scripts/quality-check.sh passes locally (fast suite: 1399 passed, 15 skipped; frontend: 103 passed; Black/Ruff/TypeScript/ESLint clean)
  • .venv/bin/pytest -m slow passes (394 passed, 3 skipped)
  • New unit tests: entity parsing (happy path, hourly upsampling, stale/malformed/short-horizon/unsupported-interval failure modes), strategy dispatch + no-silent-fallback, health-check integration, and an end-to-end test that runs a shaped EV-load series through the real DP optimizer and confirms it produces a different (correct) plan than a flat series
  • Local E2E verification via docker-compose.ci.yml (real backend + mock-HA): configured the new strategy and a shaped entity via the real PATCH /api/settings API, confirmed GET /api/consumption-forecast-comparison returns the correct shaped hourly profile (EV hour at 16.0 kWh vs 1.2 kWh baseline) through the live system, confirmed the health check reports OK with the exact shaped series and then a clear ERROR (no silent fallback) when the entity's raw_today data was removed, and confirmed the new frontend strings are present in the actual served JS bundle

Closes #428

johanzander and others added 4 commits July 30, 2026 23:46
…#428)

BESS optimizes against forecasts but never modelled loads itself — for
consumption every existing strategy was either a constant or an in-house
statistical model, none able to express a known, shaped load (an EV
session, weather-driven aircon) the user can predict but a trimmed-mean
baseline deliberately discounts as an outlier.

Adds `ha_consumption_series`: a new strategy reading a user-authored HA
entity with `raw_today`/`raw_tomorrow` timestamped {start, value}
attributes, mirroring the existing price-manager pattern rather than
solar's hour-indexed parser (which structurally can't represent
sub-hourly data). Accepts 15- or 60-minute record spacing, normalized
onto the DP's quarter-hour grid; a missing, stale, malformed, or
horizon-short series is an explicit ConsumptionForecastUnavailableError,
never a silent fallback to a flat profile — unlike ha_statistics's
fixed-value fallback on insufficient data.

Also renames the scalar `sensor` strategy's misleading "Consumption
Forecast" UI category to "Consumption Average (48h Rolling)", since it
was never a real forecast, and adds an example HA template so the seam
is usable without starting from a blank YAML file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ption-forecast-series

# Conflicts:
#	CHANGELOG.md
…ption-forecast-series

# Conflicts:
#	CHANGELOG.md
@johanzander

Copy link
Copy Markdown
Owner Author

Still conflicted — merge attempted twice today, both aborted

This PR is CONFLICTING against main and has been since ~2026-08-02. Two attempts today, neither landed:

  1. A manual sweep-prs pass merged origin/main and hit semantic conflicts in 5 files, so it aborted per that skill's rule against resolving someone else's logic conflicts. Tree left clean.
  2. A background session (pr437-conflicts) was dispatched to resolve them and produced nothingHEAD still 0a6e91f4, 3 commits ahead, no MERGE_HEAD, no tracked file written in 2.5 hours. Stopped. Nothing lost, nothing done.

The conflicts

git merge origin/main conflicts in:

  • core/bess/ha_api_controller.py
  • core/bess/sensor_collector.py
  • core/bess/settings_store.py
  • core/bess/tests/unit/test_prediction_health_check.py
  • frontend/src/components/settings/HomeFormSection.tsx

These are logic conflicts, not mechanical ones. Resolving them means preserving both this branch's external-consumption-forecast feature and ~2 weeks of main drift in the same files — per docs/agents/rules.md, with no silent fallbacks or defensive error handling added while reconciling.

One trap worth knowing before you start

The worktree accumulated 150 untracked files byte-identical to origin/main, which blocked the merge with The following untracked working tree files would be overwritten by merge before any conflict was even reported. Git truncates that list at ~4KB, so it cannot be enumerated from the error message — derive it instead:

comm -12 <(git ls-files --others --exclude-standard | sort) \
         <(git ls-tree -r origin/main --name-only | sort)

Then delete only the entries that are byte-identical to origin/main (verify with git show origin/main:$f | cmp -s - "$f"); the merge restores them. That has already been done once, so it may not recur — but check, because it presents as an unrelated failure.

johanzander added a commit that referenced this pull request Aug 17, 2026
…e-argued (#636)

* feat: put PRs on the board so a decision about one is recorded, not re-argued

The board held issues only, so every judgement about a PR had nowhere to live.
"#167 and #354 are blocked", "#437 and #490 are lower priority, later" were
real decisions the maintainer had already made, and the rhythm pass re-reported
all four as due on every tick because nothing recorded them. The same
conversation happened every 30 minutes.

Projects v2 takes PRs as items with the identical field set, so the fix is
membership rather than a parallel mechanism. A PR card carries the same
`Priority` and `Awaiting` an issue card does; the digest emits them as
`pr_board`, and `backlog-rhythm.sh` joins by number and suppresses on them:
an `Awaiting` means parked on someone, `P4` means later-not-never.

`content.type` is what separates PR cards from issue ones, confirmed against a
real card rather than assumed — an added PR reports "PullRequest" with
number/title/url/repository alongside it. Numbers are unique across issues and
PRs in one repository, so this cannot collide with the existing issue lookup.

Suppressed PRs are COUNTED AND LISTED, never dropped: the pass ends with
`deferred: 4 (#490 priority P4; #167 awaiting discussion; ...)`. Silently
vanishing would trade one failure for another — the goal is to stop re-asking
about a settled decision, not to lose the item.

CONTRACT CHANGE: an APPROVED, green, still-draft PR is now its own action,
`mark_ready`, and it is the one thing no board decision can defer. It used to
hand back to `implement-issue` like any other unfinished draft, on the
principle that this pass must not grow a second review loop. That principle
still holds, but it is what left #629 sitting approved, green and draft: the
remedy on offer was a whole `implement-issue` session, and nobody spends one of
those to run a single command. `gh pr ready` is a terminal action, not a loop,
so naming it here duplicates nothing.

`awaiting_maintainer` is deliberately NOT carved out the same way. An approved
PR waiting on a merge is not broken; it is the maintainers call when to take
it, and P4 is exactly how they say later. #490 sat approved for a day and was
reported every tick as though that were news.

Live effect: 31 actions -> 27, with one `deferred: 4` line in place of four
recurring items, and #631 correctly still reported as merge-ready.

No CHANGELOG entry: agent tooling, no user-visible effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012LExo6fcbup75vtc9NfoAR

* fix: mark_ready needs green checks, not just a clean merge

Found by running the rule against the live fleet on its first pass, which is
the only reason it was caught before the PR merged.

`mergeable` reports whether the branch merges cleanly and nothing else, so it
reads MERGEABLE while CI is still running or has failed outright. #633 was
APPROVED, MERGEABLE and had Algorithm tests and E2E still IN_PROGRESS, and the
rule duly reported "gh pr ready 633 — then it is the maintainers to merge".
GitHub itself disagreed: `mergeStateStatus` was BLOCKED.

Flipping a red or pending PR out of draft is worse than leaving it there.
`ready` is supposed to mean the maintainer can merge without checking anything
else, and that claim is the only thing making the flag worth setting.

So `mark_ready` now also requires every check to have concluded SUCCESS,
SKIPPED or NEUTRAL. SKIPPED is green on purpose: this repo path-filters
Algorithm tests and Docker build, so every backend-only PR skips them and
treating that as not-green would withhold the action from almost everything.
An empty rollup is green too — a PR with no checks configured has nothing
failing.

The deferred list mirrors the same condition, so an approved-but-pending PR
with a P4 card appears in exactly one place rather than both.

Live effect: #633 falls back to resume_implementation until its CI settles,
which is the correct answer and the one the first version got wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012LExo6fcbup75vtc9NfoAR

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
johanzander added a commit that referenced this pull request Aug 18, 2026
… branch

Two halves of the same failure: work landing outside a worktree, and
nobody noticing when it does.

## The hook

CLAUDE.md has said "never edit any file on main, even a one-line doc fix"
unconditionally for a long time, and it keeps being skipped. The reason
is structural, not carelessness: it is prose, so it has to be REMEMBERED
at the moment of the first edit — and that is exactly the moment a
session which opened as a question has no reason to reconsider it. Six
live sessions currently sit in the main checkout for perfectly good
read-only reasons; nothing catches the one that quietly starts editing.

check-worktree-path.sh already guarded CROSS-checkout edits and passed
same-checkout ones, so main-to-main sailed through. It now also refuses
any edit made from the main checkout, detected by --git-dir equalling
--git-common-dir. That is a path comparison, the only shape
docs/agents/rules.md sanctions here — it never guesses what a command
will touch. Linked worktrees and sibling checkouts both differ, so both
still work; the rule is "be in a worktree", not "be under .claude/".

The denial names the remedy (EnterWorktree) and says what the main
checkout still does — questions, gh, backlog, dispatch — because a block
without a next move gets worked around.

Residual gap, stated plainly: this governs Edit/Write/NotebookEdit. A
Bash `sed -i` still writes. Guarding that would mean parsing command
strings, which rules.md forbids for this hook and which has produced
false positives here four times.

## The detector

pr-state.sh gains a local-writer section. GitHub cannot see this: a
branch with two writers looks normal through the API, because the
divergence exists only between a local checkout and the remote and it
collapses into an ordinary merge the moment someone reconciles.

#619 is the worked example. One writer took the branch at 08:09 and
worked from that base; another pushed 23031e7 at 09:34. The reviewer
reviewed 23031e7 three times, twice with blocking findings, while the
first line never held that commit. Fifteen hours later it landed as
`Merge remote-tracking branch 'origin/fix/...' into fix/...` — a branch
merged into itself, which is the fingerprint. `git rev-list --left-right`
would have caught it at 09:34.

Run against the live fleet it also surfaces the precursor state: #437 is
5 commits behind its own remote and #614 is 3, so a commit in either
worktree diverges immediately.

Skipped LOUDLY outside a checkout, since this script is also meant for a
container fleet where a silent skip would read as "no divergence found".

Verified by mutation: disabling the main-checkout guard reddens 2 of the
6 hook tests. The divergence detector is tested against a real two-clone
scenario, not fixtures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bess-product-owner

Copy link
Copy Markdown
Collaborator

Parked: not to be merged yet — awaiting user feedback

Maintainer decision, recorded here so it stops being re-derived every backlog pass. Board fields set to match:

  • Priority: P4
  • Awaiting: reporter

Together those suppress this PR from the rhythm pass's action list. It is still counted and listed as deferred, so it stays findable — it is parked, not dropped.

Do not resolve the conflicts yet. This PR has been CONFLICTING since ~2026-08-02 with semantic conflicts in five files:

  • core/bess/ha_api_controller.py
  • core/bess/sensor_collector.py
  • core/bess/settings_store.py
  • core/bess/tests/unit/test_prediction_health_check.py
  • frontend/src/components/settings/HomeFormSection.tsx

Two attempts have already been spent on them — a sweep-prs pass that correctly aborted rather than resolve someone else's logic conflicts, and a background session that wrote nothing in 2.5 hours. A third attempt would cost the same and land in the same place, because the blocker is not the merge: the feature's shape needs user feedback first, and resolving conflicts against a design that may still change is wasted work.

Unpark by clearing Awaiting once that feedback is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accept an external consumption forecast as a time-series entity

2 participants