idea: risk-aware overnight consumption buffer for floor-proximity decisions (asymmetric loss)

[johanzander]

Idea

Surfaced while investigating #381 (see conclusion posted there). Not proposing
this for implementation now — recording it as a future improvement that needs
real data before it's worth building.

The observation

The DP currently plans against a single point-forecast (the mean expected
consumption) for every period, including the pre-dawn hours with no PV to
rescue a bad forecast. But the cost of a forecast miss is asymmetric there:

• Underestimating overnight consumption risks the battery hitting the
  floor before dawn, forcing an import at the buy price — a real, immediate
  cost.
• Overestimating overnight consumption just means the DP holds back energy
  that turns out to be unneeded. That energy isn't lost — it's still in the
  battery and gets deployed at the next-best opportunity on a later re-plan.
  The cost here is much smaller: a delayed-use timing cost, not a lost sale.

This is a classic asymmetric-loss ("newsvendor") setup: planning against the
mean forecast is not the same as planning against the cost-minimizing
forecast when the two error directions cost so differently. A conservative
consumption estimate (e.g. a P70–P75 quantile) for the no-PV tail of the
horizon would trade a small, ongoing delayed-use cost for a reduction in
floor-breach risk.

Why this is not a fix for #381

#381's incident was a one-off ~25% overnight consumption miss, costing
~€0.15–0.20. That single data point doesn't tell us whether floor breaches
from forecast error are a recurring pattern or a rare tail event — and
applying a conservative buffer has a cost on every night (not just the ones
that miss), so it's only worth it if the aggregate saved-breach cost exceeds
the aggregate delayed-use cost across many nights/users.

What's needed before building this

• Historical overnight forecast-vs-actual consumption deltas across multiple
  users/nights, to see if floor-breach-causing misses are common or rare.
• If common enough to matter: a study of what buffer size (if any) is net
  positive, and whether it should scale with proximity to the floor / hours
  until PV, rather than being a flat adjustment.
• This is a forecast/risk-modeling change to the DP's planning input for the
  no-PV tail of the horizon — not a new manual SOC knob (that idea was
  already evaluated and rejected in feature: computed overnight reserve floor (Min SOC + historical night use) so one knob covers evening export + night load #381).

Status

Idea only. No action planned until supporting data exists.

[johanzander]

Heads-up on two related issues:

#466 found a different piece of the same story. The optimizer sometimes picks IDLE over "power the house from the battery" when the two look almost exactly equally good on paper. But they're not equally safe: if the household uses more power than predicted, the battery-powering mode just covers it, while IDLE locks the battery so the extra usage gets bought from the grid. The fix for that (design spec): when it's basically a coin flip, always pick the safe option. That fix needs no changes to the consumption forecast, so it doesn't replace this issue.

This issue stays open for what the tie-break can't fix: nights where the forecast itself is too low and the battery runs empty before morning. That still needs the "plan for a bit more than the average night" idea described here, and we still want real data from more users before building it.

Also see #487: if night consumption really is systematically under-measured while the inverter sleeps, that would make the problem in this issue worse.

[johanzander]

Update: the data blocker is lifted, the forecast error is confirmed — but the cause is probably not the point-forecast

Came back to this while adjudicating #393 against three of one user's real bundles (25→26, 26→27, 27→28 Jul, Belgian wide-spread tariff, 15 kWh / 5 kW Growatt). Three things changed.

1. "No historical forecast-vs-actual data" is out of date

This issue's "what's needed before building this" section assumes we can't measure forecast-vs-actual overnight consumption. We can now. PredictionSnapshot carries the full DailyView — predictions merged with actuals — and #409 persists the store in the shared per-day file, so it survives into the debug bundle. The 28 Jul bundle carries 27 snapshots, one per optimization run, covering the whole night at 15-minute cadence.

So the data-collection phase this issue was waiting on is largely already running. Any recent bundle from a user with an overnight problem is a sample.

2. The forecast error is real and large

Measured against what each run actually planned:

Night	Planned battery discharge	Realised consumption	Error
26→27 Jul	1.565 kWh	3.10 kWh	-50%
27→28 Jul	2.70 kWh	3.70 kWh	-27%

On 26→27 the plan additionally started 0.55 kWh below where it expected to be at midnight. Combined deficit 2.09 kWh against 1.08 kWh realised import — consistent, and enough to explain the pre-dawn floor breach on its own.

Worth being precise about what this does and doesn't show: judged against the inputs each run actually had, both plans were sound. No algorithm defect. This is purely a P-vs-R forecast gap.

3. But the reason the DP can't absorb that error looks like the terminal value, not the point-forecast

This is the part that changes the proposed direction.

On the 28 Jul run, _calculate_terminal_value produced:

buy_based  = median_buy 0.3662 * eff_d - cycle = 0.3129
sell_cap   = max_sell    0.2046 * eff_d - cycle = 0.1594
export_prices_vary = True  ->  terminal_value = 0.1594

That 0.1594 matches the run's last-period shadow_price (0.15937) exactly — the cap is binding and it halves the value of carried energy on this tariff.

Consequence: overnight the DP values stored energy at 0.153-0.181 EUR/kWh while the morning buy price it would be displacing is 0.335-0.350. It therefore has almost no incentive to carry reserve past midnight, and plans to land exactly on min_soe with zero slack — directly visible in the 26 Jul 14:39 run, which puts SoE at 1.8 kWh at 06:45 and holds it flat until solar at 11:00.

With zero planned slack, every kWh of forecast error converts 1:1 into grid import. That is the actual transmission mechanism from "forecast was off" to "battery empty at dawn."

The arbitrage-consistency cap (#126/#244/#246) exists to stop the DP chasing a fictitious future bonus instead of exporting now, and on Nordic-shaped markets it stays inert as designed. But on a market where buy is ~2.4x sell, it conflates "realisable by export" with "realisable at all" — the energy's real next use is displacing a 0.35 EUR morning purchase, not a 0.20 EUR export.

Where that leaves this issue

The asymmetric-loss / conservative-quantile idea in the original description is still sound in principle, and it does attack the shared cause behind both this issue and #393. But it now looks like the second-best lever. A conservative consumption quantile pays a cost on every night to buy slack; correcting the terminal value would make the DP want to carry appropriate reserve on its own, for the right economic reason, without a buffer knob.

Concrete question to investigate: should the sell cap apply at all when the periods just past the horizon boundary are known to be high-buy, no-PV hours? The cap's own justification is that holding charge forgoes a real in-horizon export opportunity — which doesn't hold when the boundary is midnight and the next several hours are pre-dawn consumption at 3x the sell price.

Sizing, so this stays honest: this mechanism is worth roughly EUR 0.04/night on the one night where it bit, across a three-night sample with two import nights. Real and structural, but small. It should not jump the queue ahead of things with larger measured impact.

Caveats

• Three nights, one user, one market shape. The terminal-value finding is a mechanism confirmed in code and matched against logged values, but its economic weight is measured on a very thin sample.
• That user's saved sensor config has a stale battery_discharge_stop_soc binding pointing at the off-grid register, so his hardware floor was 10% while the optimizer planned against 12%. That widens the effective gap between planned and actual reserve on these nights and is being fixed separately — some of the measured breach is attributable to it rather than to forecast error.
• His HA automations touching the same registers are an unresolved confounder for the LOAD_SUPPORT shadow-price gate: near-tie residual leaves some overnight grid imports uncovered #393 half; less so for this one, since the floor breach is visible in the SoE trajectory regardless of what the rate register was doing.

Still no implementation planned. But the blocker has moved: it's no longer "we need data," it's "decide whether the terminal-value cap or a consumption quantile is the right lever," and the data to decide that is now being collected automatically.

[johanzander]

Your 2026-08-07 comment here ("the reason the DP can't absorb that error looks like the terminal value, not the point-forecast") was correct, and it now has a reproduction on a second market shape.

Filed as #602. Investigating #595 (Nordic/SE3, cycle_cost 0.4 SEK/kWh) hit the same mechanism you found on Frank's Belgian bundle: the sell cap binds and the DP plans to land on min_soe with zero slack.

Two things #602 adds beyond what was known here:

1. The terminal row is linear and unbounded in quantity (dp_battery_algorithm.py:1224-1227), so boundary SOE is a step function of the terminal value — flat at min_soe from 0.0 all the way to 0.65, then the entire transition inside 0.69–0.79. No scalar produces a moderate carry, so "pick a better terminal value" cannot work as a fix.
2. Your suggested lever overshoots on its own. Against an oracle that genuinely sees tomorrow (boundary SOE 4.37 kWh), production lands at 2.02 and removing the second cycle_cost deduction lands at 13.80 — an overshoot ~4× the undershoot it fixes. The proposed direction is a concave/piecewise terminal row instead, where a quantity (expected overnight consumption) carries the weight rather than a price point-estimate.

That second point is what makes the conservative-quantile idea in this issue's original description look better than "second-best lever", not worse: the robustness table in #602 shows the correct carry swings 2→18 kWh on what tomorrow turns out to be, so a point estimate of any kind is high-variance there.

Leaving this open — the forecast-quantile question is genuinely separate from #602's structural defect, and #602 does not subsume it.