LOAD_SUPPORT shadow-price gate: near-tie residual leaves some overnight grid imports uncovered

[johanzander]

Context

Split out from PR #385 (LOAD_SUPPORT shadow-price discharge gate, closing #384) during code review. The CHANGELOG and docs/agents/bess-knowledge.md entries for #385 originally cited #276 as the tracked follow-up for a documented residual limitation, but #276 is closed and is about an unrelated topic (DP grid-discretization/interpolation error from #275, concluded "financially optimal" per the #275/#276/#285 investigation — see PR #286). This issue is the correct tracking home for the #385-specific residual instead.

The residual limitation

docs/agents/bess-knowledge.md's "Known limitation, not yet fixed" note (added by #385): during a sustained overnight LOAD_SUPPORT drawdown, shadow_price tends to sit within a cent or two of buy_price for the entire stretch — close to the defining condition for choosing LOAD_SUPPORT at all — so intra_period_discharge_gate's strict inequality (buy_price × efficiency_discharge >= shadow_price) rarely opens in that regime. Small, real overnight grid imports (observed: 0.1-0.2 kWh/night, a few tens of cents) can still occur even with ample SOE headroom above the floor.

#385 explicitly rejected widening the gate's tolerance to paper over this — the gap between "genuinely arbitrary near-tie" (safe to open) and "real reserve correctly protected for a meaningfully pricier future peak" (must stay closed) is not currently distinguishable from the gate alone.

What this issue proposes investigating

Whether the near-tie is itself a symptom of DP value-function granularity/precision near the LOAD_SUPPORT decision boundary, or a fundamentally separate question requiring a different signal than shadow price alone to disambiguate "arbitrary tie" from "real future value." No prototype yet — filed so the residual isn't lost, not because a specific fix direction is proposed.

Note on the originating worked example

The specific worked example used during #385's investigation (Frank's debug bundle, period 6 on 2026-07-22 01:30: DP wrote a 9% discharge-rate ceiling, but the reported battery_discharged for that period was 0.2 kWh — ~1.8x what a hard 9% power cap should have permitted) turned out likely confounded by the sensor_collector.py runtime measurement-lag bug fixed in #387/#390. #387's fix only corrects forward (post-merge) collection via a new live per-minute power-sample buffer — it cannot retroactively reprocess that historical bundle, so this specific numeric discrepancy cannot be conclusively re-derived as clean or artifactual after the fact. It should not be relied on as evidence for or against this issue's residual gap; the residual gap itself is independently supported by the DP-replay evidence in #385 (Frank's actual price/consumption/solar history, not his disputed battery_discharged counter reading, replayed through the real optimize_battery_schedule + _apply_period_schedule path) and the architectural reasoning above.

Related

• fix: LOAD_SUPPORT discharge ceiling leaks real-time load spikes to grid #385 — where this residual was first documented, PR discussion has full review context
• feature: computed overnight reserve floor (Min SOC + historical night use) so one knob covers evening export + night load #381 — original report
• sensor_collector.py runtime-mode counter diffing not pinned to period boundaries, causing lag/misattribution #387 — sensor-collector measurement-lag fix, explains why the specific period-6 worked example is unreliable
• Investigate continuous-action/exact-piecewise-linear reformulation of the DP to eliminate discretization interpolation error #276 (closed) — previously (incorrectly) cited here; unrelated DP-discretization topic, not this issue's mechanism

[Frank-Leysen]

Cross-note from live b24 afternoon (26 Jul ~14:40–14:48 CEST) — not the overnight shadow-price near-tie residual tracked here.

Daytime event after upgrade to 9.9.0b24: discharge rate 0% under house deficit, brief battery→grid export during Storing Solar, then Battery First + ~3.6 kW grid charge at SOC 14% while the schedule strip showed Selling to Grid.

Full write-up + debug: #352 (comment)
Debug gist: https://gist.github.com/Frank-Leysen/da24149bff468d09f13765e691b5fa3c

[johanzander]

Reverting the LOAD_SUPPORT branch of intra_period_discharge_gate (#384/#385)

Follow-up investigation prompted by a second, independent real-world report (v9.9.0b24, TOU mode, GEN3 hardware): the DP plan showed a 5% LOAD_SUPPORT discharge rate, but the hardware log showed the register written at 100% for the same period, during a genuine solar deficit. We couldn't recover a planned-vs-applied trace for that exact window (the reporter's add-on restarted mid-day, wiping the in-memory schedule history for that period), so this specific incident isn't independently provable from logs alone — but it lines up with what #385 already told us about this gate.

Re-reading #385's own PR description: replaying Frank's original captured data (the case #384 was built for) through the real DP showed the gate does not open for the 3 leaked periods that motivated #384 — during a sustained overnight LOAD_SUPPORT drawdown, shadow_price sits within a cent or two of buy_price for the whole stretch, so the strict inequality rarely trips there. That gap was consciously deferred to this issue (#393) rather than blocking the merge.

Separately, replaying a full day of real DP output from the second reporter's own system (different day, different price shape) through the same gate condition (buy_price * eff_d >= shadow_price) shows it evaluates true for 51 of 67 LOAD_SUPPORT periods (76%) — i.e., in practice the gate is open far more often than "only when a real spike needs covering and the reserve genuinely isn't needed elsewhere," which was the design intent. That's a broad discharge-ceiling override for LOAD_SUPPORT, running on a shadow_price signal already documented here as unreliable in exactly this regime.

Net assessment: #384's fix shipped without validated benefit for its own motivating case, and independently exercises a wide, mostly-untested behavior surface for LOAD_SUPPORT in general — including plausibly explaining the new report above.

Decision: revert LOAD_SUPPORT from this gate (back to b23's plan-scaled-only cap for LOAD_SUPPORT — no shadow-price override). SOLAR_EXPORT/SOLAR_STORAGE keep the gate as before (older, unrelated, pre-dates #384). The original overnight-leak problem #381/#384 described stays open and tracked here, to be revisited with a mechanism actually validated against real captured data before it ships again — not reintroduced blind. A regression test built from the second reporter's real captured buy/shadow-price data will guard against this specific over-triggering pattern going forward.

PR incoming.

[Frank-Leysen]

Follow-up after b25 install — morning import looks like reserve depletion, not gate behaviour (27 Jul 2026)

Thanks for the revert rationale — makes sense, and aligns with what we saw on b24: the #384 gate was never going to help our original overnight near-tie case, and the broad 76% open rate on the second reporter's data is a good reason to pull LOAD_SUPPORT back out until something is validated properly.

Installed 9.9.0b25 yesterday evening + HA restart. Temp #352 HA guard remains off for a clean test.

This morning (~06:00–07:00 CEST)

• ~1.2 kWh grid import at ~0.37 EUR/kWh (peak morning buy window)
• SOC had fallen to 10% by ~05:19 — at the on-grid discharge stop, so battery discharge was 0 W despite Load First + discharge rate 100%
• House load ~300–400 W, PV just starting → remainder from grid

This does not look like the LOAD_SUPPORT gate over-triggering (or under-triggering) from #384. It looks like insufficient overnight reserve: yesterday afternoon's planned GRID CHARGING window did run (SOC climbed from ~13% to a peak of ~37% around 16:40), but the battery was then drawn down steadily overnight to the hardware floor before sunrise.

Relation to #393

This is the reserve-too-tight / plan-vs-reality side of the overnight problem, not evidence for or against the reverted gate. Happy to capture a debug bundle if useful — but unless you want a specific window, I suspect an overnight capture (plan + actual LOAD_SUPPORT periods + SoE trajectory) is more informative for #393 than a mid-morning snapshot with SOC already at the stop.

Let me know if you want a debug from last night or prefer we wait for the next overnight leak with b25.

[Frank-Leysen]

Overnight avoidable import while SOC still above stop (26→27 Jul, b24→b25 restart morning)

Correction / clarification to my earlier morning note: the ~1.2 kWh morning import at SOC=10% is only half the story. There were also clear avoidable imports overnight while SOC was still well above the discharge floor.

Live HA entity history (CEST)

At 22:00 BESS wrote number.pv_growatt_ems_discharging_rate = 0% (stayed 0 until the HA restart / b25 window). SOC at that point was still ~18–19% (on-grid stop = 10%, off-grid stop = 12%).

Window (CEST)	SOC	Grid import	Battery discharge	Notes
22:00–22:48	18%→17%	~100 W continuous	low / capped	rate already 0%
22:49–23:02	~17%	~1.2–1.25 kW	~300 W	classic boiler-reheat spike; shortfall bought from grid
00:33–03:27	13%→11%	short ~100 W pulses	—	same pattern on smaller spikes
04:04–04:17	~11%	~1.2–1.5 kW	—	second large spike; still above on-grid stop
after ~05:19	10%	continuous	0 W	hardware floor — this part is expected

So: LOAD_SUPPORT with a plan-scaled (here: zero) discharge ceiling → Load First cannot follow the overnight boiler / load spikes → retail import while reserve was still available. Matches the original #381/#393 overnight-leak class, not the morning empty-battery case.

Debug (b25, 08:07 CEST)

https://gist.github.com/Frank-Leysen/5c22ca4021401d366a2c4fb5b70f5d43

Notes on that bundle:

• Export is post-b25 + HA restart (system_uptime_hours: 0, DailyView had missing periods from the restart).
• Still useful for today's actual overnight SoE / import trajectory and the observed LOAD_SUPPORT periods.
• The cleanest rate=0 write evidence for the 22:49 window is in HA entity history above (b24 was still running then; temp bug: low-rate BATTERY_EXPORT writes grid_first at a forced sub-load discharge rate, causing avoidable grid import on any load spike #352 guard was off overnight).

Happy to pull another overnight debug on the next occurrence under clean b25 (no mid-night restart) if you want a pristine planned-vs-applied trace.

[Frank-Leysen]

Overnight boiler-spike import still present on b26 (27→28 Jul)

First clean overnight under 9.9.0b26 (temp #352 HA guard off). Daytime yesterday was stable after ~08:20 CEST (no further avoidable import/export), but the overnight LOAD_SUPPORT leak is still there.

Observed (CEST)

• Overnight grid import: ~0.3 kWh @ ~0.11 EUR — each time coincided with hot-water boiler reheat spikes
• Small overnight export: ~0.2 kWh @ ~0.03 EUR (secondary)
• Morning SOC: ~14–15% (better than the previous night's floor at 10% — reserve held longer)
• Mode: Load First throughout

Discharge-rate history (HA entity)

number.pv_growatt_ems_discharging_rate overnight stayed in the 5–12% band (e.g. 6% from 01:00, 5–6% after 04:00). On a 5 kW Growatt that is only ~250–600 W — not enough to cover a ~1–1.5 kW boiler spike → shortfall bought from the grid while SOC was still above the on-grid stop (10%).

Same class as earlier #393 / boiler reports: plan-scaled LOAD_SUPPORT ceiling can't follow intra-period spikes.

Debug (06:35 CEST, b26)

https://gist.github.com/Frank-Leysen/74940b6cebff6e3a7d6c5d646d0673bc

Happy to capture another night if useful; this one is a clean b26 overnight with the boiler pattern clearly visible in HA history + debug.