bug: discharge gate reads a raw snapped V grid difference while the policy uses _interpolate_value — shadow price 42% above the optimizer's own objective

[johanzander]

Reported by @Frank-Leysen on #520 — Growatt MOD 5000TL3-XH, BESS 10.1.0b7, debug bundle.

Symptom

2026-08-13, SOLAR_STORAGE / Load First, EV charging + pool pump. ems_discharging_rate was 100% for every period 13:00→14:15, 0% for period 58 (14:30) only, then 100% again at 14:45 — one isolated period importing ~1.1 kW from grid with SOC at 65% and 8 kWh of headroom.

The seam

Policy selection (_best_action_at_continuous_state, dp_battery_algorithm.py:1518) and reported future_value (:1774, :2021) evaluate the value function through _interpolate_value at the continuous SOE. _record_marginal_value (:913) alone uses a raw backward difference between snapped grid cells:

shadow_price = (V[t, i] - V[t, i - 1]) / SOE_STEP_KWH   # :913
allowed      = buy_price_t * efficiency_discharge >= shadow_price   # :915

For SOLAR_STORAGE the planned discharge rate is 0, so max(planned, gate) (battery_system_manager.py:2686) makes this the entire hardware decision.

_record_marginal_value's own docstring cites the P1 mirrored-implementation bug class as the reason both call sites route through one function. The two value estimators were never unified the same way.

Evidence

Replay at the production-faithful horizon (134 periods, terminal value 0.20289 via _calculate_terminal_value, battery_system_manager.py:1923-2013) reproduces the recorded shadow price bit-for-bit. The result is identical across horizon 38/134 × terminal 0/0.20289.

1. Production's own objective contradicts it. Finite-differencing reward_objective_cost over the exact cell the shadow price claims to measure, on the production grid itself:

cell	production objective	_record_marginal_value reports
[9.8750, 9.9000]	+0.17978	+0.25554

A +42% error — not a coarse-vs-fine comparison, an internal inconsistency.

2. Concavity violation in V[0].

cell [9.8250, 9.8500] -> +0.23602
cell [9.8500, 9.8750] -> +0.23602
cell [9.8750, 9.9000] -> +0.25554   <-- reported shadow_price
cell [9.9000, 9.9250] -> +0.17978
cell [9.9250, 9.9500] -> +0.17978

A single interior cell exceeding both neighbours. V(t,·) is concave in SOE for this problem (piecewise-linear reward, linear transition, convex feasible set), so this is np.round grid snapping (:1320, :1415) leaking into a one-cell difference.

3. Independent derivation from prices and physics. The plan from 14:30 charges on solar and saturates at max_soe = 15.0 kWh at period 71 (17:45). A marginal kWh stored at 14:30 therefore never reaches the evening BATTERY_EXPORT block — it displaces solar charging. A +0.25 kWh perturbation changes exactly one period's flows in 134 periods: period 71 grid_exported 0.052 → 0.310 kWh at sell 0.1404, less avoided charging wear.

sell(17:45) / efficiency_charge + cycle_cost
  = 0.1404 / 0.97 + 0.035
  = 0.1797       ≈ 0.17978

0.25554 corresponds to no price/physics combination that exists in this horizon.

4. The closed region is exactly one grid cell. Sweeping initial_soe at production resolution, all else fixed:

initial_soe (kWh)	gate
9.8850	OPEN
9.8900 – 9.9100	CLOSED
9.9150	OPEN

CLOSED for soe ∈ [9.8875, 9.9125) — precisely the set where round((soe − 1.8) / 0.025) == 324. 25 Wh wide, OPEN on both sides, with the objective exactly linear at 0.17978 across it and beyond in both directions. No kink, so no economic boundary.

SoC reaches the optimizer as whole percent (every initial SoE in the bundle is an exact integer % of 15.0 kWh: 13.000, 14.000, … 66.000, 65.000), a 150 Wh quantum. 9.900 kWh = 66.000% lands dead-centre on the corrupted node.

Correct gate: 0.2581 × 0.95 = 0.24516 > 0.17978 ⇒ should have been OPEN, margin +0.06538 EUR/kWh.

Cost impact — small, file as correctness not money

There is a genuine kink at SOE ≈ 9.85: below it the battery no longer saturates at period 71, the kWh does reach the evening export, and the true marginal value rises to 0.23602 (which is exactly the resolution-stable value at 14:45 — a useful cross-check), collapsing the margin to +0.00914.

Integrating the real value function against what was actually metered for period 58 (grid_imported 0.2 kWh):

covered import	avoided import	opportunity loss	net
0.2 kWh (metered)	0.05161	0.04684	+0.0048 EUR

Under half a eurocent for the one period that is in the data.

Scope caveat: the bundle was exported at 14:52, so it covers period 58 (CLOSED) and period 59 (OPEN, 100%) only. The reporter observed 0% again at 15:00 and 15:16 — periods 60/61 — which are not in this bundle and are therefore unverified. Whether those are the same artifact or a different cause needs a fresh export before the per-incident cost can be totalled.

Why existing issues don't cover this

• bug: SOE-grid discretization noise can pick sub-optimal charging window on near-tied periods #450 / fix: hybrid grid-DP + windowed exact-PWL tie resolution for #450 (alternative to PR #461's MILP) #467 established this exact mechanism and built tie_detection + PWL re-solve — but those protect action selection. The gate reads shadow_price raw and goes through none of it.
• LOAD_SUPPORT shadow-price gate: near-tie residual leaves some overnight grid imports uncovered #393 is about genuine near-ties, where shadow_price legitimately sits a cent from buy_price. This is not one: the resolution-stable margin is +0.065, 7× outside the band.
• Hysteresis: don't rewrite inverter schedule when the new plan is a near-tie with the applied one #485 hysteresis would mask the symptom without fixing the wrong value.
• Refining SOE_STEP_KWH is not the fix — 0.05 also gets it right. 0.025 is simply unlucky for this state.

Direction (not prototyped; docs/agents/optimizer-architecture.md is normative)

Make the gate consume the same value estimate the policy does.

Do not switch to a forward difference. _local_value_slope (:1447) already uses a forward difference at int(idx) while _record_marginal_value uses a backward difference at round(idx). For concave V a forward difference systematically under-values stored energy and would over-open the gate everywhere — it is only correct here by accident.

Any candidate fix should be measured against the fixture corpus for gate-flip rate between adjacent constant-price periods.

Separate concern, not this bug

The load forecast for this period was 1.96 kW against 4.8 kW actual (unmodelled EV charger + pool pump). That belongs with the vpp_power load-tracking discussion with @ridax67 on #520.

[bess-agent]

Status: held behind Phase 4a. #579 does not close this.

This issue is still open and unfixed. Recording the disposition so it isn't
re-derived.

What #579 fixes, and what it doesn't

#579 fixes a real but
different defect: _record_marginal_value snapped with round() while
_interpolate_value floors, so any SoE in the lower half of a cell was priced
off the cell below. That is worth fixing on its own merits.

It does not change the period reported here. Replaying period 58 at that PR's
head reproduces the bundle bit-for-bit and the gate is still CLOSED:

p58 soe=9.9000 SOLAR_STORAGE shadow=0.25553757 buy*eff=0.24515589 allowed=False

At soe = 9.9, idx = (9.9 - 1.8)/0.025 = 324.0 is exactly on a grid point, so
round() and ceil()-1 pick the same cell and the half-cell correction is
identically zero. And because SoC arrives as whole percent,
(k*0.15 - 1.8)/0.025 = 6k - 72 is always an integer — every optimization's
entry SoE lands on a node, so the offset fix can never reach period 0, which is
the period that drives the hardware write.

Closes #571 has been removed from that PR accordingly.

The actual mechanism here

The concavity violation, as the original writeup states. On the production grid
one interior cell sits above both neighbours, which a concave V cannot do:

[9.8500,9.8750] -> +0.23602
[9.8750,9.9000] -> +0.25554   <- read by the gate
[9.9000,9.9250] -> +0.17978

Re-solving at SOE_STEP_KWH = 0.005 gives 0.17978, matching this issue's
independent price/physics derivation (0.1404/0.97 + 0.035) to five digits, and
opening the gate.

Candidate estimators scored against that 0.005 reference across all 37 fixtures,
split by whether the coarse V is locally concavity-violating:

	n	current (backward)	central	concave envelope
violating	217	49.3% wrong gate, err 3.11	46.5%, 1.90	27.7%, 1.32
clean	1642	25.8%, 0.81	23.5%, 0.77	23.1%, 0.85

Reading the slope off the upper concave envelope gives 0.22889 < 0.24516 here
⇒ OPEN, the correct decision.

Two notes for whoever picks this up:

• The gate-flip rate between adjacent same-price periods, suggested in the
  writeup as the acceptance metric, does not discriminate: 9/702, 10/702,
  9/702 across the three estimators. Use the fine-grid reference instead.
• Reproduce from period 58, not 59: horizon 134, initial_soe 9.9,
  initial_cost_basis 0.0363146679449133, built by prepending period 58 onto
  the regression_2026_08_13_145213 fixture using the bundle's own 14:30:10
  prediction snapshot. The period-59 fixture is a different run against a
  different V and matches nothing recorded in the bundle.

Why held

Both this fix and #579 are deferred behind Phase 4a. The parent plan records
estimator unification as a deliberate behaviour change
(2026-08-09-optimizer-target-architecture.md: "unifying them would be a
behavior change"), and 4a — approved decision D1 — relocates
intra_period_discharge_gate into a new core/bess/execution_model.py leaf,
with the requirement that 4a stay behaviour-neutral so 4b's delta stays
readable. Landing either gate change first tangles that measurement.

Neither fix technically depends on 4a; this is sequencing, to avoid measuring
the same delta twice on code that is about to move.