investigate: near-threshold BATTERY_EXPORT flips between runs (root cause unverified)

[johanzander]

Status: original analysis was wrong — re-scoped to real findings

The original plan (repurpose min_action_profit_threshold as a per-period
sell_price − cycle_cost floor) is withdrawn. It rested on an incorrect
diagnosis and would have encoded a gross-value error. The real findings are below;
follow up separately, and verify before any code change.

Symptom (real, economically negligible)

In some optimization runs, slot 15:45 (period 63) is scheduled as BATTERY_EXPORT
with ~0.1 kWh / 4% discharge; in later runs the same slot is SOLAR_EXPORT (no
discharge). Battery is full (SOE 20.0 kWh) with a solar surplus. Net effect of the
trade ≈ −0.04 SEK (a small loss). Visible as a brief SOC dip / red bar.

Why the original fix was wrong

1. The mechanism is not missing. _compute_reward
   (core/bess/dp_battery_algorithm.py:376-389) already raises the discharge floor
   to sell_price when solar will replenish the battery. In the 11:31 run,
   cost_basis 0.62 > sell 0.46 correctly blocked the discharge.
2. sell_price − cycle_cost is the gross-value fallacy. The real alternative to
   discharging at 15:45 is "let solar charge one more slot and export one slot
   earlier", so the value captured is only the sell-price differential (~6 öre),
   not the full sell price. Against ~40 öre wear that is a loss. A flat
   sell − wear floor compares gross value to wear and would re-introduce exactly
   this error. (See the governing law in docs/agents/bess-knowledge.md.)

Candidate real causes (NEITHER verified — do not fix yet)

A. Shadow-price volatility across runs. The flip is driven by shadow_price for
period 63 moving across re-optimizations near the discharge threshold (e.g.
different measured SOC / solar forecast between the 10:16 and 11:31 runs).

B. Anti-cycling gate over-values stored energy in solar-surplus slots. The gate
uses effective_value = max(buy×eff_d, sell×eff_d)
(dp_battery_algorithm.py:356-359). In this slot solar already covers all home
load, so there is no grid import to avoid — the discharged kWh can only be
exported, making the true value sell×eff_d (~0.41), not buy×eff_d (~0.95).
Including the buy term lets a marginal discharge slip past the gate.

Next step (separate work)

Build a reproduction test (solar > home, battery full, small local sell spike) and
determine which of A/B actually drives the flip before changing any code. Then
decide on the fix. Do not implement a min_action_profit margin-floor parameter.

[bess-manager-claude-bot]

This issue proposes a semantic change to min_action_profit_threshold: shift from a post-optimization session-level gate to a per-period margin floor checked inside the DP loop. The issue includes a concrete plan with affected files and rationale. This is queued for human review to validate the design tradeoffs (e.g., removing the session gate vs. keeping it as a separate parameter). Reply @claude-bot analyze to request deep code-level analysis of the current implementation and the proposed changes.

[johanzander]

Additional analysis: why the true gain is much less than sell_price − wear_cost

The initial framing ("6 öre margin above wear cost") understates how unprofitable this trade is.

The correct opportunity cost frame

When the battery is still in a solar charging window (SOLAR_STORAGE), the alternative to BATTERY_EXPORT at slot T is not "do nothing." It is: continue charging from solar at slot T, finish charging one slot earlier, and start SOLAR_EXPORT one slot earlier.

The true gain from selling at T is therefore:

True gain = sell_price[T] − sell_price[when_solar_export_would_start] − wear_cost

In the 15:45 case:

True gain = 0.46 − ~0.40 (end-of-charging sell price) − 0.40 (wear cost)
          = −0.34 SEK/kWh  →  a clear loss

The 6 öre is the price differential between selling at 15:45 vs. the counterfactual solar export — not the net gain. And that 6 öre differential is well below the 40 öre wear cost.

The shadow price mechanism is supposed to catch this — and sometimes does

This is exactly what cost_basis and shadow_price are designed to capture. In the 11:31 optimization run at period 63 (15:45):

"cost_basis": 0.6219,
"shadow_price": 0.8755,
"sell_price": 0.46

Because sell_price (0.46) < cost_basis (0.62), the optimizer correctly chose NOT to discharge → SOLAR_EXPORT, battery_action ≈ 0. The mechanism worked.

But earlier runs got it wrong due to forecast sensitivity

Optimization runs from 10:16 to 11:16 all scheduled BATTERY_EXPORT at 15:45 with −0.1 kWh discharge. These runs must have computed a lower cost_basis at period 63 (below 0.46 SEK) because the solar production forecast at 10:16 differed from the 11:31 forecast. As the solar forecast updated through the morning, the shadow price at period 63 drifted across the threshold.

This is a near-threshold sensitivity problem: when the true gain is close to zero, small changes in the solar forecast can flip the decision from correct (no discharge) to incorrect (discharge). The per-period min_action_profit_threshold fix adds a safety margin that prevents the optimizer from acting in this zone of uncertainty, regardless of which way the shadow price happens to land on a given run.

Implication for the fix

The min_action_profit_threshold floor should be expressed in terms of margin above wear cost per kWh, not just absolute sell price. The filter condition should be:

if sell_price[t] - cycle_cost_per_kwh < min_action_profit_threshold:
    # Don't discharge — margin is below minimum

A default of 0.05 SEK/kWh (5 öre) would have blocked the 15:45 trade (6 öre margin - noise, not signal) while leaving the evening peak (margins of 1.5–2.0 SEK/kWh) completely unaffected.

[bess-manager-claude-bot]

Thanks for the detailed analysis. I've classified this as an investigation task queued for implementation:

1. Root cause: either shadow-price volatility (A) or anti-cycling gate logic (B) — both plausible, neither verified yet.
2. Next step: build a reproduction test (solar > home, battery full, small sell spike) and confirm which path drives the flip.
3. Defer implementation: no code change until the test confirms the mechanism.

This is ready for Stage 2 analysis if you'd like deep code-level verification of the proposed test strategy or the candidate mechanisms. Reply @claude-bot analyze to request it.

[johanzander]

Closing this in favor of #204.

This thread reached a self-contradictory state: the body was edited to withdraw
the sell_price − cycle_cost floor as a "gross-value fallacy," but the comment
above still argues for exactly that withdrawn approach, and no later comment
reconciled the two.

Re-analyzed against a fresh debug bundle (docs/bess-debug-2026-06-28-113159.md,
2026-06-28) with scripts/extract_decision_evidence.py. Candidate B from this
issue is confirmed as the actual root cause, not just a plausible candidate:
_compute_reward's anti-cycling discharge gate includes avoid_purchase_value (buy_price × efficiency_discharge) in a max() even in periods where solar
already covers all home load and there's no grid import for a discharge to
displace — letting the gate's assessed value hugely overstate what the
discharge can actually realize (export_value only). This held across all 6
optimization runs captured that day for period 63 (15:45); only the
value-function comparison in the last run (11:31) happened to save it, not the
gate itself.

Comment above's min_action_profit_threshold = sell_price − cycle_cost proposal
should not be implemented — cost_basis already bakes in wear cost via FIFO
accounting at charge time, so a second flat margin floor doesn't fix the
gate itself and would only paper over some cases while leaving the bug live in
others.

#204 has the full root-cause writeup, code citations, and a scoped fix
direction (exclude avoid_purchase_value from the gate when there's no
counterfactual grid import), plus the full debug bundle as a linked gist.