Promotion gate: candidate with significantly better paired evidence rejected (NO_GO) on underpowered critical slices

[BMeyendriesch]

Summary

On my installation the promotion gate returned NO_GO for a candidate that was significantly better than the serving champion on the shared paired evidence — 7.86 vs 9.64 kWh daily energy MAE, with the 95 % confidence interval of the difference entirely below zero. It was rejected solely on the critical-slice checks, three of which had too few independent days to be evaluated at all.

The evaluation record is stored in full in ai_model_evaluations_v2, so this is not a reconstruction. Everything below is copied from that row.

I also want to raise a design question: with min_independent_eligible_days = 7 and critical_slice_min_independent_days = 3 across 6 weather regimes per cycle, I do not see how all slices can ever be adequately powered.

Environment

Home Assistant	core-2026.8.2 (HAOS)
SFML at evaluation time	V44.0.2
SFML now	V44.2.0 (installed 2026-08-17 09:21)
Plant	15.30 kWp, Kostal Piko, 3 strings (89°/45°, 269°/44°, 179°/37°)
Database	154 MB, in continuous use since January 2026
Model type	tiny_lstm

Timeline

Date	Event
2026-07-28 23:33	Champion trained, R² 0.930 — still the serving model today
2026-07-29 23:32	Candidate 1 71028c55 created, R² 0.857
from 2026-07-30	Every night: AI model training skipped: candidate shadow evaluation pending for 71028c55…
2026-08-02 03:02	Candidate 2 4c44d872, R² 0.838
2026-08-03 15:46	Candidate 3 768c5a52, R² 0.849 (manual retrain_ai_model call)
2026-08-11	contract_version moved model_candidate_shadow_v1 → v2, evidence counter reset
2026-08-16 23:30	Evaluation runs, 7/7 independent days reached → NO_GO
2026-08-17 00:25	Queue advances to candidate 3, counter back to 0
2026-08-17 09:21	Update to V44.2.0; terminal event no_go written for candidate 1

Net effect: the serving model has not changed since 2026-07-28, and system_status reports Last ML training was 20 days ago.

The evaluation record

From ai_model_evaluations_v2, evaluation 5f691d22-266a-482c-a1a7-449e6e7af7b2, contract cycle_aware_model_promotion_v2:

decision:                       NO_GO
reason_codes:                   ["CRITICAL_SLICE_REGRESSION", "CRITICAL_SLICE_UNDERPOWERED"]
eligible_hour_count:            198
eligible_independent_day_count: 7

daily_energy_mae:               candidate 7.856   champion 9.636
candidate_minus_champion_daily_energy_mae_ci_95:  [-3.057, -0.526]
cycle_mean_daily_energy_deltas: post_midnight -1.743   pre_sunrise -1.818

Per-day paired deltas (negative = candidate better): −0.929, −3.252, −2.650, +0.040, −1.800, +0.675, −4.545. The candidate was better on 5 of 7 days and never worse by more than 0.7 kWh.

The critical slices

allowed_daily_energy_regression is 0.0 for all twelve slices, min_independent_days is 3:

slice	days	adequately powered	delta (kWh)
post_midnight · bright_mixed	1/3	false	+0.0730
post_midnight · dry_cloud_edge	3/3	true	−1.1560
post_midnight · low_ghi_overcast	3/3	true	−0.0200
post_midnight · low_sun_uncertain	2/3	false	+0.0305
post_midnight · mixed_uncertain	7/3	true	−0.4051
post_midnight · stable_good	6/3	true	−0.9880
pre_sunrise · bright_mixed	2/3	false	+0.0430
pre_sunrise · dry_cloud_edge	3/3	true	−0.5280
pre_sunrise · low_ghi_overcast	3/3	true	−0.0047
pre_sunrise · low_sun_uncertain	3/3	true	+0.0204
pre_sunrise · mixed_uncertain	7/3	true	−0.7953
pre_sunrise · stable_good	6/3	true	−0.9235

Four slices show a positive delta between 20 and 73 Wh. Three of those four are flagged as not adequately powered, so the same slices trigger both reason codes. A model that is on average 1.8 kWh per day better is blocked by differences that are two orders of magnitude smaller and, in three cases, rest on one or two days of data.

The design question

Seven independent days have to be distributed over 6 weather regimes per cycle, and each slice needs at least 3 of them. Seven days can supply 3+ days to at most two regimes. At least four of the six regimes per cycle will therefore be underpowered in any evaluation window, unless the weather is unusually monotonous for a week.

If CRITICAL_SLICE_UNDERPOWERED is blocking on its own, the gate looks structurally unpassable at the current settings. If it is only advisory and CRITICAL_SLICE_REGRESSION did the blocking, then a 20 Wh difference against a 0.0 allowance is doing the same job — with the regression measured on one or two days in three of the four affected slices.

Would it be worth considering one of these:

• ignoring slices that are not adequately powered instead of counting them as regressions,
• a tolerance for allowed_daily_energy_regression that scales with the noise of the slice rather than a hard 0.0,
• or letting the overall result outweigh slice regressions when the global confidence interval is entirely on the candidate's side.

Secondary observation

The governance has never activated a model on this installation:

ai_active_model_pointer:     0 rows
ai_model_promotion_events:   0 rows
ai_model_evaluations:        0 rows      (ai_model_evaluations_v2: 1 row)

and sensor.solar_forecast_ml_ai_metrics reports:

serving_source:   legacy_fallback
bootstrap_active: true

So the serving R² 0.930 model is being used through the legacy fallback path rather than through the governance pointer. I mention it because it may be related to the bootstrap issues discussed in the forum and addressed in V44.0.4.

Relation to #185

This is not the same failure as #185. There the candidate never moves and reset_ai_model does not clear it. Here the chain works: shadow runs accumulate, the evaluation runs, a decision is recorded, a terminal event is written and the queue advances. My problem is the content of the decision, not a stuck state.

Reproducing the readout

import sqlite3, json
con = sqlite3.connect("file:/config/solar_forecast_ml/solar_forecast.db?mode=ro", uri=True)
c = con.cursor()
c.execute("SELECT decision, reason_codes_json, decision_json FROM ai_model_evaluations_v2")
dec, reasons, dj = c.fetchone()
d = json.loads(dj)
print(dec, reasons)
print(d["daily_energy_mae"], d["candidate_minus_champion_daily_energy_mae_ci_95"])
for s in d["critical_slice_results"]:
    print(s["name"], s["eligible_independent_day_count"], s["adequately_powered"],
          round(s["candidate_minus_champion_daily_energy_loss"], 4))

Note on versions

The evaluation above ran under V44.0.2, i.e. before the promotion gate rework in V44.0.4. I have been on V44.2.0 since 2026-08-17, and the next candidate is currently at 1 of 7 days, so the first decision under the reworked gate is due around 2026-08-24. I am reporting this now because the recorded decision documents the slice arithmetic clearly, and because the thresholds in the stored evaluation_config_json may well be unchanged. I will follow up here with the outcome of the next evaluation.

Happy to supply the complete evaluation_config_json and decision_json, the shadow run tables, or anything else that helps.

[Zara-Toorox]

Hi all..!
Thanks for the complete ai_model_evaluations_v2 record. That is enough to treat this as a real gate decision, not a stuck queue.

The recorded NO_GO under V44.0.2 is consistent with that build: the evaluation submitted every observed weather regime on both cycles. Three of the four positive slice deltas were not adequately powered (1/3 or 2/3 days). Those slices still produced CRITICAL_SLICE_UNDERPOWERED, and the gate failed even though the paired daily evidence was clearly better (7.86 vs 9.64 kWh MAE, CI entirely below zero). That is the failure you documented. It is not the same bug as #185.

That behaviour is already corrected in current source (after V44.2.0). Underpowered regimes are no longer part of the promotion payload. A regime is only evaluated when both post_midnight and pre_sunrise have at least 3 independent days. Otherwise the candidate stays in shadow collection (pending) instead of receiving a terminal no_go. On your stored slice counts, bright_mixed and low_sun_uncertain would be held out entirely, including the +20 Wh pre_sunrise · low_sun_uncertain row. The remaining powered regimes in that record were all negative for the candidate.

So the next evaluation on V44.2.0 can still repeat the old arithmetic. The first run that matters is the one after you have a build that includes this writer change. Then the outcome is either GO / NO_GO on the powered subset, or more waiting if fewer than 7 common days remain after the filter — not a terminal reject because a 1- or 2-day slice was underpowered.

What stays intentional: allowed_daily_energy_regression = 0.0 on slices that are adequately powered. A real, well-supported weather-context regression still blocks promotion. We are not changing that on the back of this issue. A noise floor or a rule that the global CI overrides powered-slice regressions would be a separate product decision.

serving_source: legacy_fallback with an empty ai_active_model_pointer matches “no successful governed promotion yet”. The Jul 28 model can still serve through the legacy weight path. That is related to bootstrap/governance, not to this NO_GO record.

Happy to look at the next ai_model_evaluations_v2 row after the updated build. (after my actual vacation)

[BMeyendriesch]

Thanks for the detailed answer — that matches the record, and I agree the powered-slice rule should stay as it is.

Status update: V44.2.2 installed on 2026-08-20 at 08:07 local, after refreshing the HACS repository information. No errors on startup. Current candidate state (same candidate 768c5a52, no new training in between):

eligible_independent_days:  3 / 7
post_midnight_runs:         4
pre_sunrise_runs:           4
pending_candidate_count:    2
validation_status:          in_progress
serving_source:             legacy_fallback (champion of 2026-07-28, R² 0.930)

So the evidence path is advancing normally; at one day per night the gate should reach 7/7 around 2026-08-24. I will post the resulting ai_model_evaluations_v2 row here — whichever way it goes — so you have a first real-world evaluation from the new writer. No action needed from your side until then; enjoy the vacation.

[BMeyendriesch]

Follow-up: the evaluation for candidate 768c5a52 has completed under V44.2.2 — and the powered-slice fix works as intended.

ai_model_evaluations_v2, evaluation 86b2e7dd, created 2026-08-23T22:08:11+02:00:

decision:            NO_GO
eligible days/hours: 7 / 174
reason_codes:        CANDIDATE_NOT_SUPERIOR, REQUIRED_CYCLE_REGRESSION, CRITICAL_SLICE_REGRESSION
daily_energy_mae:    candidate 4.25 kWh vs champion 4.38 kWh
MAE delta CI95:      [-0.73, +0.56]  (spans zero)
cycle deltas:        post_midnight -0.68 kWh (better), pre_sunrise +0.43 kWh (worse)

Every critical slice was adequately powered this time (3-5 independent days each, minimum 3) — no CRITICAL_SLICE_UNDERPOWERED anywhere. Three slices genuinely regress: dry_cloud_edge on both cycles and rain_overcast (post_midnight). So unlike the V44.0.2 case documented above, this NO_GO looks substantive rather than an artifact: the candidate is only marginally better overall and clearly worse pre-sunrise. No complaint from my side — the gate did its job.

The bookkeeping afterwards was clean as well: terminal event no_go (model-candidate-retention-v1) at 22:08:31, and the queue advanced to the last pending candidate 4c44d872 (trained 2026-08-02, R2 0.838), now validating at 0/7 with pending_candidate_count: 1.

One question: that remaining candidate is older and weaker than the serving champion (R2 0.838 vs 0.930), so a third NO_GO in about a week seems likely. Once the queue is empty, will nightly training resume automatically, or does it need a nudge?