fix(output): apply load_scaling_dynamic and manual_load_adjust to today_remaining (#4511)

* fix(output): apply load_scaling_dynamic and manual_load_adjust to today_remaining (#4496 follow-up)

#4506 fixed today_remaining diverging from the plan's own remaining-load
total by applying the flat load_scaling factor, but missed two other
multipliers step_data_history() (fetch.py) also applies when building the
plan's load_minutes_step: load_scaling_dynamic (per-minute - carries
saving-session/free-electricity-event scaling, and any per-window
override from rates_import_override or the manual API) and
manual_load_adjust (additive, per-minute).

Confirmed via a live report on the same issue: a user with a 2-hour
"power up" (free electricity) event set to load_scaling: 1.5 via the
manual API saw today_remaining still diverge from the plan's own total
even after updating to the #4506 fix, since only the flat 1.05 base
load_scaling was being applied. Replayed their attached debug.yaml
directly - their load_scaling_dynamic dict does carry 1.5 for the 2-hour
window as expected; load_today_comparison() never looked at it. After
this fix, the same replay's plan-vs-sensor ratio drops from a much
larger gap to 1.0076, matching the same small residual seen on the
original #4506 replay.

New tests confirm today_remaining scales with load_scaling_dynamic
(uniformly across the day, mirroring the existing load_scaling test) and
reflects a single manual_load_adjust entry additively. Both confirmed to
fail without this fix and pass with it.

* cspell: rephrase comments to avoid a reporter's GitHub handle

Follow-up to the previous commit - cspell flagged the handle as an
unknown word; reworded rather than adding it to the dictionary.

* fix(fetch): reword days_previous_auto log to not imply it ran this cycle

Raised alongside the #4496 follow-up investigation: the "days_previous_auto
enabled - using weighted-bucket historical load forecast..." log line
fires unconditionally every cycle purely because days_previous_auto
defaults to True, regardless of whether the weighted-bucket forecast is
actually used that cycle. Load ML (or any source that sets
load_forecast_only) takes precedence and skips it entirely -
fetch_sensor_data() already has its own, correctly conditional "Using
weighted-bucket historical load forecast over N days" line that only
logs when the fallback genuinely runs.

Reworded to describe what's enabled/configured, not what happened, so a
Load ML user reading the log isn't misled into thinking both forecast
sources are being blended every cycle when they aren't - confirmed via
code trace, not just the log wording, that no double-application
actually occurs.

Author: chalfontchubby

apps/predbat/fetch.py

@@ -2366,7 +2366,16 @@ def fetch_config_options(self):
         self.load_forecast_history = self.get_arg("days_previous_auto", True)
         if self.load_forecast_history:
             window_days = min(max(self.days_previous) if self.days_previous else 7, LOAD_FORECAST_HISTORY_MAX_DAYS)
-            self.log("days_previous_auto enabled - using weighted-bucket historical load forecast over up to {} days".format(window_days))
+            # Config-time log only - describes what's enabled, not what happened this cycle. This
+            # runs unconditionally every cycle regardless of whether the weighted-bucket forecast
+            # actually gets used: Load ML (or any other source that sets load_forecast_only) takes
+            # precedence and skips it entirely (fetch_sensor_data(), guarded by
+            # "not self.load_forecast_only"). The "using weighted-bucket..." wording previously
+            # here read as if it was happening every cycle regardless, which is what actually gets
+            # logged only when the forecast is genuinely used (fetch_sensor_data()'s own "Using
+            # weighted-bucket historical load forecast over N days" line) - confusing on a Load ML
+            # setup where this fallback is rarely/never actually invoked (#4496 follow-up).
+            self.log("days_previous_auto enabled - will fall back to a weighted-bucket historical load forecast over up to {} days if no other load forecast source takes precedence".format(window_days))
             self.max_days_previous = window_days + 1
         elif self.holiday_days_left > 0:
             self.days_previous = [1]

apps/predbat/output.py

@@ -2588,19 +2588,34 @@ def load_today_comparison(self, load_minutes, load_forecast, car_minutes, import
             load_value_pred += forecast_value_pred
             load_value_pred_raw += forecast_value_pred
 
-            # For FUTURE minutes only, apply load_scaling so the published predicted/adjusted
-            # curves (and their today_remaining attribute) match step_data_history() (fetch.py),
-            # which the plan itself uses to build load_minutes_step as
-            # (value + load_extra) * scale_fixed, where scale_fixed includes load_scaling.
-            # Minutes already elapsed today are deliberately left unscaled: load_total_pred_now
+            # For FUTURE minutes only, apply load_scaling, load_scaling_dynamic, and
+            # manual_load_adjust so the published predicted/adjusted curves (and their
+            # today_remaining attribute) match step_data_history() (fetch.py), which the plan
+            # itself uses to build load_minutes_step as
+            # (value + load_extra) * scaling_dynamic * scale_fixed, where load_extra includes
+            # manual_load_adjust, scaling_dynamic is load_scaling_dynamic, and scale_fixed
+            # includes the flat load_scaling. load_scaling_dynamic carries saving-session/
+            # free-electricity-event scaling as well as any per-window override from
+            # rates_import_override/the manual API (e.g. a "power up" event) - a first pass at
+            # this fix (#4506) only applied the flat load_scaling and missed both of these,
+            # confirmed against a real follow-up report on issue #4496 where a 1.5x
+            # load_scaling_dynamic override for a 2-hour power-up event wasn't reflected in
+            # today_remaining at all.
+            #
+            # Minutes already elapsed today are deliberately left untouched: load_total_pred_now
             # below feeds the actual-vs-predicted divergence ratio, which compares actual
-            # consumption against the RAW model, not a load_scaling-corrected one. Previously
-            # load_scaling was never applied here at all, so with load_scaling != 1.0 the
-            # today_remaining attribute diverged from the plan's own remaining-load total by
-            # exactly that factor (issue #4496).
+            # consumption against the raw model, not an adjusted one.
             if minute >= minutes_now:
-                load_value_pred *= self.load_scaling
-                load_value_pred_raw *= self.load_scaling
+                manual_adjust = 0.0
+                if self.manual_load_adjust:
+                    manual_adjust = self.manual_load_adjust.get(minute, 0) * step / float(self.plan_interval_minutes)
+                    manual_adjust = max(manual_adjust, -load_value_pred)
+                load_value_pred += manual_adjust
+                load_value_pred_raw += manual_adjust
+
+                scaling_dynamic = self.load_scaling_dynamic.get(minute, 1.0) if self.load_scaling_dynamic else 1.0
+                load_value_pred *= self.load_scaling * scaling_dynamic
+                load_value_pred_raw *= self.load_scaling * scaling_dynamic
 
             # Track (but no longer exclude) periods where import exceeds raw load, assumed to
             # include deliberate battery charging (overnight for example). The house's own load

apps/predbat/tests/test_load_today_comparison.py

@@ -310,6 +310,99 @@ def _test_load_scaling_applied_to_predicted(my_predbat, failed):
     return failed
 
 
+def _test_load_scaling_dynamic_and_manual_adjust_applied_to_predicted(my_predbat, failed):
+    """
+    Follow-up regression test for issue #4496: the fix in _test_load_scaling_applied_to_predicted
+    only applied the flat self.load_scaling, but step_data_history() (fetch.py) also applies
+    self.load_scaling_dynamic (per-minute - saving-session/free-electricity-event scaling, and
+    any per-window override from rates_import_override/the manual API) and self.manual_load_adjust
+    (additive, per-minute). Confirmed against a real follow-up user report where a 1.5x
+    load_scaling_dynamic override for a 2-hour "power up" event wasn't reflected in
+    today_remaining at all even after the first fix landed.
+    """
+    print("  test: load_scaling_dynamic and manual_load_adjust are applied to the predicted (today_remaining) load curve")
+
+    saved = {
+        "car_charging_hold": my_predbat.car_charging_hold,
+        "car_charging_energy": my_predbat.car_charging_energy,
+        "iboost_energy_subtract": my_predbat.iboost_energy_subtract,
+        "iboost_energy_today": my_predbat.iboost_energy_today,
+        "base_load": my_predbat.base_load,
+        "load_forecast_only": my_predbat.load_forecast_only,
+        "days_previous": my_predbat.days_previous,
+        "days_previous_weight": my_predbat.days_previous_weight,
+        "load_minutes_age": my_predbat.load_minutes_age,
+        "load_scaling": my_predbat.load_scaling,
+        "load_scaling_dynamic": my_predbat.load_scaling_dynamic,
+        "manual_load_adjust": my_predbat.manual_load_adjust,
+        "now_utc": my_predbat.now_utc,
+        "midnight_utc": my_predbat.midnight_utc,
+        "minutes_now": my_predbat.minutes_now,
+    }
+
+    try:
+        my_predbat.car_charging_hold = False
+        my_predbat.car_charging_energy = None
+        my_predbat.iboost_energy_subtract = False
+        my_predbat.iboost_energy_today = None
+        my_predbat.base_load = 0.0
+        my_predbat.load_forecast_only = False
+        my_predbat.days_previous = [1]
+        my_predbat.days_previous_weight = [1.0]
+        my_predbat.load_minutes_age = 1
+        my_predbat.load_scaling = 1.0
+
+        midnight_utc = datetime(2026, 1, 1, 0, 0, 0, tzinfo=UTC)
+        minutes_now = 780  # 13:00, well clear of the day boundary
+        my_predbat.midnight_utc = midnight_utc
+        my_predbat.now_utc = midnight_utc + timedelta(minutes=minutes_now)
+        my_predbat.minutes_now = minutes_now
+
+        load_minutes = build_cumulative(0.02, 3000)  # 0.02 kWh/min -> 0.1 kWh per 5-min bucket
+        load_forecast = {}
+        import_minutes = build_cumulative(0.0, 3000)
+
+        # Baseline: no dynamic scaling, no manual adjustment
+        my_predbat.load_scaling_dynamic = {}
+        my_predbat.manual_load_adjust = {}
+        my_predbat.load_today_comparison(load_minutes, load_forecast, {}, import_minutes, minutes_now=minutes_now, step=5, save=True)
+        baseline_remaining = my_predbat.dashboard_values[my_predbat.prefix + ".load_energy_predicted"]["attributes"]["today_remaining"]
+
+        # load_scaling_dynamic applied uniformly across every future minute at 2.0x - the whole
+        # remaining-today total should double, exactly like the flat load_scaling test does
+        my_predbat.load_scaling_dynamic = {minute: 2.0 for minute in range(minutes_now, 24 * 60, 5)}
+        my_predbat.load_today_comparison(load_minutes, load_forecast, {}, import_minutes, minutes_now=minutes_now, step=5, save=True)
+        dynamic_scaled_remaining = my_predbat.dashboard_values[my_predbat.prefix + ".load_energy_predicted"]["attributes"]["today_remaining"]
+
+        expected_dynamic = round(baseline_remaining * 2.0, 2)
+        if abs(dynamic_scaled_remaining - expected_dynamic) > 0.02:
+            print("  ERROR: today_remaining with load_scaling_dynamic=2.0 across the day should be ~{} (2x baseline {}), got {}".format(expected_dynamic, baseline_remaining, dynamic_scaled_remaining))
+            failed = True
+        else:
+            print("  PASS: today_remaining scales with load_scaling_dynamic ({} -> {} at 2x)".format(baseline_remaining, dynamic_scaled_remaining))
+
+        # manual_load_adjust applied additively at a single future minute
+        my_predbat.load_scaling_dynamic = {}
+        manual_adjust_minute = minutes_now + 60
+        manual_adjust_kwh = 6.0
+        my_predbat.manual_load_adjust = {manual_adjust_minute: manual_adjust_kwh}
+        my_predbat.load_today_comparison(load_minutes, load_forecast, {}, import_minutes, minutes_now=minutes_now, step=5, save=True)
+        manual_adjusted_remaining = my_predbat.dashboard_values[my_predbat.prefix + ".load_energy_predicted"]["attributes"]["today_remaining"]
+
+        expected_delta = manual_adjust_kwh * 5 / float(my_predbat.plan_interval_minutes)
+        expected_manual = round(baseline_remaining + expected_delta, 2)
+        if abs(manual_adjusted_remaining - expected_manual) > 0.02:
+            print("  ERROR: today_remaining with a manual_load_adjust of {}kWh at minute {} should be ~{} (baseline {} + {:.2f}), got {}".format(manual_adjust_kwh, manual_adjust_minute, expected_manual, baseline_remaining, expected_delta, manual_adjusted_remaining))
+            failed = True
+        else:
+            print("  PASS: today_remaining reflects manual_load_adjust ({} -> {}, +{:.2f}kWh)".format(baseline_remaining, manual_adjusted_remaining, expected_delta))
+    finally:
+        for key, value in saved.items():
+            setattr(my_predbat, key, value)
+
+    return failed
+
+
 # ---------------------------------------------------------------------------
 # Entry point
 # ---------------------------------------------------------------------------
@@ -320,13 +413,15 @@ def test_load_today_comparison(my_predbat):
     Unit tests for load_today_comparison() covering the None-guard fix
     for dp2() calls when filtered_today() returns None, the
     import-exceeds-load regression (batpred#4154, #2537), and the
-    load_scaling-not-applied regression (#4496).
+    load_scaling/load_scaling_dynamic/manual_load_adjust-not-applied
+    regression (#4496).
     """
     failed = False
     print("**** Running load_today_comparison tests ****")
 
     failed = _test_none_guard_no_crash(my_predbat, failed)
     failed = _test_import_exceeding_load_still_counted(my_predbat, failed) or failed
     failed = _test_load_scaling_applied_to_predicted(my_predbat, failed) or failed
+    failed = _test_load_scaling_dynamic_and_manual_adjust_applied_to_predicted(my_predbat, failed) or failed
 
     return failed