perf(plan): 21% faster planning, and cars in the random benchmark (#4529)

* perf(kernel): feed the limit arrays straight to array.array (-3% planning time)

double_array was handed [float(limit) for limit in limits] on every simulation. array.array already
coerces each item to a double in C, so both the float() call and the intermediate list were wasted
work: 0.89us -> 0.53us per call, and it stops allocating a throwaway float object per limit.

Measured on the 20-scenario random benchmark: 51.01s -> 49.45s and 49.47s over two runs, with every
scenario byte-identical on metric, cost, cost_pv10, cost_pv90, soc_min, soc_final, battery_cycles
and export_kwh. kernel_parity and model_kernel pass.

Two things deliberately NOT changed, having measured them and found them slower:

  - the window fields keep their list comprehensions. map(operator.itemgetter("start"), windows) is
    0.88us -> 1.05us at these list lengths, the per-item call overhead outweighing the comprehension.
  - predict_soc keeps its indexed loop. dict(zip(range(...), soc_out)) is 13.95us -> 18.69us,
    because iterating a ctypes array boxes each double through the sequence protocol.

Both are noted in comments so the next person does not repeat the experiment. line_profiler had
suggested these lines were ~29% and ~13% of the wrapper; its per-line overhead badly inflates
lines executed 73k times, and a microbenchmark tells a different story.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(plan): memoise the charge selection in the levels pass (-11% planning time)

The price-threshold scan in optimise_charge_limit_price_threads sits four loops deep:

    for max_charge_slots / for max_export_slots / for try_charge_freeze / for try_export_freeze

but which charge windows the threshold selects depends only on loop_price, max_charge_slots and
try_charge_freeze - not on either export loop it is nested inside. It was therefore rebuilt
identically once per (max_export_slots, try_export_freeze) pair. Line profiling put it at ~52M
iterations on the heaviest benchmark scenario, the single hottest loop in planning.

Memoised on (max_charge_slots, try_charge_freeze) rather than hoisted, so the iteration order - and
with it every tie-break in the search below - is untouched. Sharing the cached objects between
iterations is safe because neither is mutated after it is built: all_n is copied into pred_item and
charge_mods is only ever read.

The freeze skip also folds into the selection condition, dropping a bare `pass` branch that was
executing 19M times. The cheap freeze test still comes first so the dict lookup is still
short-circuited for the ~37% of iterations that take it.

Measured on the 20-scenario random benchmark: 49.45s -> 44.08s and 43.69s over two runs, every
scenario byte-identical on metric, cost, cost_pv10, cost_pv90, soc_min, soc_final, battery_cycles,
export_kwh and import_kwh_battery.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(plan): stop hit_car_window rounding every slot, and give it a cache

hit_car_window computed dp2(window["kwh"]) for every car charging slot before the intersection test
that discards nearly all of them. For anyone with an EV that made it the most expensive function in
planning: on a car scenario it was 5.04s cumulative of an 11s plan, driving 15.9M dp2 and 16.1M round
calls, while the C++ kernel took 1.4s.

Testing the intersection first and only rounding a slot that actually overlaps is 6.45us -> 1.08us
per call at 48 slots. dp2 is pure, so moving it last cannot change the answer.

It is still asked the same question about the same few windows ~1.2M times per plan, 99.8% of them
from optimise_charge_limit_price_threads, so that caller now keeps a (start, end) -> hit dict for the
length of the pass and hands it in - the same shape as the hit_charge_cache already beside it. The
cache is caller-owned rather than held on self so its lifetime belongs to whoever knows when
car_charging_slots can change.

The random benchmark could not see any of this because it had no cars, so it now generates them:
roughly half the scenarios get 1-2 cars with 4-20 charging slots. The car block is drawn from a
separate RNG stream seeded off the scenario seed, so re-generating reproduces every pre-existing
parameter and data profile bit-identically and old baselines stay comparable - verified. Scenarios
written before this carry no "cars" entry and still run car-less.

apply_scenario_to_predbat now sizes every per-car attribute predbat indexes by car_n, not just the
ones a scenario varies: set_rate_thresholds takes max(car_charging_plan_max_price[:num_cars]) and
raises on an empty slice.

Measured on the 20-scenario benchmark with cars: 53.66s -> 43.50s for the reorder, -> 42.23s with the
cache, every scenario byte-identical on metric, cost, cost_pv10, cost_pv90, soc_min, soc_final,
battery_cycles, export_kwh and import_kwh_battery at each step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(random): rebase the checked-in benchmark baseline onto the car scenarios

random_scenarios.yaml now generates cars for roughly half the scenarios, so the stored baseline -
produced against the car-less set - no longer describes the same workload. Comparing a future run
against it would have shown a large spurious diff on all 11 car scenarios.

Regenerated against the same template the previous baseline used (cases/predbat_debug_agile1.yaml),
with the hit_car_window work in place. 20 scenarios, none failed, all carrying cost_pv10/cost_pv90.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(random): write the results JSON with a trailing newline

json.dump leaves the file without one, so the end-of-file hook rewrote the checked-in baseline every
time it was regenerated - which is what produced the stray pre-commit.ci fixup commit on the earlier
branch. Emit it at the writer so the file lands clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Unit test settings

* Test updates

* Test updates

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: 3 people

apps/predbat/plan.py

@@ -406,6 +406,7 @@ def optimise_charge_limit_price_threads(
         # for 221 distinct answers. Only the collision itself is cached - the limit that follows from it
         # depends on charge_mods/best_limits_reset and changes from trial to trial.
         hit_charge_cache = {}
+        hit_car_cache = {}  # (start, end) -> does this window hit a car charging slot
 
         # Start loop of trials
         for loop_price in all_prices:
@@ -431,33 +432,47 @@ def optimise_charge_limit_price_threads(
                 charge_freeze_options = [True, False] if (self.set_charge_freeze and not coarse) else [False]
                 export_freeze_options = [True, False] if (self.set_export_freeze and not coarse) else [False]
 
+                # Which charge windows the price threshold selects depends only on loop_price (fixed here),
+                # max_charge_slots and try_charge_freeze - not on either of the export loops it is nested
+                # inside, so without this it is rebuilt identically once per (max_export_slots,
+                # try_export_freeze) pair. That scan was the single hottest loop in planning, ~52M
+                # iterations on a heavy scenario. Memoised rather than hoisted so the iteration order, and
+                # with it every tie-break in the search below, is untouched.
+                #
+                # Sharing the cached objects between iterations is safe because neither is mutated after it
+                # is built: all_n is copied into pred_item, and charge_mods is only ever read.
+                charge_selection_cache = {}
+
                 for max_charge_slots in charge_slot_choices:
                     for max_export_slots in export_slot_choices:
                         for try_charge_freeze in charge_freeze_options:
                             for try_export_freeze in export_freeze_options:
-                                all_n = []
                                 all_d = []
-                                count_c = 0
                                 count_d = 0
-                                charge_mods = {}  # window_n -> freeze flag, for charge windows modified from the reset limits
                                 export_mods = {}  # window_n -> freeze flag, for export windows modified from the reset limits
 
-                                for price, window_n, freeze in price_set_charge:
-                                    if loop_price >= price:
-                                        if freeze and not try_charge_freeze:
-                                            pass
-                                        elif count_c < max_charge_slots and (window_n not in charge_mods):
-                                            all_n.append(window_n)
-                                            charge_mods[window_n] = freeze
-                                            count_c += 1
+                                charge_selection = charge_selection_cache.get((max_charge_slots, try_charge_freeze))
+                                if charge_selection is None:
+                                    all_n = []
+                                    count_c = 0
+                                    charge_mods = {}  # window_n -> freeze flag, for charge windows modified from the reset limits
+                                    for price, window_n, freeze in price_set_charge:
+                                        if loop_price >= price:
+                                            if not (freeze and not try_charge_freeze) and count_c < max_charge_slots and (window_n not in charge_mods):
+                                                all_n.append(window_n)
+                                                charge_mods[window_n] = freeze
+                                                count_c += 1
+                                    charge_selection_cache[(max_charge_slots, try_charge_freeze)] = (all_n, charge_mods)
+                                else:
+                                    all_n, charge_mods = charge_selection
 
                                 for price, window_n, freeze in price_set_export:
                                     if loop_price < price:
                                         # For prices above threshold try export
                                         if freeze and not try_export_freeze:
                                             pass
                                         elif count_d < max_export_slots and (window_n not in export_mods):
-                                            if not self.car_charging_from_battery and self.hit_car_window(export_window[window_n]["start"], export_window[window_n]["end"]):
+                                            if not self.car_charging_from_battery and self.hit_car_window(export_window[window_n]["start"], export_window[window_n]["end"], cache=hit_car_cache):
                                                 pass
                                             elif not self.iboost_on_export and self.iboost_enable and self.iboost_plan and (self.hit_charge_window(self.iboost_plan, export_window[window_n]["start"], export_window[window_n]["end"]) >= 0):
                                                 pass
@@ -3175,16 +3190,21 @@ def optimise_swap_export(self, record_charge_windows, record_export_windows, dro
         """
         swapped_target = {}
         curr = self.currency_symbols[1]
+        first = True
 
         if self.calculate_best_export and record_export_windows >= 2:
             swapped = True
             while swapped:
                 selected_metric, selected_battery_value, selected_cost, selected_keep, selected_cycle, selected_carbon, selected_import, select_export = self.run_prediction_metric(
                     self.charge_limit_best, self.charge_window_best, self.export_window_best, self.export_limits_best, end_record=self.end_record
                 )
-                self.log(
-                    "Swap export optimisation started metric {}{}, cost {}{}, battery_value {}kWh, min_improvement_swap {}{}".format(dp2(selected_metric), curr, dp2(selected_cost), curr, dp2(selected_battery_value), self.metric_min_improvement_swap, curr)
-                )
+                if first:
+                    self.log(
+                        "Swap export optimisation started metric {}{}, cost {}{}, battery_value {}kWh, min_improvement_swap {}{}".format(
+                            dp2(selected_metric), curr, dp2(selected_cost), curr, dp2(selected_battery_value), self.metric_min_improvement_swap, curr
+                        )
+                    )
+                first = False
                 swapped = False
 
                 for window_n_target in range(record_export_windows - 1, 0, -1):
@@ -3423,12 +3443,15 @@ def optimise_swap_charge(self, record_charge_windows, debug_mode=False):
 
         swapped_target = {}
         swapped = True
+        first = True
         while swapped:
             selected_metric, selected_battery_value, selected_cost, selected_keep, selected_cycle, selected_carbon, selected_import, selected_export = self.run_prediction_metric(
                 self.charge_limit_best, self.charge_window_best, self.export_window_best, self.export_limits_best, end_record=self.end_record
             )
-            self.log("Swap charge optimisation started metric {}{}, cost {}{}, min_improvement_swap {}{}".format(dp2(selected_metric), curr, dp2(selected_cost), curr, min_improvement_swap, curr))
+            if first:
+                self.log("Swap charge optimisation started metric {}{}, cost {}{}, min_improvement_swap {}{}".format(dp2(selected_metric), curr, dp2(selected_cost), curr, min_improvement_swap, curr))
             swapped = False
+            first = False
 
             for window_n_target in range(record_charge_windows - 1, 0, -1):
                 window_start_target = self.charge_window_best[window_n_target]["start"]
@@ -5141,16 +5164,38 @@ def car_charge_slot_kwh(self, minute_start, minute_end):
             car_charging_kwh = dp2(car_charging_kwh)
         return car_charging_kwh
 
-    def hit_car_window(self, window_start, window_end):
-        """
-        Does this window intersect a car charging window?
+    def hit_car_window(self, window_start, window_end, cache=None):
+        """Does this window intersect a car charging window?
+
+        cache, when given, is a caller-owned dict of (start, end) -> hit. The optimiser asks the same
+        question about the same handful of windows millions of times per plan, so the caller keeps a dict
+        for as long as car_charging_slots cannot change underneath it and the scan collapses to a lookup.
+        Deliberately not held on self: the lifetime then belongs to whoever knows when the slots change.
+
+        The slot scan tests the intersection before dp2(): rounding every slot's kwh up front made this the
+        most expensive function in planning for anyone with an EV, since the rounding was being done for
+        nearly every slot the overlap test then discarded (6.45us -> 1.08us per call at 48 slots). dp2 is
+        pure, so testing it last cannot change the answer.
         """
-        if self.num_cars > 0:
-            for car_n in range(self.num_cars):
-                for window in self.car_charging_slots[car_n]:
-                    start = window["start"]
-                    end = window["end"]
-                    kwh = dp2(window["kwh"])
-                    if end > window_start and start < window_end and kwh > 0:
-                        return True
-        return False
+        if self.num_cars <= 0:
+            # No car, no cache work - this is the common case and it has to stay a single test
+            return False
+
+        key = (window_start, window_end)
+        if cache is not None:
+            hit = cache.get(key)
+            if hit is not None:
+                return hit
+
+        hit = False
+        for car_n in range(self.num_cars):
+            for window in self.car_charging_slots[car_n]:
+                if window["end"] > window_start and window["start"] < window_end and dp2(window["kwh"]) > 0:
+                    hit = True
+                    break
+            if hit:
+                break
+
+        if cache is not None:
+            cache[key] = hit
+        return hit

apps/predbat/prediction_kernel.py

@@ -282,24 +282,30 @@ def select_array_typecode(candidates, ctype):
 
 
 def double_array(values):
-    """Create a ctypes double array from a Python list.
+    """Create a ctypes double array from any iterable of numbers.
 
     Built via array.array rather than (ctypes.c_double * n)(*values): the latter unpacks the list as
     positional arguments and is several times slower, which matters because these are rebuilt on
     every simulation. from_buffer returns a view over the array.array, and ctypes keeps the backing
     object alive through the view's _objects, so the buffer cannot be collected while the kernel is
     using it. Each pool worker is a separate process (multiprocessing with fork), so no buffer is
     ever shared between workers.
+
+    An iterable is taken rather than a list so callers can hand over a map/generator and skip
+    materialising an intermediate list - array.array consumes it in C and coerces each item to a double
+    itself, which is what the caller's float() was doing.
     """
     if DOUBLE_TYPECODE is None:
+        values = list(values)
         return (ctypes.c_double * len(values))(*values)
     backing = array.array(DOUBLE_TYPECODE, values)
     return (ctypes.c_double * len(backing)).from_buffer(backing)
 
 
 def int32_array(values):
-    """Create a ctypes int32 array from a Python list - see double_array for why array.array is used"""
+    """Create a ctypes int32 array from any iterable of ints - see double_array for why array.array is used"""
     if INT32_TYPECODE is None:
+        values = list(values)
         return (ctypes.c_int32 * len(values))(*values)
     backing = array.array(INT32_TYPECODE, values)
     return (ctypes.c_int32 * len(backing)).from_buffer(backing)
@@ -507,10 +513,14 @@ def run_prediction_kernel(pred, charge_limit, charge_window, export_window, expo
 
     n_steps = pred.forecast_minutes // PREDICT_STEP
     scenario = PkScenario()
-    scenario.charge_limit = double_array([float(limit) for limit in charge_limit])
+    # The limits are fed to array.array directly: it coerces each item to a double in C, so the float() and
+    # the intermediate list the comprehension built are both wasted work (measured 0.89 -> 0.53us per call).
+    # The window fields keep their list comprehensions - map(itemgetter(...)) measured *slower* here at these
+    # list lengths, the per-item call overhead outweighing what the comprehension costs.
+    scenario.charge_limit = double_array(charge_limit)
     scenario.charge_start = int32_array([window["start"] for window in charge_window])
     scenario.charge_end = int32_array([window["end"] for window in charge_window])
-    scenario.export_limits = double_array([float(limit) for limit in export_limits])
+    scenario.export_limits = double_array(export_limits)
     scenario.export_start = int32_array([window["start"] for window in export_window])
     scenario.export_end = int32_array([window["end"] for window in export_window])
     soc_out = (ctypes.c_double * n_steps)()
@@ -540,6 +550,8 @@ def run_prediction_kernel(pred, charge_limit, charge_window, export_window, expo
     # Assemble the same return value as the Python engine - prediction.py:626-628, 1266-1284
     predict_soc = {}
     if not cache:
+        # Indexed loop, not dict(zip(range(...), soc_out)): iterating a ctypes array boxes each double
+        # through the sequence protocol and measured 34% slower than subscripting it.
         for k in range(n_steps):
             predict_soc[k * PREDICT_STEP] = soc_out[k]
 

apps/predbat/tests/test_optimise_all_windows.py

@@ -140,26 +140,24 @@ def run_optimise_all_windows(
 
 
 def run_optimise_all_windows_kernel_tests(my_predbat):
-    """Run the optimise all windows tests with the Python engine and again with the C++ kernel, comparing runtime.
+    """Run the optimise all windows tests with the C++ kernel
 
     Both runs must pass their normal assertions; the kernel run dispatches every supported
     prediction to the C++ kernel. Returns True on failure.
     """
 
-    start = time.time()
-    failed = run_optimise_all_windows_tests(my_predbat)
-    python_time = time.time() - start
-    print("Optimise all windows tests (Python engine) took {} seconds".format(round(python_time, 2)))
-
     available, required_failure = kernel_available()
-    if not available:
-        return required_failure
-
-    start = time.time()
-    failed |= run_optimise_all_windows_tests(my_predbat, prediction_kernel=True)
-    kernel_time = time.time() - start
-    print("Optimise all windows tests (C++ kernel) took {} seconds".format(round(kernel_time, 2)))
-    print("C++ kernel speedup: {}x".format(round(python_time / kernel_time, 1)))
+    failed = False
+    if available:
+        start = time.time()
+        failed |= run_optimise_all_windows_tests(my_predbat, prediction_kernel=True)
+        kernel_time = time.time() - start
+        print("Optimise all windows tests (C++ kernel) took {} seconds".format(round(kernel_time, 2)))
+    else:
+        start = time.time()
+        failed |= run_optimise_all_windows_tests(my_predbat)
+        python_time = time.time() - start
+        print("Optimise all windows tests (Python engine) took {} seconds".format(round(python_time, 2)))
     return failed