Load Remaining Today Does Not Match Planned Values

[nbullus]

Describe the bug

The predbat.load_energy_predicted and predbat.load_energy_adjusted sensors both have a today_remaining attribute that calculates the remaining load (up untill midnight).

If the in-day adjustments is turned 'on', then the remaining adjusted load will differ to the remaining predicted load. If in-day adjustments is turned 'off', both sensors track the same remaining load as you would expect.

However, the total accumulated half-hourly load values reported in the plan differs to what the aforementioned sensors state is the remaining load for today.

Note: This has been working previously after a fix due to a previous reported issue on the chat. Looks like the same issue has resurfaced but I have no further details on what was fixed.

Expected behaviour

The today_remaining attribute should match the load that the plan is currently working too.

Predbat version

8.48.0 (Beta)

Environment details

• GE AIO
• Standard HAOS installer

Screenshots

In-day adjustment is currently turned off and the today_remaining attribute is set to 5.23 kWh on both sensors.

However, the total planned remaining load is 5.52 kWh which is currently 0.29 kWh out.

Log file

predbat-5.log

predbat_plan-4.html

apps.yaml-3.txt

Predbat debug yaml file

predbat_debug.yaml-4.txt

[chalfontchubby]

Are you able to identify a previous version that worked please.

[nbullus]

Are you able to identify a previous version that worked please.

Hi Rik. I can't I'm afraid as it was well over 12 months ago. I'm pretty sure it got raised on the chat or on the FaceBook group post rather than a ticket. From memory I think the load from the last 30 min slot had been missed off. I'm sure if you searched the old release notes you may be able to find when it was raised and trace it from there. Failing that may just to need to re-investigate from fresh. Sorry I'm mot able to pin things down further. Nigel.

[gcoan]

@chalfontchubby another set of data points from my own log shows there is something not quite right in these sensors

Plan shows 15.59kWh of load remaining today (and I did add up the load column to confirm this is correct, its not a plan render issue)

Load energy predicted is 14.25kWh, energy used today is 0.27kWh and remaining is 13.99kWh

Load energy predicted adjusted is 17.44kh, used today is 3.46kWh and remaining today the same 13.99kWh

I am using load ML and looking in the logfile shows load energy used today is 3.6kWh and forecast at midnight is 17.7kWh

The same logfile shows today's actual load at 3.37kWh

None of these figures matches !

But what I also notice is that there is a logfile message about days_previous_auto is enabled, Since I am using loadML, days_previous shouldn't be used at all so I wonder if this is the cause of the issue (or just a logging error), is predbat double applying in-day load adjustment?

predbat.log

[nbullus]

@chalfontchubby another set of data points from my own log shows there is something not quite right in these sensors

Plan shows 15.59kWh of load remaining today (and I did add up the load column to confirm this is correct, its not a plan render issue)

Load energy predicted is 14.25kWh, energy used today is 0.27kWh and remaining is 13.99kWh

Load energy predicted adjusted is 17.44kh, used today is 3.46kWh and remaining today the same 13.99kWh

I am using load ML and looking in the logfile shows load energy used today is 3.6kWh and forecast at midnight is 17.7kWh

The same logfile shows today's actual load at 3.37kWh

None of these figures matches !

But what I also notice is that there is a logfile message about days_previous_auto is enabled, Since I am using loadML, days_previous shouldn't be used at all so I wonder if this is the cause of the issue (or just a logging error), is predbat double applying in-day load adjustment?

predbat.log

Yes my 'today so far' attribute is also out. Just out of interest, are you running the latest 8.48.0 (Beta) version as this includes a key fix correcting the predicated/actual data points. My version for some reason reverted back to 8.47.7 overnight and values are saw toothing up and down again. I've put it back to the 8.48.0 (Beta) version again but the figures are still wrong as you point out. Also I turned off in-day adjustment temporarily for a whole day and the figures are still wrong so double applying in-day adjustments may not be the issue. Even at midnight after it first calculated the plan it did not match any of the values.

[nbullus]

@chalfontchubby another set of data points from my own log shows there is something not quite right in these sensors
Plan shows 15.59kWh of load remaining today (and I did add up the load column to confirm this is correct, its not a plan render issue)
Load energy predicted is 14.25kWh, energy used today is 0.27kWh and remaining is 13.99kWh
Load energy predicted adjusted is 17.44kh, used today is 3.46kWh and remaining today the same 13.99kWh
I am using load ML and looking in the logfile shows load energy used today is 3.6kWh and forecast at midnight is 17.7kWh
The same logfile shows today's actual load at 3.37kWh
None of these figures matches !
But what I also notice is that there is a logfile message about days_previous_auto is enabled, Since I am using loadML, days_previous shouldn't be used at all so I wonder if this is the cause of the issue (or just a logging error), is predbat double applying in-day load adjustment?
predbat.log

Yes my 'today so far' attribute is also out. Just out of interest, are you running the latest 8.48.0 (Beta) version as this includes a key fix correcting the predicated/actual data points. My version for some reason reverted back to 8.47.7 overnight and values are saw toothing up and down again. I've put it back to the 8.48.0 (Beta) version again but the figures are still wrong as you point out. Also I turned in-day adjustment 'off' temporarily for a whole day and the figures were still wrong, so double applying in-day adjustments may not be the issue. Even at midnight after it first calculated the plan it did not match any of the values.

[chalfontchubby]

Found and fixed one confirmed root cause: PR #4506.

load_today_comparison() (the code behind today_remaining on load_energy_predicted/load_energy_adjusted) never applied load_scaling to its predicted curve, while the plan's own remaining-load total does. I replayed @nbullus's attached debug.yaml directly and it matches exactly - their load_scaling is 1.05, the plan showed 5.531kWh remaining, the sensor showed 5.23kWh (1.0575 ≈ that scaling factor). Fixed, residual gap after the fix is ~0.7%.

@gcoan - your numbers further up (14.25 / 17.44 / 15.59 / 17.7 / 3.37-3.6, none agreeing, with Load ML enabled and the days_previous_auto question) look like they could be a different or additional issue - I wasn't able to verify that one since there's no debug.yaml attached to your report, only a log link. Once PR #4506 lands, could you check whether it resolves things for you too? If the mismatch is still there afterwards, could you open a separate issue with a debug.yaml attached (same as this one) so it can be replayed the same way? Easier to track down as its own thing than as a second bug buried in this thread.

[gcoan]

I was running an older version 8.46.11
Here's my debug.yaml, my load scaling is also 1.05 although there is a power up session for 2 hours today from 12-2 with a load scaling override of 1.5 for that period

predbat_debug.yaml-6.txt

I've now upgraded to the latest 8.48.0 (beta)

predbat_debug.yaml-7.txt

predbat.log

plan shows 11.81kWh remaining

load energy predicted 10.48

load energy predicted adjusted also 10.48 remaining, 15.18 at midnight

10.48 * 1.05 =11.004 but as I said there is a load scaling of 1.5 for the 2 hour power up, so that might be the discrepancy

log file shows loadML predicting 15.4kWh at midnight

[nbullus]

Found and fixed one confirmed root cause: PR #4506.

load_today_comparison() (the code behind today_remaining on load_energy_predicted/load_energy_adjusted) never applied load_scaling to its predicted curve, while the plan's own remaining-load total does. I replayed @nbullus's attached debug.yaml directly and it matches exactly - their load_scaling is 1.05, the plan showed 5.531kWh remaining, the sensor showed 5.23kWh (1.0575 ≈ that scaling factor). Fixed, residual gap after the fix is ~0.7%.

@gcoan - your numbers further up (14.25 / 17.44 / 15.59 / 17.7 / 3.37-3.6, none agreeing, with Load ML enabled and the days_previous_auto question) look like they could be a different or additional issue - I wasn't able to verify that one since there's no debug.yaml attached to your report, only a log link. Once PR #4506 lands, could you check whether it resolves things for you too? If the mismatch is still there afterwards, could you open a separate issue with a debug.yaml attached (same as this one) so it can be replayed the same way? Easier to track down as its own thing than as a second bug buried in this thread.

Thanks Rik.

I take it the missing load scaling will get applied to 'today_remaining' on both sensors - predicted and adjusted?

Nigel.

[chalfontchubby]

I take it the missing load scaling will get applied to 'today_remaining' on both sensors - predicted and adjusted?

Yes - the fix is in the shared calculation both sensors are built from (`load_value_pred`/`load_value_pred_raw` in `load_today_comparison()`), not a per-sensor patch, so both `load_energy_predicted` and `load_energy_adjusted`'s `today_remaining` get it. Added a regression test that checks both independently to be sure (scales a baseline by 1.5x load_scaling and confirms each sensor's today_remaining moves by the same factor) - both pass.

#4506 is merged now, so this should be live for you once you update.

[gcoan]

#4506 is merged now, so this should be live for you once you update

Hi Rik, I've just taken a look at the code fix, and its unfortunately incomplete

it looks from the code that you have applied load_scaling (default 1.05) to the calculated load

but there are several different load scaling factors in predbat, the base one, one for saving sessions and one for free electricity events

and you can also override the load prediction for each individual slot, either via apps.yaml rates or using the predbat manual api as I am doing here for a power up event

I don't know how predbat holds this internally but I assume it has a load scaling per slot which is then applied to the loadML/days_previous load prediction as part of creating the plan