Fix for get_mix_forecast ValueError: cannot convert float NaN to integer

[paulhomes]

When running emhass 0.13 with set_use_adjusted_pv: true I was finding that NaN values for PV after sunset were causing the following ValueError:

emhass-test  | [2025-04-03 17:30:33 +1000] [25] [ERROR] Exception on /action/naive-mpc-optim [POST]
emhass-test  | Traceback (most recent call last):
emhass-test  |   File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
emhass-test  |     response = self.full_dispatch_request()
emhass-test  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
emhass-test  |     rv = self.handle_user_exception(e)
emhass-test  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
emhass-test  |     rv = self.dispatch_request()
emhass-test  |          ^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
emhass-test  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
emhass-test  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/src/emhass/web_server.py", line 414, in action_call
emhass-test  |     input_data_dict = set_input_data_dict(
emhass-test  |                       ^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/src/emhass/command_line.py", line 351, in set_input_data_dict
emhass-test  |     P_PV_forecast = fcst.get_power_from_weather(
emhass-test  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/src/emhass/forecast.py", line 713, in get_power_from_weather
emhass-test  |     P_PV_forecast = Forecast.get_mix_forecast(
emhass-test  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
emhass-test  |   File "/app/src/emhass/forecast.py", line 614, in get_mix_forecast
emhass-test  |     df_forecast.iloc[0] = int(round(first_fcst))
emhass-test  |                               ^^^^^^^^^^^^^^^^^
emhass-test  | ValueError: cannot convert float NaN to integer

The line numbers above are out of sync with respect to version 0.13 forecast.py as this was a custom docker image with other changes (those from #499) in order to get set_use_adjusted_pv working in my install.

This change replaces NaN with zero which I assume is appropriate in this instance?

[davidusb-geek]

Hi Paul, thanks for the fix. However I think this is not needed, you just need to add your sensor name to the sensor_replace_zero list. There is already a method to do this and it is actually used when retrieving data for the PV adjust method

[paulhomes]

Hi David, I just checked and I already have the PV sensor listed under sensor_replace_zero but it is still getting a NaN value at that point in the code. I only just started getting these errors with version 0.13 so thought it might be the PV adjustment code that might be generating the NaNs. I'll have another look.

[paulhomes]

Hi David,

I did some debugging/logging today and think I have found the source of the issue. The fix I proposed in this PR resolved the issue for me, but I think there is an underlying issue to be resolved. I saw that, as you mentioned, the sensor_replace_zero config option should have prevented this and I did have my PV sensor listed there. I traced the data and saw that the retrieve_hass.py prepare_data method looks like it is supposed to do this via the var_replace_zero list. I added some extra logging and saw it remove my PV sensor from the var_replace_zero list:

self.var_list=['sensor.load_power_less_deferrable', 'sensor.total_dc_power', 'sensor.test_p_pv_forecast']
var_load=sensor.load_power_less_deferrable
load_negative=False
set_zero_min=True
var_replace_zero=['sensor.total_dc_power', 'sensor.test_p_pv_forecast']
var_interp=['sensor.total_dc_power', 'sensor.load_power_less_deferrable']
var_replace_zero=[]
var_interp=[]
new_var_replace_zero=[]
new_var_interp=[]

It seems to be due to code added recently (which explains why it worked for me in 0.12.8). If I read it correctly, it appears to empty var_replace_zero and var_interp unless they contain all of the sensors in self.var_list. In my case, and I'm still an emhass newbie so it may be incorrect, I have different sensors in those 2 lists. When I edited config.json to change that section from this:

    "sensor_power_load_no_var_loads": "sensor.load_power_less_deferrable",
    "sensor_replace_zero": [
        "sensor.total_dc_power",
        "sensor.test_p_pv_forecast"
    ],
    "sensor_linear_interp": [
        "sensor.total_dc_power",
        "sensor.load_power_less_deferrable"
    ],

... to this (so the lists were the same):

    "sensor_power_load_no_var_loads": "sensor.load_power_less_deferrable",
    "sensor_replace_zero": [
        "sensor.load_power_less_deferrable",
        "sensor.test_p_pv_forecast",
        "sensor.total_dc_power"
    ],
    "sensor_linear_interp": [
        "sensor.load_power_less_deferrable",
        "sensor.test_p_pv_forecast",
        "sensor.total_dc_power"
    ],

... the debug logging shows the lists were retained:

self.var_list=['sensor.load_power_less_deferrable', 'sensor.total_dc_power', 'sensor.test_p_pv_forecast']
var_load=sensor.load_power_less_deferrable
load_negative=False
set_zero_min=True
var_replace_zero=['sensor.load_power_less_deferrable', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']
var_interp=['sensor.load_power_less_deferrable', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']
var_replace_zero=['sensor.load_power_less_deferrable', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']
var_interp=['sensor.load_power_less_deferrable', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']
new_var_replace_zero=['sensor.load_power_less_deferrable_positive', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']
new_var_interp=['sensor.load_power_less_deferrable_positive', 'sensor.test_p_pv_forecast', 'sensor.total_dc_power']

It then replaced the missing PV values with zeros, as expected, and continued on successfully.

I'm wondering if that recent change in retrieve_hass.py prepare_data method is incorrect and it should only be removing individual sensors that are not in the var_list? If so then, as you said, this PR becomes unnecessary when that is fixed.

[paulhomes]

Actually I wonder if it was my addition of the "sensor.test_p_pv_forecast" into my config.json (as part of using the new PV adjustment in 0.13) that triggered this exception (and highlighted this issue). The recent code changes to the retrieve_hass.py prepare_data method would have already been in in 0.12.8.

[paulhomes]

Do you think that instead of:

        if isinstance(var_replace_zero, list) and all(
            item in var_replace_zero for item in self.var_list
        ):
            pass
        else:
            var_replace_zero = []

... it should be (switching the order of the 2 lists):

        if isinstance(var_replace_zero, list) and all(
            item in self.var_list for item in var_replace_zero
        ):
            pass
        else:
            var_replace_zero = []

Or better still, only remove those sensors from var_replace_zero that are not present in self.var_list (and log a warning) rather than empty the entire list?

The same goes for the code that processes var_interp too I think.

[davidusb-geek]

Hi Paul thanks for looking deeply into this. In my case I need to deep dive to find the root cause of the issue.
If you can write a small unit test in test_retrieve_hass.py that reproduce this issue it would a lot more easy for me to finally fix and even avoid this type of problem in the future by keeping that unit test. You could follow the many other test in that file as a template to build this new test

[paulhomes]

Hi David, I'll certainly try add a unit test. I had some knowledge issues previously in trying to get the unit tests running (I run everything in docker) but I'll have another try with this as it would be good to be able to run the unit tests.

[davidusb-geek]

Or better still, only remove those sensors from var_replace_zero that are not present in self.var_list (and log a warning) rather than empty the entire list?

So do you think that this should do the trick?

if isinstance(var_replace_zero, list):
    original_list = var_replace_zero[:]
    var_replace_zero = [item for item in var_replace_zero if item in self.var_list]
    removed = set(original_list) - set(var_replace_zero)
    for item in removed:
        self.logger.warning(f"Sensor '{item}' in var_replace_zero not found in self.var_list and has been removed.")
else:
    var_replace_zero = []

[paulhomes]

Hi David, That replacement code worked nicely. Thanks. I've added a related unit test as suggested and reverted the original redundant change from earlier. Something you may notice in the test, that tripped me up initially, is that I had to copy the var_list from the test into the RetrieveHass instance. It looks that doesn't get populated when the data is loaded from the file data/test_df_final.pkl in setup.

[paulhomes]

Now I see other tests are failing and, looking at the warnings, I wonder if it is the same issue with the var_list. I'll look into it some more.

[paulhomes]

It looks like the RetrieveHass prepare_data method relies on the RetrieveHass instances self.var_list being populated and it is not always being populated when the tests data are setup (via self.rh.var_list). Now that the prepare_data method var_replace_zero and var_interp lists are being validated against var_list, and their contents pruned based on that sometimes empty var_list, it is causing test failures. I could ignore var_list validation if it is empty when prepare_data runs, but it seems better to ensure it is populated as required by the callers.

I have run out of time today so will try and have another look tomorrow.

By the way, I am not running the unit tests against my actual HA instance yet (until I learn enough about the tests, I want to avoid them potentially making/publishing changes). My thinking being that they will run with data files and mocks like I imagine they do when running in github based on PR commit triggers. Is this a valid way to be running the tests?

[davidusb-geek]

Thanks for looking into this.
By default the unit tests are launched with a debug=True flag everywhere and a retrieve_data_from_file=True option, so that we don't need to retrieve any actual data from HA and by default the test won't publish anything neither. All requests are mocked. And you are right, this is how they are launched on the automated CI tests on github

[sonarqubecloud]

Quality Gate failed

Failed conditions
3.3% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

[paulhomes]

Thanks David, that's good to hear. I initially ran the tests in a VM with no network access to see what failed, and opensnitch to see what connections were attempted. I noticed a connection to open-meteo which was expected and a connection to myhass.duckdns.org which I redirected to localhost so it would fail fast.

I ran through all the tests looking for where it was complaining about missing columns in the empty var_list and found all the instances (I think) where data was loaded from a pickle file and added lines to copy the loaded var_list into the rh.var_list and now all the tests are passing here. However, for some reason it is failing on github for macos-latest. I don't yet know why it is failing for macos but not ubuntu or windows.

There is also a SonarCloud failure related to code duplication in tests/test_forecast.py - not sure what to do about that as it looks necessary.

I used docs/develop.md to setup my testing environment and have some feedback/suggestions:

In Step 2 - Develop / Method 1 - Python Virtual Environment it says to use pip with requirements.txt but there is no requirements.txt in the main directory - only one in the scripts dir. It looks like the runtime dependencies are
now handled in pyproject.toml so should that requirements.txt step be removed as the python3 -m pip install -e . step handles the requirements?

When I initially tried to run the unit tests I got ModuleNotFoundError: No module named 'requests_mock' and then saw the test dependencies are also in the pyproject.toml so was able to resolve it by running:

python3 -m pip install -e '.[test]'

Is this the correct way to setup Python Virtual Environment for testing now - should that step (or whatever else is best) be added to the doc? I can submit a PR if that helps.

[davidusb-geek]

When I initially tried to run the unit tests I got ModuleNotFoundError: No module named 'requests_mock' and then saw the test dependencies are also in the pyproject.toml so was able to resolve it by running:

python3 -m pip install -e '.[test]'

Is this the correct way to setup Python Virtual Environment for testing now - should that step (or whatever else is best) be added to the doc? I can submit a PR if that helps.

Hi Paul, yes this is the correct way using the python3 -m pip install -e .[test] command.
The documentation is just outdated as we have just recently this year dropped the requirements.txt.

[davidusb-geek]

I ran through all the tests looking for where it was complaining about missing columns in the empty var_list and found all the instances (I think) where data was loaded from a pickle file and added lines to copy the loaded var_list into the rh.var_list and now all the tests are passing here. However, for some reason it is failing on github for macos-latest. I don't yet know why it is failing for macos but not ubuntu or windows.

As you can see the test is now passing for MacOs. From time to time some tests will fail annoyingly like this. A manual relaunch of the failed job typically fix it.

The solarcloud fail can be ignored sometimes, depending on the type of failed check.
In this case it can be ignored I think, is not a major security fail

[davidusb-geek]

So with this this PR should be good to go right?

[paulhomes]

I think it is good to go. All of the unit tests are passing and I am now running it in my main emhass docker instance using the day-ahead and mpc optimizations with pv adjust. If there are any remaining issues it is likely to be from any scenarios not covered by unit tests where data is loaded from pickle files and the prepare_data method is used.