weather_service issues when using opne-meteo as API provider

[toffee]

Today I've looked why I get errors in weather_service when using open-meteo and found out the following issues (don't think are wrong configuration in my environment):

1. the open-meteo forecast url request only hourly data but later in the code there is a check for 15 min data presence. I solved the error by removing or "minutely_15" not in data
2. Not sure if the code is corect. Solved by replacing the line with if field in ("index", "validFrom"):
3. In db you explicitly expect all values to be NOT NULL but seems (at least in open-meteo case) not all values are provided because I get the following error:

May 03 14:10:49 jupiter weather_service[12618]: [INFO] - [lib.provider.provider:116] - Forecast data published • Total: 6048
May 03 14:10:49 jupiter weather_service[12618]: [INFO] - [lib.provider.provider:125] - Current data published • Total: 9
May 03 14:10:49 jupiter weather_service[12618]: [ERROR] - [lib.consumer.provider:110] - ["`airTemperatureInCelsius`='16.2'", "`feelsLikeTemperatureInCelsius`='15.2'", "`relativeHumidityInPercent`='57'", "`windDirectionInDegree`='73'", "`windSpeedInKilometerPerHour`='3.0'", "`windGustInKilometerPerHour`='18.7'", "`cloudCoverInOcta`='8.0'", "`thunderstormProbabilityInPercent`='1'", "`freezingRainProbabilityInPercent`='0'"]
May 03 14:10:49 jupiter weather_service[12618]: [ERROR] - [lib.consumer.provider:138] - Topic: ['jupiter', 'weather', 'provider', 'forecast', 'refreshed']
May 03 14:10:49 jupiter weather_service[12618]: [ERROR] - [lib.consumer.provider:139] - <class 'MySQLdb.IntegrityError'>: (1364, "Field 'hailProbabilityInPercent' doesn't have a default value")
May 03 14:10:49 jupiter weather_service[12618]: Traceback (most recent call last):
May 03 14:10:49 jupiter weather_service[12618]:   File "/opt/weather_service/lib/consumer/provider.py", line 111, in on_message
May 03 14:10:49 jupiter weather_service[12618]:     raise e
May 03 14:10:49 jupiter weather_service[12618]:   File "/opt/weather_service/lib/consumer/provider.py", line 99, in on_message
May 03 14:10:49 jupiter weather_service[12618]:     isModified = db.insertOrUpdate(validFrom.timestamp(),update_values)
May 03 14:10:49 jupiter weather_service[12618]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 03 14:10:49 jupiter weather_service[12618]:   File "/opt/weather_service/lib/db.py", line 69, in insertOrUpdate
May 03 14:10:49 jupiter weather_service[12618]:     self.cursor.execute(u"INSERT INTO {} SET {} ON DUPLICATE KEY UPDATE {}".format(self.config.db_table,",".join(insert_values),",".join(fields)))
May 03 14:10:49 jupiter weather_service[12618]:   File "/usr/lib/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
May 03 14:10:49 jupiter weather_service[12618]:     res = self._query(mogrified_query)
May 03 14:10:49 jupiter weather_service[12618]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 03 14:10:49 jupiter weather_service[12618]:   File "/usr/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
May 03 14:10:49 jupiter weather_service[12618]:     db.query(q)
May 03 14:10:49 jupiter weather_service[12618]:   File "/usr/lib/python3.12/site-packages/MySQLdb/connections.py", line 257, in query
May 03 14:10:49 jupiter weather_service[12618]:     _mysql.connection.query(self, query)
May 03 14:10:49 jupiter weather_service[12618]: MySQLdb.IntegrityError: (1364, "Field 'hailProbabilityInPercent' doesn't have a default value")

Solutions can be:

• provide default values MODIFY 'hailProbabilityInPercent' INT DEFAULT 0
• accept NULL MODIFY 'hailProbabilityInPercent' INT DEFAULT NULL

[toffee]

The third issue seems resolved after 1 h (when next fetch was performed):

May 03 15:05:00 jupiter weather_service[12618]: [INFO] - [lib.provider.provider:116] - Forecast data published • Total: 6048
May 03 15:05:00 jupiter weather_service[12618]: [INFO] - [lib.provider.provider:125] - Current data published • Total: 9
May 03 15:05:00 jupiter weather_service[12618]: [INFO] - [lib.consumer.provider:113] - Forecasts processed • Total 6048 • Queries: 336 • Updated: 94 • Inserted: 82