Advanced options suggestions box

[autoSteve]

The newly minted "advanced options" feature carries with it a starting point for integration tweaks.

I'd like to hear your suggestions for more.

Some that I have already dreamed up recently added...

• Exclude as insignificant automated dampening factors that were < 0.95, but have been delta-logarithmically adjusted beyond 0.95 because of a lesser than ideal forecasted interval. Or set this as the default behaviour and have an option to not exclude as insignificant.
• Adjustable level for what is considered a "similar" past estimated actual for an interval time. This is a hard-coded 90% of the peak currently, and lowering this threshold would allow more generation values to be considered as being the peak on that front.
• The time to collect past estimated actuals, between midnight and sunrise, still randomised as within fifteen minutes of the specified time. I chose midnight-local-plus-up-to-15-mins as an arbitrary time of day to do this, but maybe this is a crazy time of Solcast API busyness in some parts of the world.

[autoSteve]

automated_dampening_insignificant_factor_adjusted added...

[autoSteve]

automated_dampening_similar_peak added...

[gcoan]

I have just read through the Advanced options https://github.com/BJReplay/ha-solcast-solar/blob/main/ADVOPTIONS.md and TBH, I couldn't see there was any of the options I wanted to set.

I could reduce forecast_day_entities as I only have ever looked at the forecast up to 5 days ahead, but muhuh, what's the point?

forecast_history_max_days might be nice to increase, but again, not sure there is real benefit to seeing more than 2 years in the energy dashboard.

automated_dampening_generation_history_load_days I did wonder about, I think this option only affects the initial load of generation history into auto-dampening, and thereafter as a-d runs it will build up and use up to automated_dampening_model_days (default 14) days of history?
If so, might be worth making this more explicit in the documentation.
As I think about it I will try increasing automated_dampening_model_days a bit to give it more rope to work with in gloomy periods.

So take this as positive confirmation of your design decisions embedded in the integration @autoSteve

Cheers

[autoSteve]

estimated_actuals_fetch_delay added...

[gcoan]

I’d like to add an extra option for delaying when the collection of actual PV generation occurs, adding an offset from the currently hard coded collection at midnight https://github.com/BJReplay/ha-solcast-solar?tab=readme-ov-file#key-input-actual-pv-generation-for-your-site

The estimated actuals would then be scheduled to be after the PV generation collection.

The reason for this is that I have a lot going on at midnight on my HA, my ‘inverter energy today’ statistics are all being reset to zero, predbat is doing a comparison of different energy tariff projections to work out which is most cost effective, and Solcast is doing the PV generation collection and then estimated actuals. You can actually hear the HA server running hard for about 10 minutes as the fan kicks in with the high CPU load !

But its not just my ears that I’m making this suggestion for, the inverter sensor midnight reset isn’t working properly, I’m seeing a noticeable delay in some of the sensors being reset and by reducing server load I hope to eliminate server load as being a cause of this:

Of course if this is impossibly difficult to do, I understand

[autoSteve]

Well planted, @gcoan.

I can't believe I missed it in the "suggestions box".

Implemented for v4.4.8.

[gcoan]

I try!

Put it in the right place even without having to read the readme. Whatever next ...

[saturnv5]

I'd love an option to apply the logarithmic/square-root adjustment to PV dampening for manually set dampening values (via the json file @ 30m intervals) - currently only available when automatic dampening is enabled.

I'll be switching energy plan soon, which will require solar curtailment throughout much of the day in summer, so auto dampening won't work well. I can easily manually set dampening factors, but I still want a non-linear adjustment to account for cloudy conditions.

Sorry if this is already an option somewhere and I just missed it in the documentation.

[BJReplay]

which will require solar curtailment throughout much of the day in summer, so auto dampening won't work well

@saturnv5 are you sure that solar curtailment and auto-dampening won't work well?

It sounds like you need https://github.com/BJReplay/ha-solcast-solar?tab=readme-ov-file#optional-input-site-export-to-the-grid-combined-with-a-limit-value

You can possibly use https://github.com/BJReplay/ha-solcast-solar?tab=readme-ov-file#modifying-automated-dampening-behaviour if you are choosing when to curtail: turn on the flag when curtailing. Both approaches give you automated dampening, curtailment, and your choice of algo.

Do let us know how you get on: Adding advanced options keeps @autoSteve out of trouble, but I suspect you can get what you want with the options that are already there.

[saturnv5]

@BJReplay I thought about using those options, but it doesn't quite fit what I want to accomplish. My new plan will offer zero feed-in tarriff from 8am to 5:30pm, so I'd prefer to just switch my system to self-consumption during the day (i.e. solar only powers the house + charging house battery + EV battery). There's no point working the inverter if I'm getting nothing for exporting.

However, I have lots of automations that rely on accurate hour-by-hour potential solar forecast - as if solar was not curtailed (e.g. when to charge the battery vs. the EV, how much to charge, when to heat water etc.). Using manual dampening vastly underestimates the potential solar production in cloudy conditions, which is why I'd like to still take advantage of the non-linear adjustment used with auto dampening.

[BJReplay]

There's no point working the inverter if I'm getting nothing for exporting.

I beg to differ: I would understand if you were being charged for exports (feed in) during those hours, or were not allowed to export, but every watt that you export is a watt not being generated by fossil fuel.

The reason that you get paid nothing for it is that the wholesale market value is zero (or negative), but that wholesale price is what drives

• retirement of fossil fuel from the grid
• investment in grid scale batteries
• investment in aggregated residential dispatch (e.g. paying you to charge your EV or household battery)
• and so on

In other words, it drives change.

And, separately, the inverter does not work harder when feeding in vs when matching load, but simply works differently.

But, in any case, autoSteve will or won't consider your request depending on his mood, and how convincing you are.

[autoSteve]

Give me something harder.

"granular_dampening_delta_adjustment".

Target v4.4.10

[saturnv5]

Thanks @autoSteve

@BJReplay I totally get what you mean. I might compromise and export in the mornings and afternoon periods, even when it's 0c feed-in. However, in the middle of the day, my inverter gets very loud and hot when the panels approach 20kW production, and it heats up my garage to over 40C.

[Nilogax]

Is there any use for the ancient data held in the likes of solcast.json? Might I suggest a max_solcast_history option to set a max number of days to keep. I'm certainly not bothered about what my system did in June last year!

If this doesn't float your boat @autoSteve I'm happy to have a crack at implementing it. The pytest? Not so much 😱

[autoSteve]

Ask, and you shall already have received, my dear @Nilogax.

forecast_history_max_days, which is settable minimum 22 days, up to max 3,560.

So I'm well ahead of you, but so slow to the pytest party on this one. Should work. Do lemme know if it results in a mushroom cloud... (Take a backup. 😅)

[Nilogax]

Doh! That's me and @BJReplay exposed as guilty of not reading the doco 😂

[autoSteve]

(Despite the name, this actually impacts all long-term held data. Dampened forecast, undampened forecast and dampened/undampened estimated actuals. Your name is better, so I wish you'd suggested it earlier.)

[autoSteve]

(Begins coding of AdvancedOption™️ aliases, along with deprecation warnings... 😂)

[BJReplay]

Doh! That's me and @BJReplay exposed as guilty of not reading the doco 😂

That's unfair - I have read it - very carefully (just ask @autoSteve about some of the things that I pick up on review), I just haven't necessarily remembered... 🤷

[Nilogax]

Do lemme know if it results in a mushroom cloud...

Crash test dummy reporting in. Custard and mushroom free system this morning, with all json files truncated to 60 days. Great success!

[BJReplay]

OK, so I'm not convinced that this is an advanced option suggestion box, or a service (action) suggestion, but I know you like a challenge...

My use case:

My algorithm to optimise discharge into Globird Zero Hero relies on:

1. Overnight consumption (overnight further defined below)

That's it.

But overnight is where it gets interesting - I'm not defining it as sunset to sunrise, but 6pm to the point at which solar power exceeds household load the next day. Once I know that, and I know current SOC, I can work out how much I can discharge, provided I know when I'll be able to rely on the sun again tomorrow. There's a smidge of forecast, but not much.

I'm using a trailing 14 days to determine that time - from that point on, I'm self-sufficient.

Now, if I ignore generation prior to that point, I'm ignoring a reasonable (at this time of year maybe 1 to 1.5kWh) amount of self-consumption - so up to 10% of available battery capacity.

I've got the "figure out the point at which I'm self-sufficient" working well (included below FYI): Find the average hour over the last 14 days where battery power is less than zero (it is charging) between midnight and 10 (I don't care if it is after 11am - I'm on the grid then anyway, baby).

SELECT	AVG(strftime("%H",BP.start_ts, "unixepoch", "localtime")) AS Hour
FROM	statistics as BP 
JOIN	statistics_meta as SM
	ON	BP.metadata_id = SM.id
	AND	SM.statistic_id = 'sensor.legacygateway_battery_power'
WHERE	date(BP.start_ts, "unixepoch", "localtime") BETWEEN date('now','-14 days') AND date('now')
AND	strftime("%H",BP.start_ts, "unixepoch", "localtime") BETWEEN "00" AND "10"
AND	min < 0

I've got the "overnight consumption until self-sufficient" working well (included below, too) - there's a helper input variable and an un-shown automation to set it - I was using a template that was directly referencing the self-sufficient point SQL Sensor from above, but it was unreliable across restarts - I think there was an initialisation order issue. Get the average of the sum of the mean consumption grouped by day for the last 14 days from 6pm to the break-even hour as determined in the previous step.

SELECT	AVG(Consumption) AS Consumption
FROM	(
	SELECT	SUM(mean) AS Consumption
	,	date(start_ts, "unixepoch", "localtime") AS Date
	FROM	statistics
	WHERE	metadata_id = ( SELECT id FROM statistics_meta where statistic_id = 'sensor.legacygateway_load_power' ) 
	AND	(strftime("%H",start_ts, "unixepoch", "localtime") BETWEEN "18" AND "23"
		OR strftime("%H",start_ts, "unixepoch", "localtime") BETWEEN "00" AND "{{ "%02d" | format(states('input_number.solar_excess_hour') | int ) }}")
	AND	date(start_ts, "unixepoch", "localtime") BETWEEN date('now','-14 days') AND date('now','-1 days')
	GROUP BY date(start_ts, "unixepoch", "localtime")
)

I can (and have been) periodically changing X in the Custom Next X hours Sensor for the Solcast integration - and then polling that at 6pm, and using remaining today and that sensor to give me the values I need to work out the next morning's generation (which is being used as self-consumption up to the excess generation point - but can be used - provided I don't flatten the battery - as part of my discharge planning).

Your challenge, should you choose to accept it:

• Enable altering the Custom Next X hours Sensor by setting X in an advanced option, or
• Creating an action to set X.

I know this has been asked for before, and I also know that I could probably pull 30 minute detailed forecast data and manipulate it, but who doesn't need a challenge?

[autoSteve]

Action.

I shall Urdu fur youu.

[autoSteve]

Squeezed into #434, BJ. Service action.