HEMS tutorial

[nhoening]

These notes stem from a discussion with Nicolas and Vlad, May 27th 2025 in Delft

The tutorial is part of the KIFLIN project. Another one might be added (maybe as an extension) for using FlexMeasures as an S2-based HEMS.

Purpose

For the FlexMeasures project, such a tutorial is for developers (either non-hosts, but of course also hosts), and it can make clearer how to actually build services.

This tutorial should be using the FlexMeasures client (not the CLI, only when needed to make the account by the host, and to set up / run reporters) and also give UI screenshots that shows where to edit/add the same things in the UI.

The main purpose is to

1. show how FlexMeasures can be a HEMS for a modern home (with multiple flexible devices) and
2. actually model a problem from the ground up (other than the first tutorials, where there often was a bit of magic in creating structure).

FlexMeasures can build any energy problem, and hopefully that can become clear. If a reader is interested in building a service for an SME or public swimming pool, this tutorial will hopefully also be inspiring or at least helpful to imagine how FlexMeasures is to be used.

Scope

It might not be a 100% complete at first, but we can come close.

We aim for PV, battery, EV and heat buffer. To progress, we could start with

• v1: battery
• v2: plus-EV
• v3: plus-heat-buffer

Optionally, we could discuss leaving the battery out of it, so we only do PV & smart EV charging, then add a heat battery.
It can be that the client does not contain all functionality - we'd probably take that as an opportunity to add it. Maybe the same holds for the UI (starting a scheduling job would be cool).

Time

Crucial decision: the scope of time (how many days of data) and "live" or not (taking actual data or dummy sets). We'll:

• use two weeks of history, then schedule one week
• this is not live, we have to give building consumption and solar data, which would then not match whatever price data users would get for a live session. We can give tips (notes), however, how to start using live data at each point.

The plan / steps

We will focus on writing this as a client script, then afterwards write the tutorial.

This is what our script should do

It uses the client for almost anything, but for a few steps (1, 6) we use the CLI.

The first version could have only solar and battery, so we can see that it works. Then we add other assets, like EVs and/or heat pump.

Here is the general outline of the script (I'm not giving values for everything, let's figure this out as we go):

Part 1: Setup structure

1. Make an account and admin user in the CLI.
2. Create a public price sensor (no account). (1h, EUR/kWh)
3. Create building asset + sensors: general consumption (15min resolution, kW), energy costs KPI (1D resolution, EUR/kWh) + flex-context setting (which price sensor to use, consumption limit - but note this usually is not needed for private homes, relax constraints). The consumption sensor should be added to the inflexible-devices, as well.
4. Create PV asset (child of building asset) + sensors: production (15min, kWh). Add to inflexible devices in the building's flex-context.
5. Create battery asset (child of building asset) + sensors: power (15min, kWh), state-of-charge (15min, kWh) + settings: roundtrip-efficiency (85%), soc-min (15%) and soc-max (90%). Add power sensor to inflexible devices in the building's flex-context.
6. Create a weather station asset (public, location close by) and give it an irradiation sensor (1H, W/m²) and a cloud_coverage sensor (1H, %)
7. Add sensors to sensors_to_show for the building asset. Graphs: Energy flows, Battery SoC,

Part 2: Add data

All data sets would be stored next to the tutorial. Not sure how to split them, as I would like to realistically first upload the first two weeks, then only upload the next data points step-by-step in part 3.

1. Get price data for our date range (21 days) and upload the first two weeks.
2. Get a data set for building data, as well. If this is difficult, we could make one up.
3. Get irradiation forecasts (ask Mohamed) and upload the first two weeks. Note: these include cloud coverage forecasts - important later for the tutorial.
4. Also get actual PV production and upload the first two weeks.
5. Add PV forecasts with the CLI command, using irradiance as regressor for the second week, using only the first week as training. This only works in the CLI (flexmeasures add forecasts), so I suggest we experiment with calling that from Python, to keep all this within one script. We can explain in the tutorial that one would run these in a cron job.

Part 3: Scheduling (with some light simulation elements)

Now we go step-by-step through the 3rd week. Step size: 4 hours.

• Make forecasts of solar production starting at the first hour of the first day of the third week, with a length of 24h.
• Create a schedule via the client for the same period. We should use the multi-asset scheduling, so triggering a joint schedule for all flexible devices via the asset level. In V1, the only flexible device is the battery.
• Let's assume for ow that the schedule was followed for 4h perfectly, so the first 4h of scheduled battery power should be saved as measurements (horizon 0H) to the battery sensor. Similarly, we can also add the next 4h batch of building consumption, solar production from our raw data.
• The solar production forecast was not perfect, so the actual battery SoC during these 4h needs to be computed and saved in the SoC sensor. Also, the starting SoC for the next scheduling call can be taken from this.
• Then, we move the script 4h into the future (here is how we are basically simulating), and we restart this part. We loop through this until the end of the third week.

Part 4: Reporting

• Make two reporters: for energy costs and for solar self-consumption. Look at this tutorial for help (and maybe suggest improving it, or even add an actual page for report writing in general).
• Add them to sensors_to_show_as_kpis for the building asset.
• Every day at midnight, we run the two reporters. This probably requires a CLI call, as well.

(For V2 - this is not worked out in detail)

• Create 2 ESVE assets (child of building asset), which makes one charge point with two connectors. Each gets these sensors: power (15min, kW), state-of-charge (15min, kWh), soc-min (15min, kWh), soc-max (15min, kWh) + fill in flex-model settings: roundtrip-efficiency 85% and usually soc-min: 20%
• While scheduling, at 6pm every evening, add dynamic SoC targets for the next day: occupants work hybrid, so on 3 days per week (Tue-Thu) we need to drive to work and require 80% in the morning. Car leaves at 7 or 8am and comes back from work at 1pm, so during this time no scheduling should happen. I want to ask Ahmad how to do this in the best way. On other days, the car can charge freely.
• We can use random trips for shopping, so the schedule will have some unforeseen events.

(FOR V3 - this is not worked out in detail)

• Create heat pump / buffer asset (child of building asset) + sensors: TODO: learn from the InsideOut project, but simplify things like COP. Here, talking to Felix, TNO and Itho might be good (what is the minimal setting?)

Initial ideas about writing the tutorial

• Start with a picture/diagram of the house / assets (gotten from the UI). State the goal of the tutorial - what is the site/building like and what is the challenge that is being addressed.
• Explain data scope: We will add data & set points for the next week and start scheduling today and tomorrow.
• Add a note about scripting: we show client code, which can be useful to script such a setup for scale (UI is more useful for adaptations/service). Also note that scripting can be done for self-hosters in custom code (in plugin, can become CLI or API commands).
• Explain setting up each asset in its own section.
• Explain the site asset's graph page and how we set it up with the client.
• Add a movie showing the site in the UI, playing back some data / schedule, then showing the KPI for costs and self-consumption of the scheduled day
• ???

Next steps

• Initial outline [Nicolas]
• Prepare client, so it can update flex context and model [Nicolas]
• Prepare client, add ability to upload a data set [Vlad]
• Prepare CLI: Ability to add default asset types [Nicolas]
• V1: Solar & battery Part 1: setup structure [Vlad]
• V1: Solar & battery Part 2: add data [Vlad]
• V1: Solar & battery Part 3: scheduling
• V1: Solar & battery Part 4: reporting
• V2: Add 2 EVs
• V3: Add heat buffer

Later

• Add API command for triggering forecasting jobs, plus a client function. Then alter the tutorial.
• Adapt the data upload API endpoint to describe a date range within the file's data to use. E.g. we upload three weeks, but for some sensors we only want to save the first two weeks.
• Let the flex-context accept a SoC sensor from where to look up the SoC at a given time
• Scheduling via the UI. It's not going to be very soon and not highly critical to add this. If flex-context and flex-model are on assets, we can add a simple date picker maybe for adding SoC targets).

[Flix6x]

My comments:

• I made some edits to your steps so that now a hierarchical asset structure would be created. In the hierarchy I propose, the building represents the site, rather than one of the buildings on the site (which we usually do in simulation projects), because I feel that having the building be the top-level asset is more in line with a residential setting than an enterprise setting.
• Rather than building, I would call it home (likewise, signalling a residential setting).
• One missing step: Use flexmeasures-client to post realizations (measurements) to the FlexMeasures API.
• The KPI sensor doesn't serve a purpose in this tutorial, unless the user posts their self-computed cost data, because setting up a cost reporter to fill the KPI sensor with data is currently still a backend feature.
• The general consumption sensor on the building is meant to measure the power flow at the main meter, right? Or is it there to measure and forecast the residual demand of the building (so excluding the power flow of the flexible assets)? This requires some thought if this sensor is to be used as an inflexible-device-sensor.

[nhoening]

Thanks. Is adding/patching a reporter (with raw JSON for now) a possible new client feature?

I'd include listing and possibly deleting them, also.

[Flix6x]

Yes, that would be needed. But also a trigger endpoint and/or some way of configuring it as a recurring job. All out of scope of this tutorial, imo.

[nhoening]

Okay, but we can add/replace and run reporters via the CLI right? That is probably okay for now, too.

[Flix6x]

I agree. Accordingly, I revised the comment in your task description about when to switch from the client perspective to the host perspective (account setup + reporter setup/trigger).

[nhoening]

We'd need to use some flex-model fields here, and the preferred way for some of them would be to set them on the asset, and have a lighter scheduling call.

But until #1429 is ready, we will need to pass them with each API call. Let's make a note there that this would be an iteration.

[VladIftime]

PR in flexmeasures-client : FlexMeasures/flexmeasures-client#134