Add an API endpoint for upstream update URL retrieval

[fonix232]

Is your feature request related to a problem? Please describe.

With all the usermods floating around, it's getting harder and harder to track which specific build one needs and how one can check if there are updates. Many clients (incl. wled-native, the Home Assistant integration, or anyone utilising libraries like Frenck's python-wled) have now built-in update pathways, however these are currently hard-coded to using the main repo - not exactly optimal to have HA or wled-native prompt you to update your device to a potentially incompatible build. An example would be my neat little 64^2 HUB75 matrix for which I'm using an Adafruit MatrixPortal S3, and the MoonModules fork that supports the protocol - which is still on 0.14.5, yet I'm getting constantly notified that there's an update.

Describe the solution you'd like

Add a simple API endpoint that returns a hardcoded value set during compilation, so that forks can provide their own GitHub address, which in turn can be used to check for new releases and updates.

Describe alternatives you've considered

I initially thought of adding fork detection + hardcoded update paths to the clients themselves, however upon seeing frenck/python-wled#1597 (which, based on Frenck's comments, I believe was the same core idea as mine), I realised that that idea pushes the responsibility of maintenance of the client developers. By having the firmware+device combo be able to tell "hey I'm this fork, and you can find updates for me [here]", this issue is alleviated, and is the superior solution.

Additional context

Obviously this would be a change that needs to propagate to the forks and would only become useful once widely adopted - therefore I think the sooner this gets pushed to main, the better.

On the technical side, this is a relatively straightforward thing to add, with the following steps:

1. Create a new const (say, WLED_OTA_REPO)
2. Assign a value to this const during the build process using build flags, otherwise leave it as an empty string.
3. in wled00/json.cpp's serializeInfo method, add a new entry to the root JSON object:

root[F("ota_repo")] = F(WLED_OTA_REPO);

And from this point on, any client library can utilise this field's value to check for OTA updates instead of relying on hardcoded URLs.

Caveats:

• all usermod builds/forks need to implement this step in their build process
• all clients need to transition to this approach while also keeping the legacy way of hardcoded base URL
• the above two means this change needs to be well communicated to all parties affected
• this would only support public GitHub projects as an update source

[softhack007]

@fonix232 thanks for your proposal!

I agree that - especially with HA users being quite relaxed about "letting HA pick and install updates for me" - the current approach of HA will end in a usability disaster, and lots of devices going offline (running bootloops because they were flashed with the wrong firmware update).

It might be possible to add new json endpoint like /json/fwinfo so that 3rd party tools can read out the necessary information.

Not sure if we would ever manage to get all usermod authors into the same boat, and extend your idea to usermods. In fact there are >30 usermods right now, that can be combined arbitrarily - each combination is a different firmware.bin file.
Normally you need a custom compile (DIY) to add them. Moreover, lots of usermods are not maintained by their original authors anymore, and some usermods won't even compile out-of-the-box when they were orphaned since long time.

afaik @Moustachauve has asked for something similar a time ago, and we agreed that WLED_BRAND and WLED_PRODUCT_NAME could be used to help 3rd party tools understand if its "vanilla" WLED or a fork like MoonModules WLED.

https://github.com/Aircoookie/WLED/blob/709aeff9ea4cc502270b05d9e73971adb9802e89/wled00/json.cpp#L805-L806

This is also implemented in MoonModules WLED - I can't speak for other forks, like WLEDtubes by @SteveEisner and @craiglink.

WLED-MM has WLED_PRODUCT_NAME "MoonModules" while vanilla AC WLED has WLED_PRODUCT_NAME "FOSS". Both forks will report WLED_BRAND "WLED".

The current solution only allows to identify the fork, however its true that each fork has own naming conventions for firmware files in the "release" section of the repo.

We always wanted to find a way to atomaticially mark custom builds as "custom", to prevent friendly HA from overwriting it with a standard build from another repo. But this has not been implemented yet - we are currently struggling with more basic things like nightly builds using github CI actions (github API's were changed several times during the last year).

I'm using an Adafruit MatrixPortal S3, and MoonModules fork that supports the protocol - which is still on 0.14.5, yet I'm getting constantly notified that there's an update.

This sounds like a bug in the HA update procedure - We (MM team) actually changed our versioning scheme with the last release, in particular to stop such crazy update notifications. WLED-MM latest version is "14.5.0-beta" without the leading "0.", so HA should not try to overwrite a "14.x.y" with a "0.15.z" from another repo.

[softhack007]

PS: i have some boards at home running custom builds of ESPHome, and for some reason HA is able to distinguish custom builds (don't update) from vanilla standard builds (offer updates). Maybe we could learn from ESPHome?

[blazoncek]

FYI WLED's info object:

{
	"info": {
		"ver": "0.15.1-b1",
		"vid": 2501090,
		"cn": "Kousen",
		"release": "dev_release",
...
		"fs": {
			"u": 102,
			"t": 983,
			"pmt": 0
		}
...
		"arch": "esp32",
...
		"u": {
			"Temperature": [...],
			"PIRsensorSwitch": [...],
			"MultiRelay": [...],
			"AudioReactive": [...],
...
		}
...
		"brand": "WLED-BB",
		"product": "Srg74-Wemos",
		"mac": "3c6105XXXXXX",
		"ip": "192.168.XXX.XXX"
	}
}

IMO plenty of information to decide if correct firmware (from official repo) is available. Adding custom repositories (as would be needed for above example) is beyond WLED unless auto-update is implemented in WLED itself.

[softhack007]

.... and here comes the same info from MoonModules WLED:

{
    "ver": "14.5.0-beta",
    "vid": 2501070,
    "release": "esp32S3_16MB_PSRAM_M_HUB75",
    "rel": "esp32S3_16MB_PSRAM_M_HUB75",
...
    "fs": {
        "u": 8,
        "t": 10354,
        "pmt": 0
    },
...
    "arch": "ESP32-S3",
...
    "clock": 240,
    "flash": 16,
...
    "psram": 8245467,
    "psrmode": "🚀 OPI",
...
    "u": {
        "Autosave": [...],
        "AudioReactive": [...]
    },
...
    "brand": "WLED",
    "product": "MoonModules",
    "mac": "dcdaXXXXXXX",
    "ip": "192.168.X.X"
}

So yes, lots of info to decide where to locate updates; the most important fields are aligned across our forks already:

• "product": "MoonModules" tells you its not from the main WLED repo that uses "FOSS"
• "ver": "14.5.0-beta" -> version number = release tag
  (or something ending in -dev if it comes from the development head, like 14.5.1-dev)
• "release": "esp32S3_16MB_PSRAM_M_HUB75" is the suffix of the firmware filename you need

So putting it all together -->WLEDMM(product) _ 14.5.0-beta(ver) _ esp32S3_16MB_PSRAM_M_HUB75(release) .bin is the file that was used.

I don't fully agree with @blazoncek, because it would be simple to add one extra attribute to info that identifies the URL, like

"repo": "https://github.com/MoonModules/WLED/releases",
"file_prefix": "WLEDMM",

we could automatically populate this field from package.json using a small platformio script - similar to how we extract the "ver" field - i.e. url: "git+https://github.com/Aircoookie/WLED.git" --> -D WLED_REPO = "https://github.com/Aircoookie/WLED/releases"

the downside is the string occupies ~60 extra bytes on flash, which is a very tight resource already.

[blazoncek]

@softhack007 I said "unless auto-update is implemented in WLED itself". This does not exclude implementation.

And BTW there is intent on doing that.

[netmindz]

I tried to start similar fix for HA to be repo aware but blocked at the first hurdle by the author of python-wled

frenck/python-wled#1576

If we try your model of per-device repo URL for each WLED device rather than selecting your repo on HA preferences, then we would still need to either create and maintain a fork of python-WLED and puruade HA to use that or trying and puruade the author to reconsider

[softhack007]

... as @frenck is also the maintainer of the HA integration for wled, I think the outlook for making this happen is not good 🤷‍♂️ at all.

Edit: @netmindz your short discussion with him sounds like HA has no interest to even think about forks that are not OTA compatible with AC WLED
... so my conclusion is that HA devs simply prefer to overwrite any WLED installation with AC WLED, no matter if users will lose functionality or even devices may crash (bootloop) after uploading an incompatible binary. Why am I not surprised 🙄 ?
NB: the OTA file format does not allow us to perform any additional compatibility checks before writing to flash.

[netmindz]

I don't fully agree with @blazoncek, because it would be simple to add one extra attribute to info that identifies the URL, like

"repo": "https://github.com/MoonModules/WLED/releases",

It would be better to read this from git remote info rather than a value that needs editing when people create their own fork

we could automatically populate this field from package.json using a small platformio script - similar to how we extract the "ver" field - i.e. url: "git+https://github.com/Aircoookie/WLED.git" --> -D WLED_REPO = "https://github.com/Aircoookie/WLED/releases"

the downside is the string occupies ~60 extra bytes on flash, which is a very tight resource already.

You only need to know the namespace and project, e.g "AirCoookie/WLED" as the rest can be hard coded prefix and suffix in the python script used in HA or on in the mobile app

[netmindz]

... as @frenck is also the maintainer of the HA integration for wled, I think the outlook for making this happen is not good 🤷‍♂️ at all.

Edit: @netmindz your short discussion with him sounds like HA has no interest to even think about forks that are not OTA compatible with AC WLED
... so my conclusion is that HA devs simply prefer to overwrite any WLED installation with AC WLED, no matter if users will lose functionality or even devices may crash (bootloop) after uploading an incompatible binary. Why am I not surprised 🙄 ?
NB: the OTA file format does not allow us to perform any additional compatibility checks before writing to flash.

From his limited feedback it felt more like he just didn't want to open the can of worms of doing anything at all with anything that isn't AC - just in case there are differences in the actual API in those forks.

What I'm asking for and what my PR does is nothing to do with the API, just allowing a hard coded value in the library to be one that can be overriden, so doesn't really change the likelihood of people reporting issues with the library being used against anything other than AC

[blazoncek]

If you will be adding this field to JSON API be sure to measure generated JSON size on ESP8266 and most used usermods included.

simply prefer to overwrite any WLED installation with AC WLED

I totally understand that. It would be a nightmare from the support perspective otherwise. That is also why I was pushing so hard to get many usermods included by default.

does not allow us to perform any additional compatibility checks before writing to flash

This is not entirely true. If you try to OTA S2 binary to S3 it will be unsuccessful and will not render the device inoperable. However OTA S3 OPI to S3 QSPI device will brick it.

[fonix232]

Adding custom repositories (as would be needed for above example) is beyond WLED unless auto-update is implemented in WLED itself.

I strongly disagree. This isn't a value used by the firmware itself, but rather - just like the other identifiers used by usermods/forks - a field that says "hey, I came from here, check here for updates". WLED itself doesn't need to implement auto updates, that's down to the client, but for the client to provide the right update, it needs to look in the right places for the update in the first place. Since neither the brand nor the product nor the release field tell the client where the updates reside. Adding the extra field has absolutely no negative effects, and usermod developers can even ignore them (in which case the clients can easily fall back to the current approach of checking for updates through the hardcoded URL), and adding them is quite literally just slapping a compiler parameter in the build configs (let it be the platformio INI, or a GHA workflow call to make).

PS: i have some boards at home running custom builds of ESPHome, and for some reason HA is able to distinguish custom builds (don't update) from vanilla standard builds (offer updates). Maybe we could learn from ESPHome?

Bit off topic, but ESPHome actually does offer updates for custom builds. In fact, aside from the basic Tasmota-style firmware ESPHome offers (the "initialisation build"), every single other configuration is custom and requires a custom firmware compilation (which the ESPHome addon takes care of). ESPHome has its own issues with this too (e.g. there's no way to create a "generic" firmware you'd flash on multiple devices of the same configuration, where the only values changing would be the device name/ID, and network stack, so if you have 12 of the same device with the same ESPHome config, you need to compile that 12 times), but this is actually something they do right.

So putting it all together -->WLEDMM(product) _ 14.5.0-beta(ver) _ esp32S3_16MB_PSRAM_M_HUB75(release) .bin is the file that was used.

There's actually another bug with this approach that I felt wasn't serious enough to warrant a bug report/feature request, as it only affects a small subset of devices: many older (ESP12/8266 based units mostly) can't handle the larger "full size" update BINs and need to rely on GZIP'd files. I have three such controllers in my home right now (Mostly Athom and Gledopto), and it's always a fight for them to get the right file.

Again strafing off topic, but maybe it would be worth adding a field to the info block that tells the client the OTA partition size so if part_size < update_size, and update has a GZIP variant that fits, then the clients can opt for that build. But again, this is very off topic for this FR - however I'll gladly open a separate one, if for nothing else, to discuss the viability of this.

I said "unless auto-update is implemented in WLED itself". This does not exclude implementation.

And BTW there is intent on doing that.

I think we need to be cautious with adding these extra features on firmware side. As pointed out above, WLED is already growing beyond what some older models are capable of (storage, computing performance and RAM bound), and given there's plenty of off-the-shelf models being sold with those older chipsets, dropping support for those in favour of built-in auto OTA is something many people would not be happy about. Of course I have much less visibility of this given my limited number of devices and models, so I refer to the team's (and usermod devs') expertise and experience here, but I felt it needs pointing out this possibility.

we could automatically populate this field from package.json using a small platformio script - similar to how we extract the "ver" field - i.e. url: "git+https://github.com/Aircoookie/WLED.git" --> -D WLED_REPO = "https://github.com/Aircoookie/WLED/releases"

the downside is the string occupies ~60 extra bytes on flash, which is a very tight resource already.

Given that the update checking logic would be GitHub bound anyway, these 60 bytes could be reduced to organisation/repo format only, meaning a simple -D WLED_REPO = "Aircoookie/WLED" would suffice. That's 15 bytes instead of 60.

Another potential approach would be enforcing that usermods follow a GitHub naming convention of [product]/WLED, that way there's no need to add any further fields, but it heavily limits the implementation and the usermods too - plus it would require changing the main/official builds' product field from FOSS to Aircoookie, which could cause further issues.

I tried to start similar fix for HA to be repo aware but blocked at the first hurdle by the author of python-wled

@netmindz the issue with this, as I pointed it out in my original post, is that it puts ALL the responsibility on the client developers. Right now they already have to maintain the auto OTA feature, which is hard enough, but having them monitor forks/usermods and keeping an up to date list of product name to repo mapping, is simply not tenable.

By pushing the update repo location into the firmware, this check becomes as simple as:

current_version = semver.parse(wled.info.ver)
if( wled.info.repo.isNotEmpty() )
    repo = wled.info.repo
else
    repo = Aircoookie/WLED
endif

releases = github.get_releases(repo).filter(release -> release.semver > current_version)
if( releases.isNotEmpty())
    release = releases.sort( release -> release.semver, descending).first()
    wled.update(release.file_url)
endif

The logic change here is as simple the first if/else block instead of the fixed repo. Minimal change, yet opens up OTA functionality to most forks/usermods.

@frenck was right to refuse your original idea as that gives him an immense amount of extra work not just once, but also through continuous maintenance. By lifting that out and putting the responsibility on the fork/usermod maintainer, I'm fairly certain he'd be agreeable to the above outlined pseudocode'd change.

... as @frenck is also the maintainer of the HA integration for wled, I think the outlook for making this happen is not good 🤷‍♂️ at all.

I disagree, Frenck in the past has been quite open to sensible changes. But having the client developers deal with all the possible forks, with absolutely no help coming from the device itself, that simply doesn't make sense. Why should the CLIENT be aware of all the forks, when it's absolutely not necessary, and the firmware itself could report where it came from.

Edit: @netmindz your short discussion with him sounds like HA has no interest to even think about forks that are not OTA compatible with AC WLED

To be fair, it would introduce a fair bit of complexity to be considerate of forks on their end. At the end of the day, anyone's free to fork both python-wled and the WLED integration, and add their own code. If I wanted to, I could have added the MoonModules support to python-wled and gotten my needed path of auto upgrades with much less trouble.

... so my conclusion is that HA devs simply prefer to overwrite any WLED installation with AC WLED, no matter if users will lose functionality or even devices may crash (bootloop) after uploading an incompatible binary. Why am I not surprised 🙄 ?
NB: the OTA file format does not allow us to perform any additional compatibility checks before writing to flash.

At the end of the day, it's up to the user to ensure updates are correct. HA only offers an update, and if it's incompatible, you can always disable it. But yet again I have to repeat myself, WLED at the moment offers no way to determine where a non-AC variant came from, and pushing the responsibility on the client devs to maintain a list of all active usermods, repositories, update approaches, is simply not viable. With WLED providing that info and usermod makers sticking to a good pattern in naming files, etc., support can be added without the need for continuous maintenance.