Support "new YAML" on the marketplace for rule templates, widgets and block libraries

[Nadahar]

This adds support for both the "new YAML" and the "old YAML" syntax for marketplace add-ons. The document is checked for whether it starts with version: or not, if it does, the "new" parsed is used, if not, the "old" parser is used. This is necessary to not break all the existing add-ons on the marketplace, while enabling new ones to use the "new YAML" format.

This PR is simple and should be complete, but it depends on both #5572 and #5588, and must be rebased after they have been merged. It's thus futile to start reviewing this until that has been done, which is why I'm creating it as a draft.

[Nadahar]

One thing that should be mentioned with this PR: It allows installing rule templates and widgets as isolated add-ons, but it doesn't allow any kind of "combined installation" where you can combine multiple types of objects in the same YAML file.

While the YAML format itself allows this, it doesn't automatically translate to the marketplace, because there are different Providers, and currently, they can only provide one type of objects each.

It is possible to allow combinations of rule templates and widgets, but it will require some redesign/rethinking of the marketplace handlers. The question is: How useful will that be? And if so, how should it be handled? Perhaps, instead of modifying one of the existing handlers, we should make a new "combined" handler that can provide any supported object type. But, again, how much sense does it make to bundle rule templates and widgets as "one add-on"? I don't know...

[rkoshak]

What about more than one rule in one template? I suspect the answer is "no".

But, again, how much sense does it make to bundle rule templates and widgets as "one add-on"? I don't know...

There are several rule templates on the marketplace with links to widgets and vise versa. So there is a demand for that. Usually they are widgets that require a special rule or rules to drive them. Bundling them into one would be more convenient for both developers and users.

[Nadahar]

What about more than one rule in one template? I suspect the answer is "no".

That should be fine. As many rule templates as you want in one add-on, as many widgets as you want in one add-on - it's the mixing of types that won't work as it is now.

edit: I'm not sure what more than one rule in one template means, but multiple rule templates in one YAML document should be fine.

[Nadahar]

There are several rule templates on the marketplace with links to widgets and vise versa. So there is a demand for that. Usually they are widgets that require a special rule or rules to drive them. Bundling them into one would be more convenient for both developers and users.

Ok, we should figure out how it makes sense to provide them in combination then. I don't know if that's realistic for 5.2 though, we'll probably have to make a new category on the forum with a separate "handling flow". I'm not sure of the details here, but I suspect that there can be some challenges in regard to exactly how they should be "stored" in JSONDB. Widgets are really a "UI only" concept, that is just stored in core, not "understood", while rule templates are fully integrated. So, treating them the same way doesn't necessarily make sense/isn't necessarily possible.

[Nadahar]

Currently, marketplace rule templates "live" in marketplace_ruletemplates.json in JSONDB, while widgets "live" in uicomponents_ui_widget.json. I'm not sure if the combined add-ons can be made to use these, or would need their own storage location.

Another thing is that I don't quite understand why blocks haven't been included in "the new YAML format". It seems like they should fit, which would make the "combined type" able to hold all three types. I'm not sure if that would make sense though, but it looks like blocks might have been overlooked here.

[rkoshak]

Another thing is that I don't quite understand why blocks haven't been included in "the new YAML format".

Probably mainly because no one has done it. @jimtng has mentioned more than once he's no interest in Blockly and he's the main one developing the file format for widgets. So I don't think it's been overlloked so much as deliberately ignored.

But it's not quite as useful to have a file format for block libraries. Except for archive, backup/restore, and sharing with others there is really no point in putting blocks into a separate file. You can't really use them in a file. You can't really even create Bl;ock Libraries in a file. Blockly is all graphical and not intended to be written by hand. You have to go to the UI to create them and to use them, so is it really worth while spending effort making an intermediate YAML format more human friendly?

If for some reason they become incompatable with the marketplace that is something else and should be addressed.

[Nadahar]

You have to go to the UI to create them and to use them, so is it really worth while spending effort making an intermediate YAML format more human friendly?

I know next to nothing about Blockly and blocks, so I can't know what is useful and not. What I can say is that they are, from core's POV, just another UIComponent, so I think implementing support for Widgets, Pages and Blocks is virtually identical. The difference between them only applies once they are in the UI, so I think it would have been extremely easy to add support.

Regarding Pages, I don't know if it has even been considered, or if it would be useful, but as I understand it, they can be parameterized in the same way as Widgets - so they could potentially also be "generalized" and shared, I would think.

Maybe sharing a ready-made page containing custom widget(s) might be of some value for some specific scenario?

[rkoshak]

Maybe sharing a ready-made page containing custom widget(s) might be of some value for some specific scenario?

I go to great lengths to not need a UI for my home automation so I don't know. But I could imagine sharing a whole page like that would be useful, particularly if the widgets in use are also included (i.e. you don't need to install them all separately).

[Nadahar]

particularly if the widgets in use are also included

Yeah, that was my idea - and that the whole thing could be packaged together with one or more rule templates. Regardless, that's not for this PR, as there's currently no support for sharing pages at all via the marketplace.

[Nadahar]

I've refactored the YAML handling of UI components using abstract classes and generics instead of just multiplying the same code. I've also added support for block libraries, so they can be added using blocks:. I have not modified the UI so that it adds the "outer layer" with version: etc. in the block library editor. I'm sure that's a very simple task, it basically just to repeat what was done for widgets. But, block libraries will now work if part of YAML files, and both the "old" and "new" format are now also supported for block libraries on the marketplace.

I think that's it for this PR, so I'm marking it read for review.

There is more to be done (as discussed above) regarding the marketplace in allowing "combined entries" that combines objects of different types, but will have to be after 5.2.0, because it requires some structural rearrangement.

[Nadahar]

@mherwege (or any other reviewer): To try to make this a bit easier to review, I'll try to explain a bit. The PR is really, really simple, but doesn't look so in the GitHub diff view because a lot of code has moved.

Widgets, block libraries and pages are all just RootUIComponents as far as core is concerned. All the separate them are different namespaces. The current implementation implements support for widgets and pages separately, and not block libraries, but the code is identical (except for renamed variable names). So, I rearranged this where the common code has been moved to abstract classes and only the tiny differences remain in the implementations. I also rearranged the package names, as it made no sense to me to treat these as completely different entities when they really aren't.

I wanted to add block libraries too, because I think it's confusing for users that "equal things have different rules". The choice was either to add another copy of the same code for block libraries, or to try to refactor it into something more sensible. I did the latter. The "new" code in the abstract classes are therefore almost entirely what used to live in the widget and page classes. I really wish Git, and particularly GitHub (TortoiseGit that I use shows this much better), would show such rearrangements in a different way, but that's the way it is.

After the refactoring to abstract classes, I added the little needed to support block libraries as well.

There is one exception to this, and that is what is now called YamlRootUIComponentParser (used to be YamlWidgetParser). Because what the parser provides is just RootUIComponents, I couldn't make different parsers for the different types, as OSGi would have no way to tell them apart (I would have to introduce a namespace of some sort that was also used). What I did instead is to make the "parser" handle both widgets and block libraries. It can very easily handle pages as well, but for now, they aren't supported by the marketplace, so there's no immediate need.

The fact that everything is just RootUIComponents complicate/messes things up slightly, but doing something about that is a larger task that needs more testing, so I didn't consider attempting that here. But, as far as I can tell, it all works.

All this refactoring is in the second commit, it could really be a separate PR, but I don't know if that's necessary. The first and third commits deal with the marketplace handling only, the first with widgets and rule templates, the third with block libraries (after they have become supported in the second commit). These two small commits is the "core" of this PR in the sense that they do what the description says.

The "rules" for the marketplace handling is the same for all types. It supports both the old and the new format by checking if the document starts with version:. If it does, the rest of the parsing is handed to the "new" parsers, otherwise they are handed to the "old" parsers.

[Nadahar]

Adding marketplace support for pages, should we wish to do so in the future, is extremely simple. The "main task" in doing that is probably to figure out how to organize it on the forum.

[Nadahar]

I've created a PR for the UI part for block libraries: openhab/openhab-webui#4279