Thing/Item: move into core the conversions to/from YAML code done by MainUI

[lolodomo]

When we discuss about new format for configuration files, one wish would be to use a common (YAML) syntax in MainUI code tab and in the new file format. To make this dream a reality, a required initial step is to replace the conversion to/from YAML currently hardcoded in MainUI by new REST APIs in core in charge of these conversions. MainUI would then call these new APIs when needed without handling YAML conversion by itself. This would also allow to show easily another format (like our current DSL format) in the code tab just by passing the requested format to the API.

@openhab/webui-maintainers : I need your help to identify what are the APIs you need for that. I guess two APIs are necessary, one to convert to YAML and one to parse YAML.

When entering in the code tab, you need to build the YAML code from the current internal data, these data are potentially different from the thing stored in the registry because the user may have started to change configuration parameters for example. In #4569, I already added the API GET /things/{thingUID}/syntax/generate but it is based on the thing in the registry. So I believe I should add another similar API where you can pass as parameter thing data (ThingDTO) like in the API to create a new thing, and the returned output is the generated YAML code: POST /things/{thingUID}/syntax/generate

Then when leaving the code tab, you need an API to parse the YAML code potentially updated by the user and retrieve the corresponding thing data (ThingDTO). I could add a new API POST /things/{thingUID}/syntax/parse with a parameter in which you provide the YAML code and the returned output is corresponding thing data (ThingDTO).

@florian-h05 : please confirm it would be OK and sufficient for MainUI needs.

In the code tab, I can see that all channels are listed. Does it mean you support adding/removing of channels through this tab ? Or do you show them just for information ? Or just to let the user update a channel configuration parameter ?
Of course in the future config file, the user should not define all these channels while they can be deduced from the thing type. So it is a point to clarify.

[lolodomo]

If this initial step can be achieved, we can then discuss about small improvements of the syntax to make it potentially more user friendly and of course a compromise should be found.
Defining a thing does not require a lot of information, it is relatively simple. Current syntax defined by MainUI looks not so bad and I do not imagine big changes. And we should thing to users already used to the YAML syntax in MainUI code tab.
Note that these adjustments would then require no changes in MainUI as soon as MainUI is using APIs, and they will be automatically applied to what is displayed the code tab.

[lolodomo]

In the code tab, I can see that all channels are listed. Does it mean you support adding/removing of channels through this tab ? Or do you show them just for information ? Or just to let the user update a channel configuration parameter ?

The existing API to update a thing expects at input either no channels and no change of channels for the thing is done in this case, or a list of channels and the thing will then be updated to have only this list of channels. So MainUI needs to maintain a list of all channels for a thing to call the update API. That might be the reason why all channels are shown in the code tab ? I am also asking myself if the purpose is also to let user remove channels ?

That could be something preventing to have exactly the same YAML code if MainUI needs to display all existing channels while in a config file, the user would like to consider only non default channels. Even if the syntax itself remains common.

[FlorianSW]

@FlorianSW : please confirm it would be OK and sufficient for MainUI needs.

Are you sure you wanted to ping me with this? :) I did not work on MainUI or openhab things in quite a while 🙈

[lolodomo]

Are you sure you wanted to ping me with this?

No, it was a mistake, my intention was or course to ping @florian-h05

[florian-h05]

The proposed API are sufficient in general, it of course causes some network traffic but I think we can live with that.

The existing API to update a thing expects at input either no channels and no change of channels for the thing is done in this case, or a list of channels and the thing will then be updated to have only this list of channels. So MainUI needs to maintain a list of all channels for a thing to call the update API. That might be the reason why all channels are shown in the code tab ? I am also asking myself if the purpose is also to let user remove channels ?

a. I have only tried this for Things with unmodifiable channels, there it is possible to modify the channel config.
b. I would expect that for Things with modifiable channels, you can add and remove them from the code tab as well.

In both cases, having them in the code tab is a great help. I think in b, it should be easy to keep the current behaviour because you actually need to define the channels manually, a is more difficult to handle though.

[rkoshak]

MainUI would then call these new APIs when needed without handling YAML conversion by itself. This would also allow to show easily another format (like our current DSL format) in the code tab just by passing the requested format to the API.

Could it make sense to supply the mime type as part of the request instead of creating a whole new API endpoint? If the request includes "application/yaml" instead of "application/json" and it defaults to "application/json" could that work? DSL could use "application/dsl" and so on when/if new formats are developed. Then on the server side it knows what format is desired and can convert (as necessary) back and forth for new and updates to stuff.

Or is that what you are already proposing (it doesn't seem so but I could be wrong)? Or is there something about doing that which wouldn't work?

b. I would expect that for Things with modifiable channels, you can add and remove them from the code tab as well.

I can confirm this, at least when it comes to MQTT and HTTP bindings.

That might be the reason why all channels are shown in the code tab ? I am also asking myself if the purpose is also to let user remove channels ?

My understanding is it just shows the raw JSON returned by the API converted to YAML. I don't think there is any more reasoning than that. And the raw JSON returned by the API includes all the Channels, as would be required to show all the Channels in the Design tab.

To complicate a. a little more, what if I want to modify a Channel that I haven't modified before? There'd need to be a way to only show modified Channels and show all Channels and switch between the two.

[lolodomo]

Could it make sense to supply the mime type as part of the request instead of creating a whole new API endpoint?

No because we have currently no API matching UI needs.
Your remark could be more appropriate to what I initially implemented in #4569 but doing that will be a breaking change for most common APIs (getting a thing or an item) and the existing APIs already contain parameters that are more or less specific to JSON output. And I need also a new parameter not needed for JSON output. Don't want to over complicate the existing APIs.
But I will study the idea again, at least for the API returning the list of all items and the API returning the list of all things.

I have now updated the new APIs in #4569 to limit the new APIs and to cover what is needed by MainUI.

And in parallel, I already made a POC to implement YAML for things, it is relatively simple, not a lot of code lines because of great work done by @J-N-K on YAML registry after my initial contribution for YAML tags . Maybe I will include everything in #4569 .

[spacemanspiff2007]

Could it make sense to supply the mime type as part of the request instead of creating a whole new API endpoint?

I thinks that's very unusual and would be both hard to teach and document e.g. in api explorer.
The most common way another output format can be selected is typically a query parameter.
I think a dedicated api endpoint that does the conversions is much nicer both usability and documentation wise.

[rkoshak]

I thinks that's very unusual and would be both hard to teach and document e.g. in api explorer.

I guess I don't understand why accepts media type header field exists in the first place then if it can only ever support the one media type.

[florian-h05]

Accept Media type is a HTTP header used to signal the server which media type the client accepts. It’s rather technical IMO and nothing to use to say I want format A or B.
It basically signals which technical format the browser accepts the response in.

[FlorianSW]

Could it make sense to supply the mime type as part of the request instead of creating a whole new API endpoint?

I thinks that's very unusual and would be both hard to teach and document e.g. in api explorer.
The most common way another output format can be selected is typically a query parameter.
I think a dedicated api endpoint that does the conversions is much nicer both usability and documentation wise.

Just sliding in for this single comment, so sorry if this is out of context :)

It's, from my experience, a workaround to represent different formats to serve by the server using a query Parameter. A much more standard way is in fact using Server-Side content-negotiation. This includes format, language and encoding, if supported by the server. See this MDN doc for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation

It's also untrue that the Accept header wouldn't reflect a preference of a client (format A rather than format B). It's intentionally meant for that. It's a list of media types the client is able to process, which can, optionally, also include a relative preference to each other. E.g. an accept header value can express "give me json over XML, while XML is also ok if the resource is unavailable in json, but never provide me csv". The server then needs to respond in either of the requested mime types, preferably with the one the client wants the most. If none of the requested formats is available, a status code of 415 should be the response.

So without being really involved here anymore, rather than having a query Parameter, using content negotiation would feel more naturally for an API, IMHO.

[mherwege]

It's, from my experience, a workaround to represent different formats to serve by the server using a query Parameter. A much more standard way is in fact using Server-Side content-negotiation. This includes format, language and encoding, if supported by the server. See this MDN doc for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation

I am not an API expert, but I must say I am using API's at work where the response is JSON or XML, depending on the Accept header value in the request. So I am with @FlorianSW on this. I see this as perfectly valid use of the Acceptheader and I don't think it makes using the API more complicated. We are talking about the same response, but in a different format. A generic translation endpoint could then be sent something in one format, and return in another format, depending on the Accept header.

[spacemanspiff2007]

The issue is that depending on the target format different input structures for a post request are required.
So it's not accessing the same resource but a different functionality which requires a different input.
As far as I know it's not possible to document different input structures based on the content headers which means the api documentation has to be done manually (e.g. in the openHAB docs) which is undesirable.
For returning e.g. all existing items this could be used, but it would move file based parts from one endpoint to another, so this is rather unexpected. However I don't have any strong opinion on that.