Status on available format(s) for config files and integration in Main UI

[lolodomo]

This topic is to share a status on our config files, the supported formats, the available conversion between formats and how Main UI is handling them (import from them, export to them, displaying them in code tab).

config file type	DSL format	YAML format	Integration to conversion REST API	Main UI import in UI DB	Main UI export to config file format	Config file formats displayed in Main UI
tags	no	yes	yes	no	yes (through COPY button in code tab)	yes
items	yes	yes	yes	yes	yes	yes
things	yes	yes	yes	no	yes	yes
sitemaps	yes	yes	yes	yes (through DUPLICATE button in sitemap editor)	yes	yes
persistence	yes	no	no	no	no	no
rules	yes	yes	yes	no	openhab/openhab-webui#4202	openhab/openhab-webui#4202
pages & widgets	no	yes	no	no	yes	yes

Everything else is based on properties files (list of key=value).

So I think the last config file that is fully and only available as DSL format is the persistence configuration.

Do we have something else depending exclusively on DSL ? Transformation ?

[rkoshak]

Yes, one can write a transformation using Rules DSL once the generic Script Transformation was added (OH 4.0 I think). But we don't have a YAML format for these so maybe it's not relevant.

@jimtng has a couple PRs open to create a YAML file format for Pages and Widgets. But there never was a DSL version of the file format for that so probably doesn't need to be listed here.

The row for transformations might look something like this.

config file type	DSL format	YAML format	Conversion between DSL and YAML	Conversion integrated in MainUI
Transformations	yes (but it's just the raw code and only for DSL script actions, not universal)	no	no	N/A

There is a JSON format for transformations though, which can be used to publish a transformation to the marketplace. It's just the raw JSON from the JSONDB. I'm not sure if that's something that needs to be addressed here or not.

[jimtng]

YAML pages and widgets are now supported in core #5493 (merged). Since they don't have their DSL counterparts, I have implemented the YAML-ization purely in webui openhab/openhab-webui#4115 (open). The code tab will be copy-pasteable to files. It also now supports selecting multiple pages / widgets and copying all their YAML definitions just like Things / Items.

[Nadahar]

Do we have something else depending exclusively on DSL ? Transformation ?

I don't know much about transformations, but yes, I have always expected them to be made available. Transformations are available from the marketplace, and they can be provided using JSON or "old YAML". I guess it should be possible to provide them from the "new YAML" as well. Here is an example of a JSON transformation:

{
  "uid": "config:js:ohrt_ScaleUnits",
  "label": "Scale Units",
  "type": "js",
  "configuration": {
    "function": "(function(data, lowUnit, highUnit, threshold) {\n  console.loggerName \u003d \u0027org.openhab.automation.rules_tools.ScaleUnits\u0027;\n  console.trace(\u0027ohrt_scaleUnit invoked with: \"\u0027 + data + \u0027\" \"\u0027 + lowUnit + \u0027\" \"\u0027 + highUnit + \u0027\" \"\u0027 + threshold + \u0027\"\u0027);\n  let rval \u003d Quantity(data).toUnit(lowUnit);\n  console.trace(\"To low unit result \" + rval);\n  if(rval.greaterThan(Quantity(threshold))) { \n    console.trace(\"Over threshold, converting...\")\n    rval \u003d rval.toUnit(highUnit);\n  }\n  console.debug(\"Before conversion \" + data + \" after conversion \" + rval);\n  return rval.toString();\n})(input, lowUnit, highUnit, threshold)"
  }
}

There is a JSON format for transformations though, which can be used to publish a transformation to the marketplace. It's just the raw JSON from the JSONDB.

The marketplace has support for YAML as well, although that's the "old" YAML that is a raw mapping of the JSON. It doesn't seem like anybody has used it though, but the marketplace handler supports it.

In addition, we have Blocky block libraries available in YAML on the marketplace. Those could be added too, even though they don't exist in DSL. It is a bit related to #5493, and I'm not quite sure why it wasn't included there, but it could be done by "extending" #5493 as far as I can tell.

[Nadahar]

@jimtng Is there a particular reason why you left out blocks? They seem to be closely related.

[jimtng]

@jimtng Is there a particular reason why you left out blocks? They seem to be closely related.

My main needs were for the pages / widgets. Besides, Blocks seem to be a totally different thing (automation vs UI).

[Nadahar]

Blocks seem to be a totally different thing (automation vs UI).

The blocks are UI components and "live in the UI" exclusively. Core/automation has no knowledge of blocky at all, the UI converts it to JavaScript. Which is why I see them as "related", I think it's the last of the "UI components" that can be provided.

[lolodomo]

YAML pages and widgets are now supported in core #5493 (merged).

Added in the table. Do we have an export (Copy Definition) feature in Main UI for that ?

[jimtng]

YAML pages and widgets are now supported in core #5493 (merged).

Added in the table. Do we have an export (Copy Definition) feature in Main UI for that ?

Yes we do. It doesn't use core file-format because the object structure is identical to the file structure. It just needed to be wrapped in pages: or widgets: along with version: 1 (hard coded)

[jimtng]

Blocks seem to be a totally different thing (automation vs UI).

The blocks are UI components and "live in the UI" exclusively. Core/automation has no knowledge of blocky at all, the UI converts it to JavaScript. Which is why I see them as "related", I think it's the last of the "UI components" that can be provided.

What I meant was Pages + Widgets are for user Interface (to create things people click / view), as opposed to automation / coding / rules.

I have no interest in working on Blockly / Blocks. Tried a few years ago, but found it too resistant to changes.

[Nadahar]

I have no interest in working on Blockly / Blocks. Tried a few years ago, but found it too resistant to changes.

I have no interest in Blockly either, as with many other things I make. I just suspect that the implementation is almost identical to what you already did for pages and widgets, but of course: I can't know that for absolutely sure without looking into it. I just suspect that it would be very easy for you to do with the "knowledge" you currently have from working with widgets and pages.

[jimtng]

I have no interest in working on Blockly / Blocks. Tried a few years ago, but found it too resistant to changes.

I have no interest in Blockly either, as with many other things I make. I just suspect that the implementation is almost identical to what you already did for pages and widgets, but of course: I can't know that for absolutely sure without looking into it. I just suspect that it would be very easy for you to do with the "knowledge" you currently have from working with widgets and pages.

To clarify: I don't mind Blockly and my son uses it sometimes to write rules. Even though I found it lacking every time I looked at it, I have no interest in working on / submitting PRs related to Blockly anymore. Years ago I spent a lot of time working on it + its documentation, only to be rejected. Time totally wasted there.

[lolodomo]

For the users wanting to switch from config files to objects "managed" by Main UI (using JSON format), we only have an import feature for items files in Main UI. Maybe this feature should be extended in Main UI to support things, sitemaps, ... @jimtng @mherwege WDYT ?

[Nadahar]

For the users wanting to switch from config files to objects "managed" by Main UI (using JSON format), we only have an import feature for items files in Main UI.

For rules there's no explicit "import", but you can duplicate a file based rule so that it becomes managed. This isn't in my current PR, it was part of my previous rules work in MainUI (I think it was the PR for regenerating template based rules) and is already there in the released version of OH.

I don't know if you need an explicit import in addition to that..? It achieves the exact same thing only that you don't get UID conflicts because when you duplicate the rule, you must specify a new UID (AFAICR).

[lolodomo]

I added a new column in the table for marketplace.
@Nadahar : for tules, this is now our YAML format that is supported or still an old YAML ?
For pages & widgets, is only a JSON format ?