Enhanced YAML Configuration with Variables, Includes, Packages, and Anchors#4818
Enhanced YAML Configuration with Variables, Includes, Packages, and Anchors#4818jimtng wants to merge 1 commit into
Conversation
|
@lolodomo FYI. This shouldn't conflict with anything that you're working on, I think. |
jimtng
force-pushed
the
yaml-preprocessor-secretless
branch
from
704d716 to
b4f551c
Compare
|
rebased to main |
jimtng
force-pushed
the
yaml-preprocessor-secretless
branch
from
b4f551c to
60bb8b0
Compare
jimtng
force-pushed
the
yaml-preprocessor-secretless
branch
from
60bb8b0 to
d04a9a2
Compare
|
Todo: merge lists in packages |
|
@lolodomo When you find time, I would appreciate a short review from your end. |
|
Great Work!
|
No, they only need to be defined in the include directive.
Out of the three options, I prefer
So that you can have
Good question. As it is now, it will return To fix this, I need to make it ungreedy, but it would no longer work for |
|
@spacemanspiff2007 I added a commit that supports quoted defaults (in which you can use braces).
|
|
@jimtng I really like what you did in this PR! This is going to be a great feature which will help a lot and make configuration a breeze.
In python there is a Instead of Would it also be possible to not have to use the packages:
livingroom-light1:
file: esphome-light.inc.yaml
vars:
thingid: livingroom-light1How about we also add a
I understand. I guess it's quite unexpected that |
|
I think
The the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags. Perhaps I can pass an implicit variable
I'll consider it.
I think with the exclusion of *.inc.yaml it should be enough. Adding more subdirectories to exclusion may potentially cause confusion. You can of course structure your folder as ./includes/foo.inc.yaml if you wish.
This is consistent with sh / bash / ruby, e.g.
I thought of this, but it's a can of worm that I'd rather not have to deal with yet. |
I saw it after I sent the message. Both names are fine with me.
For items and things the key is the unique identifier. Since these packages are fully merged configs they have no unique identifier (which would be the path to the file). If you find a good use case for the Will you provide support for the short syntax e.g.
I thought some more and it might make sense to keep the
Yes - I noticed how everything is inspired by esphome. Will a change in a package/included file trigger a reload of the referencing file? |
It would be entirely up to the user to use it how they see fit. For me, I would use the PACKAGE_KEY as the thingid in the example above so there is no need to specify
That is true, although that wasn't the main reason. Using I've just tried implementing what you suggested and it resulted in more complexity and code duplication. This is of course simply an implementation detail invisible to the user.
Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too. |
I think
|
I think it's rather common that these have different "rules" when it comes to substitutions, escaping etc. I'm not weighing in on whether it's a good idea here, I'm just saying that generally I think that is acceptable. But, doesn't YAML already have some "rules" regarding quoting to take into account? As far as I know, YAML only allows gives the escape character
That's preferable IMO. When making such "special codes", you might as well make them easy to use.
Don't. It shouldn't become "a mantra" that is more important than reason. People will have to learn to use arrays too, and it's the only thing that makes sense here.
In reality the terms matter less than we think, as soon as something is chosen and its meaning is well-defined and documented, people will quickly "get used to it". If the similarity with the ESPHome format is kept, parity with it might have a value in itself (for users using both systems). |
Your right about the
So this will be supported?
Let's keep the
But imho this would work the same way if you need to do changes now in an items and things file. If I reuse an |
I haven't tried this exact scenario but yes it should work just like you expected. That's essentially how
I think it depends. Besides, most people edit files in-place.
The two files are different. One is an include file, the other is the main file.
Yes, exactly the idea, but I do see the convenience of having it auto reload and on the other hand, it can be annoying for those who like total control that eventually someone would ask for the option to turn it off. I am so used to doing it manually and the way I currently do it is using one big main file, so I only need to reload one file.
It would be an effort yes, and I am currently on vacation with very limited "computer" time until end of June. This will have to be a separate PR later.
It's not merely following a mantra. I actually much prefer to have it as a map. But supporting a list is easy(easier), I'll see what it looks like and try it and if it looks OK, I'll consider supporting both syntaxes perhaps. |
I think the only sensible thing is that the templates "hot reload" too, otherwise you'd have to figure out and chase down which files are affected. I mean, the point of using a template is to avoid duplicating information. If you still have to hunt down the individual files and resave them, you could almost might as well just have copy/pasted. But, I guess that this can be a challenge implementation-wise. You would have to keep track of basically all "YAML model" files, and then trigger reparsing when required. It sounds like a path full of pitfalls. |
|
As I said, it depends on your workflow. I've been using a similar system and workflow in openhab for at least three years and preferred manual reloading. I have an implementation in mind on how to do the auto reload but will also need to add a setting not to do it for those who prefer not having it. I also have in mind how to do this A.global option through service config I'll implement this in a late pr if/when time permits or at the end of June. |
jimtng
force-pushed
the
yaml-preprocessor-secretless
branch
from
0012f04 to
7743285
Compare
|
Everybody was tired today and returned early, so I had some time to implement the include dependency tracking. Now whenever any include files are changed, the main yaml will be reloaded. |
jimtng
force-pushed
the
yaml-preprocessor-secretless
branch
from
6ad3002 to
0eecf01
Compare
No, the preprocessor is applicable to files only |
|
@jimtng |
I've just added it.
Yes.
I find it very cumbersome to use
I don't understand your point. Please elaborate. |
|
@spacemanspiff2007 you may not be aware of where this idea came from: |
Isn't that rather an argument against
Thanks for the background info.
I guess we disagree then 😉 . There should be one way to do something and we already have It's very unexpected that It's a very opinionated implementation and I really don't like it because it mixes file paths with something different without being clear about it. |
No. It's an argument against typing out
How Pythonic! I am not good with words, and it's once again almost 5am, so I asked Copilot to create an explanation: Response to Concerns About the
|
|
I'm finding myself writing "@@pkg/xxxxx" and that's still one extra character to type and it looks weird to me. Considering that
I will change it to:
|
I don't get that at all. Various kinds of "convenience syntax" is nothing new, you could argue that this whole PR is just that. You can achieve the same through YAML without it, but it might be a lot more to type. As long as they are possible to escape, and the information about how to use and escape them isn't too hard to find, I think it's fine. That said, I never read the documentation, so I don't know what the intention was for |
I asked copilot to further explain this for me: While standard relative paths (e.g.,
In short, while It seems that I need to provide further explanation and examples in the docs about this. |
That's what copilot/gemini says. Before someone says that ${OPENHAB_CONF} is more readable, let me state again that this is for convenience so I don't have to always type out that long thing everywhere in my files. You can use ${OPENHAB_CONF}/yaml in yours and we'll all be happy. Or in the AI speak:
|
This is a "two-way-street", because with relative paths, you can move things around as you want to, as long as you move them together. With fixed paths, that's not the case. I agree that "location shortcuts" are probably more useful here, most of the time, but that also depends on how one organizes the files. I've come to generally think of relative paths a "king" in most circumstances, because as long as you move the whole file structure, it's "self contained" and will work anywhere.
I don't think you need to explain this, it was just a suggestion as an alternative to having to type "the long version". It depends on the configuration, but if you have a limited number of folders and e.g. one folder for includes, keeping track of the relative paths isn't very complicated. The most flexible is to be able to do both, obviously. |
And indeed both are supported. |
This has been an eye-opener for me. Because I have been so deep in the development of this feature, I overlooked the fact that certain design choices weren't as self-evident to a first-time reader as they were to me. "Your" (collective) feedback is very valuable because it highlighted exactly where the narrative thread was missing; I now see that providing this explicit context is necessary to bridge that gap. |
|
@jimtng you love to copy paste copilot answers but I've found that almost always they are not correct at all or look superficially good but are not correct in detail. I'll address some arguments this time: 1. ${OPENHAB_CONF} already exists, but it’s not ergonomic
Valid argument, maybe an additional short alias like
Neither is the @ operator because it's an openhab specific implementation
It's a variable expansion so of course it blends in.
It was an explicit decision to make
Absolutely invalid since the documentation does not refer to the openHAB folder structure 2. openHAB has never been a “one way only” system
The yaml format is a file so there exactly two ways to define things: UI and files
What is
I can load openHAB configuration (services / addons) through files or UI but not through automation. 3. @/path is not intended to be a literal filesystem path
They actually are. After the variable expansion the include path is expected to be a valid path - relative or absolute. 4. The mental model is consistent with other ecosystems
These are aliases that are configurable by the user so they are something totally different.
The user home is not project specific but works system wide
This is just a relative import which would be the equivalent to
Bash 5. The goal is clarity and readability, not patching
That's literally the only case for this feature. Which also works well with relative paths.
Again - the Closing Thought
I find it very confusing, especially the @@ operator so for me the does not improve clarity but is yet another feature I have to learn.
Again not true. I tried to address some arguments and explain why I think they are void and I hope you see that these are not good arguments, too.
Just to play devils advocate:
I thought
I agree. Relative paths are great because they allow the creation of self sustained packages which can be moved around. As a suggestion:
You should really get more sleep. Openhab and these issues can definitely wait a little longer for your reply. |
|
An exaggerated summary: Just to be extra clear: this is meant as a humorous joke and not as an attack. I'm aware of the different nuances and subtle differences. |
|
It is clear we have different opinions and preferences. The way it is implemented right now,
If your directory is literally called
On this, I totally agree. BUT - I really want to get this done and move on to other things. |
I think that is mixing the cards a bit.
You can do it programmatically using scripts/rules, which is probably what is meant by "automation" here. Not a very convenient way to do it, but theoretically possible.
I thought it already worked with "the current subfolder", but in any case - if let's say that Generally on the use of LLMs, I'm a huge skeptic and generally don't like using them. The reason is that they are only good at making things appear correct superficially, and usually most of the details aren't correct with further scrutiny. I did a fair bit of image generation using SDXL, and it has the exact same problem - images can look amazing, but when you look at the details, it's almost always wrong in some way. I'm no LLM expert, but I'm skeptical to the claim that this is just about "tuning", I think it might be about something fundamental to the whole approach. As I understand it, the whole thing is built upon making something that "appears correct". That is the criteria that is used when iterating and making improvements to the result. The details are never in focus, and I'm not sure that a LLM can ever produce correct details. Therefore, I use it very sparingly. I have used it for writing some documentation in a couple of instances, but that is more "as an inspiration" for the general layout and what to include. I've then gone through the whole thing, correcting things, and I've usually ended up rewriting more or less every sentence. As such, I'm not sure that it does me any favors all in all, it might have been better if it just suggested which sections to include/what headlines to use, and I wrote the content myself. I therefore share the sentiment that we should limit the use of LLM output as "arguments", because what you really do then, is dump the "cognitive load" of dissecting all the half-truths on your opponent. Which is what you'd want to do in a war, but not something that is "nice" in a friendly discussion 😉 |
The There's no point in typing If you are NOT cross-referencing (e.g. including CONF/item/xxx from CONF/yaml/xxx) them, then it's a moot point to discuss this. I am not sure if both of you knew this:
If you ARE cross referencing files between different dirs, you can use ${OPENHAB_CONF} If you want something short for ${OPENHAB_CONF} you can always alias it to ${C} at the top of your files and use ${C} throughout that file. Lastly: if your main file is in CONF/yaml, |
Yes - I understood the feature and your idea.
That's what I was trying to say. You tailored a shortcut especially to your needs. Requiring the folder name after the @ makes it explicitly clear where the included file comes from. As for your special use case: |
|
I can make it customizable to please everyone: preprocessor:
path_aliases:
"@": !sub ${OPENHAB_CONF}I can even make a "global default" file that applies to all the files in that directory and subdirectories so you only define it once. The sky's the limit. |
That's not a good argument. It is also a search/replace to !sub ${OPENHAB_CONF} but I don't want to do that. |
I think it's very unexpected that
That was not an argument but rather intended as a friendly hint that your effort that you poured in your current config files is not lost and there is an easy way forward.
That's not what I asked and I definitely don't think that's something you should do.
How about allowing PS: |
I apologise. I would propose the following:
This would still give me the brevity that I want. |
|
How about we directly include the subfolder name?
That way we have the |
|
I've been wondering whether this should go into Core at all, or whether it should be implemented as an add-on. My thinking is that this is merely a secondary level of "convenience" and not in itself adding an actual core "feature". In other words, core can work fine without this PR. My concern about adding it to core is that it adds an extra level of complexity and also increase memory requirements (bloat) even if it's only a few KB/MB (I haven't measured). I'm thinking if added as an addon it can monitor CONF/yamlx (?) and generate the "compiled" final output to CONF/yaml, which core can then load as usual. It means that once it has compiled the output, its job is done and it can even be uninstalled without affecting core. I haven't tried to turn this into an addon but it will be next on my todo list. The only downside I can see is the multi-page documentation will need to live somewhere. For now it can go into
|
|
I would, if you go down that path, think about whether it would be better to create a "hook" in the YAML parser in core that you can plug into, or if it's better to just work with files as you suggest. I'm not sure what the consequences will be, how it will affect the actual use, but with a hook, one would open up for potentially also other "add-ons" that could pre-process the content in various ways. I'm not sure that there's actually a "need" for that, but it should probably be evaluated what is "the best arrangement". The argument as to whether it is "part of core or not" could be made about many things. The YAML parser itself could have been made as an add-on, the same could e.g. the DSL parsers. It's not always clear what belongs where IMO, but whatever "comes as standard" is more likely to see widespread use, I would think. |
|
As an add-on, I'm not sure which category it should fit into? System integration maybe? |
Yes, it's the IO/misc category, sometimes called "system integration". There's nothing else really, unless you consider "automation". But, "automation" in OH actually means rules/scripts - which it isn't. |
|
I've got a working add-on now. I'll close this PR and submit one to the add-on repo when it's ready. It will be called It will also be easier to test with a simple addon jar without requiring any changes to core. It monitors Thanks again for all the feedback! |
YAML Preprocessor, Includes, Packages, and Variable Substitution
Resolves: #3808, #4872, #5240
Related documentation PR: openhab/openhab-docs#2614
This PR introduces a YAML preprocessing layer to openHAB Core, enabling a more expressive and modular configuration syntax while preserving full backward compatibility with existing YAML files.
Summary of Changes
1. Exclude
*.inc.yaml,*.inc.yml, and generated debug output from model loadingThese files and directories are not meant to be parsed by
YamlModelRepositoryImpland are now excluded from model discovery:*.inc.yamland*.inc.yml— reusable fragments intended for!include, not standalone modelsCONF/<yaml|items|things|tags>/_generated/— contains fully resolved debug output produced only when thepreprocessor.generate_resolved_fileoption is enabled (default:false)2. Add a YAML Preprocessor
A new preprocessing stage is added before the YAML model repository loads the configuration.
The preprocessor expands the YAML file into a fully resolved document, supporting:
!include)This enables cleaner structure, reduced duplication, and reusable templates for Things, Items, and related metadata.
Processing Flow
Previous flow:
New flow:
The
YamlPreprocessorproduces a fully expanded YAML document that the existing model loader can consume without modification.Documentation
Full documentation for these features is available in the corresponding docs PR:
openhab/openhab-docs#2614
Preview:
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/
Individual pages (optional quick links):
Core Structure
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/
Variables
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/variables.html
Includes
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/include.html
Packages
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/packages.html
Anchors & Hidden Keys
https://deploy-preview-2614--openhab-docs-preview.netlify.app/docs/configuration/yaml/anchors.html