Skip to content

YAML things provider: create things even if binding is not yet installed #4753

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 11, 2025

Conversation

lolodomo
Copy link
Contributor

@lolodomo lolodomo commented Apr 26, 2025

Makes it consistent with managed thing provider.

Removes the OSGi reference between YamlThingProvider and YamlModelRepository and as consequence removes the circular reference.

Closes #4799

@lolodomo lolodomo requested a review from a team as a code owner April 26, 2025 09:23
@lolodomo lolodomo marked this pull request as draft April 27, 2025 22:55
@lolodomo
Copy link
Contributor Author

lolodomo commented Apr 27, 2025

I put this PR in draft mode because I encounter this strange exception:

11:41:56.337 [INFO ] [aml.internal.YamlModelRepositoryImpl] - Refreshing things from YAML model things\freebox.yaml
11:41:56.871 [WARN ] [.internal.handler.ApiConsumerHandler] - Error getting thing freeboxos:active-player:api:player properties: Permission missing: PLAYER
11:41:56.871 [WARN ] [.internal.handler.ApiConsumerHandler] - Error getting thing freeboxos:active-player:api:player properties: Permission missing: PLAYER
11:41:56.873 [ERROR] [core.thing.internal.ThingManagerImpl] - Exception occurred during notification about bridge status change on thing 'freeboxos:revolution:api:serveur': Empty key
java.lang.IllegalArgumentException: Empty key
	at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:107) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.LoginManager.openSession(LoginManager.java:124) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.openSession(FreeboxOsSession.java:101) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.execute(FreeboxOsSession.java:146) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.execute(FreeboxOsSession.java:157) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.RestManager.get(RestManager.java:74) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.RestManager.getSingle(RestManager.java:78) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.ConfigurableRest.getConfig(ConfigurableRest.java:43) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.ServerHandler.initializeProperties(ServerHandler.java:83) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.ApiConsumerHandler.initializeOnceBridgeOnline(ApiConsumerHandler.java:92) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.ApiConsumerHandler.bridgeStatusChanged(ApiConsumerHandler.java:162) ~[?:?]
	at org.openhab.core.thing.internal.ThingManagerImpl.lambda$11(ThingManagerImpl.java:823) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) [?:?]
11:41:56.901 [WARN ] [.internal.handler.ApiConsumerHandler] - Error getting thing freeboxos:player:api:player4k properties: Permission missing: PLAYER
11:41:56.905 [WARN ] [.internal.handler.ApiConsumerHandler] - Unable to retrieve Media Receivers: Unable to call RestManager constructor for org.openhab.binding.freeboxos.internal.api.rest.MediaReceiverManager
11:41:56.905 [WARN ] [.internal.handler.ApiConsumerHandler] - Unable to retrieve Media Receivers: Unable to call RestManager constructor for org.openhab.binding.freeboxos.internal.api.rest.MediaReceiverManager
11:41:56.907 [WARN ] [.internal.handler.ApiConsumerHandler] - Unable to retrieve Media Receivers: Unable to call RestManager constructor for org.openhab.binding.freeboxos.internal.api.rest.MediaReceiverManager
11:41:56.901 [ERROR] [core.thing.internal.ThingManagerImpl] - Exception occurred during notification about bridge status change on thing 'freeboxos:player:api:player4k': Empty key
java.lang.IllegalArgumentException: Empty key
	at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:107) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.LoginManager.openSession(LoginManager.java:124) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.openSession(FreeboxOsSession.java:101) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.execute(FreeboxOsSession.java:146) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.FreeboxOsSession.execute(FreeboxOsSession.java:157) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.RestManager.get(RestManager.java:74) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.RestManager.getSingle(RestManager.java:78) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.LanBrowserManager.getHost(LanBrowserManager.java:194) ~[?:?]
	at org.openhab.binding.freeboxos.internal.api.rest.LanBrowserManager.getHost(LanBrowserManager.java:216) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.HostHandler.getLanHost(HostHandler.java:116) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.HostHandler.initializeProperties(HostHandler.java:63) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.PlayerHandler.initializeProperties(PlayerHandler.java:65) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.ApiConsumerHandler.initializeOnceBridgeOnline(ApiConsumerHandler.java:92) ~[?:?]
	at org.openhab.binding.freeboxos.internal.handler.ApiConsumerHandler.bridgeStatusChanged(ApiConsumerHandler.java:162) ~[?:?]
	at org.openhab.core.thing.internal.ThingManagerImpl.lambda$11(ThingManagerImpl.java:823) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) [?:?]
11:41:56.923 [INFO ] [aml.internal.YamlModelRepositoryImpl] - Refreshing things from YAML model things\astro.yaml
11:41:57.059 [ERROR] [o.internal.handler.AstroThingHandler] - Can't update state for channel astro:moon:e2015c6f37:rise#event : org.openhab.binding.astro.internal.model.Range.getEvent()
java.lang.NoSuchMethodException: org.openhab.binding.astro.internal.model.Range.getEvent()
	at java.lang.Class.getMethod(Class.java:2395) ~[?:?]
	at org.openhab.binding.astro.internal.util.PropertyUtils.getPropertyValue(PropertyUtils.java:92) ~[?:?]
	at org.openhab.binding.astro.internal.util.PropertyUtils.getPropertyValue(PropertyUtils.java:96) ~[?:?]
	at org.openhab.binding.astro.internal.util.PropertyUtils.getPropertyValue(PropertyUtils.java:83) ~[?:?]
	at org.openhab.binding.astro.internal.util.PropertyUtils.getState(PropertyUtils.java:57) ~[?:?]
	at org.openhab.binding.astro.internal.handler.AstroThingHandler.publishChannelIfLinked(AstroThingHandler.java:179) ~[?:?]
	at org.openhab.binding.astro.internal.handler.AstroThingHandler.channelLinked(AstroThingHandler.java:248) ~[?:?]
	at org.openhab.core.thing.internal.ChannelLinkNotifier.lambda$3(ChannelLinkNotifier.java:72) ~[?:?]
	at org.openhab.core.thing.internal.ChannelLinkNotifier.call(ChannelLinkNotifier.java:96) ~[?:?]
	at org.openhab.core.thing.internal.ChannelLinkNotifier.added(ChannelLinkNotifier.java:72) ~[?:?]
	at org.openhab.core.thing.internal.ChannelLinkNotifier.updated(ChannelLinkNotifier.java:88) ~[?:?]
	at org.openhab.core.thing.internal.ChannelLinkNotifier.updated(ChannelLinkNotifier.java:1) ~[?:?]
	at org.openhab.core.common.registry.AbstractRegistry.notifyListeners(AbstractRegistry.java:398) ~[?:?]
	at org.openhab.core.common.registry.AbstractRegistry.notifyListenersAboutUpdatedElement(AbstractRegistry.java:419) ~[?:?]
	at org.openhab.core.thing.link.ItemChannelLinkRegistry.notifyListenersAboutUpdatedElement(ItemChannelLinkRegistry.java:223) ~[?:?]
	at org.openhab.core.thing.link.ItemChannelLinkRegistry.notifyListenersAboutUpdatedElement(ItemChannelLinkRegistry.java:1) ~[?:?]
	at org.openhab.core.common.registry.AbstractRegistry.updated(AbstractRegistry.java:325) ~[?:?]
	at org.openhab.core.common.registry.AbstractRegistry.updated(AbstractRegistry.java:1) ~[?:?]
	at org.openhab.core.common.registry.AbstractProvider.notifyListeners(AbstractProvider.java:66) ~[?:?]
	at org.openhab.core.common.registry.AbstractProvider.notifyListenersAboutUpdatedElement(AbstractProvider.java:91) ~[?:?]
	at org.openhab.core.model.yaml.internal.items.YamlChannelLinkProvider.createItemChannelLink(YamlChannelLinkProvider.java:94) ~[?:?]
	at org.openhab.core.model.yaml.internal.items.YamlItemProvider.lambda$17(YamlItemProvider.java:299) ~[?:?]
	at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:945) ~[?:?]
	at org.openhab.core.model.yaml.internal.items.YamlItemProvider.processChannelLinks(YamlItemProvider.java:297) ~[?:?]
	at org.openhab.core.model.yaml.internal.items.YamlItemProvider.lambda$6(YamlItemProvider.java:170) ~[?:?]
	at java.util.HashMap$EntrySet.forEach(HashMap.java:1134) ~[?:?]
	at org.openhab.core.model.yaml.internal.items.YamlItemProvider.updatedModel(YamlItemProvider.java:154) ~[?:?]
	at org.openhab.core.model.yaml.internal.YamlModelRepositoryImpl.lambda$29(YamlModelRepositoryImpl.java:585) ~[?:?]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.openhab.core.model.yaml.internal.YamlModelRepositoryImpl.lambda$28(YamlModelRepositoryImpl.java:580) ~[?:?]
	at java.util.concurrent.ConcurrentHashMap$EntrySetView.forEach(ConcurrentHashMap.java:4875) ~[?:?]
	at org.openhab.core.model.yaml.internal.YamlModelRepositoryImpl.lambda$27(YamlModelRepositoryImpl.java:573) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) ~[?:?]
	at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3612) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at org.openhab.core.model.yaml.internal.YamlModelRepositoryImpl.handleRefreshRequests(YamlModelRepositoryImpl.java:569) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) [?:?]

while I don't have it in main branch!
I don't know if it has a direct link with my change or not.

@lolodomo
Copy link
Contributor Author

The problem is because things are updated while some of them have their handler already initializing.
The problem appears now more than before because the new refresh job delays the update of few seconds, letting the time to things to start being initialized.
It is probably necessary to first request the thing manager to stop thing handlers before updating a thing.

I should also study if there is a way to update only models that are really impacted. If what triggers the need of a refresh is the start of a new binding, ideally only models containing things for this binding should be refreshed.

@jimtng
Copy link
Contributor

jimtng commented Apr 28, 2025

I've been meaning to ask you about this.
Currently you are performing checks to see if a ThingHandler exists before you'd process / register a Thing from yaml. This is also done the same way in Generic Item Provider (DSL).

As a result of this, if you added a Thing without a loaded Binding, this Thing will not appear in the system at all. You cannot see it in the UI either of course.

Only when you've installed the binding will you then register the Thing and see it in the system (Thing Registry).

This behaviour is not the same as Managed Things. Managed Things will register the Thing regardless of whether the Binding / Thing Handler exists, so that you can see it in the ThingRegistry and also on the UI.

This gives users a chance to know that they are missing the binding and the UI provides a handy link to install the binding.
See: openhab/openhab-webui#3003

IMO this is a nicer behaviour than "hiding" unknown Things like it is currently done in YAML / DSL. It will also probably solve your circular reference issue and significantly simplify the code.

@lolodomo
Copy link
Contributor Author

lolodomo commented Apr 28, 2025

IMO this is a nicer behaviour than "hiding" unknown Things like it is currently done in YAML / DSL. It will also probably solve your circular reference issue and significantly simplify the code.

Not at all unfortunately because a refresh will be needed in any case when the new thing handler is started, something that can happen at any time after the server is started (as soon as you install a new binding).

I would say that it is another point I have myself mentioned earlier but with no direct link with the current problem.

@lolodomo
Copy link
Contributor Author

lolodomo commented Apr 29, 2025

I have an idea how to avoid reloading all YAML files containing things when a binding is added. The idea is to reload only the things attached to this binding, whatever the YAML file it belongs to.
I will add the ability for a YAML provider to request the reloading of elements with an id matching a certain pattern.
An example of use with the thing provider is that when the handler factory for the Hue binding is injected, the thing provider will request the YAML model repository to refresh all models having things with an id matching "hue:.*". Like that, all things not matching this pattern (other bindings) will be ignored and not updated.
This will solve the current problem.

@Nadahar
Copy link
Contributor

Nadahar commented Apr 29, 2025

I don't know if this is a desirable approach here, I'd just like to add that in general, it might not be beneficial to make complex solutions to avoid circular references.

The alternative is to not make the reference optional, using ReferenceCardinality, ReferencePolicyOption and ReferencePolicy as needed.

The only caveat is that that means that the implementation must handle that the reference might be null, and that it might be registered and unregistered at any time. But, in reality, that might not be much of an issue to handle, it depends on the situation. If done this way, circular references aren't a problem.

@lolodomo
Copy link
Contributor Author

The only caveat is that that means that the implementation must handle that the reference might be null, and that it might be registered and unregistered at any time. But, in reality, that might not be much of an issue to handle, it depends on the situation. If done this way, circular references aren't a problem.

I will think to that. But the problem is more than the circular reference, it is what models to refresh when a new binding is loaded. I converged to a solution that is not bad I believe, more simple than what I described in my previous message.
And to make my life more complex, there is probably a bug in the freeboxos binding leading to exceptions being logged when things are started before the bridge is ready (session not yet opened and token not yet available).

@Nadahar
Copy link
Contributor

Nadahar commented Apr 29, 2025

I will think to that.

As I said, it was just a general suggestion for such situations, if and when it is a better solution. I didn't study the case here so that I could understand what is and isn't a good approach 😉

@lolodomo
Copy link
Contributor Author

I think you're right, I will restore the reference but will make it optional.

I will also add code to finally refresh only what is needed.

@lolodomo lolodomo force-pushed the fix_circular_ref branch 2 times, most recently from f9877d8 to cec746d Compare May 1, 2025 10:39
@lolodomo
Copy link
Contributor Author

lolodomo commented May 1, 2025

Ok, I made the reference optional and enhanced the system to refresh only needed things.

@lolodomo lolodomo marked this pull request as ready for review May 1, 2025 10:40
@lolodomo lolodomo marked this pull request as draft May 1, 2025 18:46
@lolodomo lolodomo force-pushed the fix_circular_ref branch from cec746d to 31b1905 Compare May 1, 2025 20:04
@lolodomo lolodomo marked this pull request as ready for review May 1, 2025 20:04
@lolodomo
Copy link
Contributor Author

lolodomo commented May 1, 2025

@Nadahar : do you know if injection of references can occur in parallel (in different threads) or if they are done in sequence ?

@Nadahar
Copy link
Contributor

Nadahar commented May 1, 2025

do you know if injection of references can occur in parallel (in different threads) or if they are done in sequence ?

I don't know. This is normally only done during startup/shutdown, or if somebody stops/starts/restarts a bundle while the system is running. I haven't seen any thread safety measures when I've looked at other code that deals with (de)registration in OH, but that doesn't mean that it's safe.

The information about the topic seems to be somewhat sparse (which usually means it's unsafe), but I can find several indications that there is no protection here:
https://docs.osgi.org/specification/osgi.core/7.0.0/framework.servicehooks.html#d0e46152
https://www.aqute.biz/appnotes/concurrency.html
https://stackoverflow.com/questions/20306581/osgi-thread-safety-cautions

My assumption has been that it is indeed unsafe, that is, that these methods are called by various threads, but that since it's only during startup and shutdown this happens, exposure is "limited" so you usually "get away with it". I do remember having had quite a "random factor" getting OH to start in the past, especially back in OH2, and I've considered it likely that this has been threading issues all along (since those are the ones that usually are quite unpredictable in their ways).

Not all classes gets their references set in the constructor, so even in cases where references are mandatory, there should be a small gap where the object has been created but the reference is invalid. Maybe OSGi has some kind of protection against this case, in that the object "isn't made available" until madatory references has been set for example, but I don't know.

Have you experienced some issues or are you just being cautious? IMO it never hurts to assume that things can be called concurrently, unless you know that it's not the case.

@lolodomo
Copy link
Contributor Author

lolodomo commented May 1, 2025

Ok, thank you for your answer.
I put again this PR in draft, I would like to try another solution avoiding to call the model repository.

@lolodomo lolodomo marked this pull request as draft May 1, 2025 22:38
@lolodomo
Copy link
Contributor Author

lolodomo commented May 2, 2025

IMO this is a nicer behaviour than "hiding" unknown Things like it is currently done in YAML / DSL. It will also probably solve your circular reference issue and significantly simplify the code.

Not at all unfortunately because a refresh will be needed in any case when the new thing handler is started, something that can happen at any time after the server is started (as soon as you install a new binding).

I would say that it is another point I have myself mentioned earlier but with no direct link with the current problem.

Thinking about it more, it could simplify the problem and the implementation if the things were created even without a thing handler. I will go in that direction.

@lolodomo lolodomo force-pushed the fix_circular_ref branch from 31b1905 to afdbef1 Compare May 3, 2025 09:17
@lolodomo lolodomo changed the title YAML things: Fix way to request models refresh to avoid a circular ref YAML things provider: create things even if binding is not yet installed May 3, 2025
@lolodomo lolodomo marked this pull request as ready for review May 3, 2025 09:18
Makes it consistent with managed thing provider.

Removes the OSGi reference between YamlThingProvider and YamlModelRepository and as consequence removes the circular reference.

Signed-off-by: Laurent Garnier <[email protected]>
@lolodomo lolodomo force-pushed the fix_circular_ref branch from afdbef1 to f0c6780 Compare May 3, 2025 09:32
@lolodomo
Copy link
Contributor Author

lolodomo commented May 3, 2025

I finally changed the YAML thing provider to create the things even when the binding is not yet installed. It makes it consistent with the managed thing provider. And I removed the reference creating a circular reference.
I am satisfied with the new code.

@lolodomo
Copy link
Contributor Author

lolodomo commented May 4, 2025

@openhab/core-maintainers : is it possible please to have a review of this PR ?
I would like to have next snapshot running without the circular reference.

Copy link
Member

@wborn wborn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the improvements/fixes!

@wborn wborn merged commit 3df3375 into openhab:main May 11, 2025
4 checks passed
@wborn wborn added this to the 5.0 milestone May 11, 2025
@wborn wborn added the enhancement An enhancement or new feature of the Core label May 11, 2025
@lolodomo lolodomo deleted the fix_circular_ref branch May 11, 2025 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature of the Core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ERROR Circular reference detected trying to get service {...ThingProvider, ...YamlThingProvider, ...YamlModelListener}
4 participants