Bundles not starting waiting forever

[mstormi]

I have some bundles that just don't start, resulting in symptoms like an empty homepage (or actually only the UI wizard is offered to run). And I'm seeing this on several OH instances I operate.
It does not happen reproducibly but once it occurs this seems to be a stubborn persistent problem.
Restarts don't help, even if I clean the cache inbetween. Running 4.3.2, I have not seen this in versions earlier than 4.3.
Actually I upgraded one box from Linux pkg 4.3.0-1 (on x86 debian) which did not have the problem to 4.3.2-1 and it showed up.

Any hint how to diagnose what those "missing references $00X" are ?

openhab> bundle:list|grep -v Acti
START LEVEL 100 , List Threshold: 50
 ID x State   x Lvl x Version               x Name
qqqqnqqqqqqqqqnqqqqqnqqqqqqqqqqqqqqqqqqqqqqqnqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
158 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: Marketplace Add-on Services
188 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: Sitemap REST Interface
192 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: UI REST Interface
286 x Waiting x  80 x 4.3.2                 x openHAB Add-ons :: Bundles :: Persistence Service :: RRD4j
291 x Waiting x  80 x 4.3.2                 x openHAB UI :: Bundles :: Basic UI
openhab> bundle:diag 158
openHAB Core :: Bundles :: Marketplace Add-on Services (158)
------------------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.core.addon.marketplace.internal.community.CommunityMarketplaceAddonService (460)
org.openhab.core.addon.marketplace.internal.community.CommunityBlockLibaryAddonHandler (458)
  missing references: $000
org.openhab.core.addon.marketplace.internal.automation.MarketplaceRuleTemplateProvider (457)
org.openhab.core.addon.marketplace.internal.community.CommunityBundleAddonHandler (459)
org.openhab.core.addon.marketplace.internal.community.CommunityRuleTemplateAddonHandler (461)
org.openhab.core.addon.marketplace.internal.json.JsonAddonService (464)
org.openhab.core.addon.marketplace.internal.community.CommunityTransformationAddonHandler (462)
org.openhab.core.addon.marketplace.internal.community.CommunityUIWidgetAddonHandler (463)
  missing references: $000


openhab> bundle:diag 188
openHAB Core :: Bundles :: Sitemap REST Interface (188)
-------------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.core.io.rest.sitemap.internal.SitemapResource (197)
  missing references: $000, $003
org.openhab.core.io.rest.sitemap.SitemapSubscriptionService (196)
  missing references: $001

openhab> bundle:diag 192
openHAB Core :: Bundles :: UI REST Interface (192)
--------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.core.io.rest.ui.internal.UIResource (203)
  missing references: $000, $001


openhab> bundle:diag 286
openHAB Add-ons :: Bundles :: Persistence Service :: RRD4j (286)
----------------------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.persistence.rrd4j.internal.console.RRD4jCommandExtension (494)
org.openhab.persistence.rrd4j.internal.charts.RRD4jChartServlet (493)
  missing references: $001
org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService (492)


openhab> bundle:diag 291
openHAB UI :: Bundles :: Basic UI (291)
---------------------------------------
Status: Waiting
Declarative Services
org.openhab.ui.basic.internal.servlet.CmdServlet (483)
org.openhab.ui.basic.internal.render.SetpointRenderer (477)
  missing references: $002
org.openhab.ui.basic.internal.render.GroupRenderer (471)
  missing references: $002
org.openhab.ui.basic.internal.render.ChartRenderer (467)
  missing references: $002
org.openhab.ui.basic.internal.render.VideoRenderer (481)
  missing references: $002
org.openhab.ui.basic.internal.render.SliderRenderer (478)
  missing references: $002
org.openhab.ui.basic.internal.render.InputRenderer (473)
  missing references: $002
org.openhab.ui.basic.internal.servlet.ManifestServlet (484)
  missing references: $000
org.openhab.ui.basic.internal.render.ButtongridRenderer (466)
  missing references: $002
org.openhab.ui.basic.internal.render.ImageRenderer (472)
  missing references: $002
org.openhab.ui.basic.internal.render.TextRenderer (480)
  missing references: $002
org.openhab.ui.basic.internal.render.SwitchRenderer (479)
  missing references: $002
org.openhab.ui.basic.internal.render.ColorpickerRenderer (468)
  missing references: $002
org.openhab.ui.basic.internal.render.FrameRenderer (470)
  missing references: $002
org.openhab.ui.basic.internal.render.PageRenderer (475)
  missing references: $002
org.openhab.ui.basic.internal.render.SelectionRenderer (476)
  missing references: $002
org.openhab.ui.basic.internal.render.WebviewRenderer (482)
  missing references: $002
org.openhab.ui.basic.internal.BasicUITile (465)
org.openhab.ui.basic.internal.render.ColortemppickerRenderer (469)
  missing references: $002
org.openhab.ui.basic.internal.servlet.WebAppServlet (485)
  missing references: $000, $001
org.openhab.ui.basic.internal.render.MapviewRenderer (474)
  missing references: $002

[cdjackson]

Restarts don't help, even if I clean the cache in between.

Did you get it to restart at all? When you say "restart" is that an OH restart or a PC/RPi restart?

I had a similar thing last week. The system stopped and I restarted OH and many bundles didn't start making the system unusable. I couldn't get OH to restart properly even after a couple of tries, but a complete PC restart fixed it. I'm running on an i86 NUC type box - it's a pretty basic debian install with the absolute minimum running other than OH.

I do have logs of my issue - of course there's nothing to say this is the same issue as yours. I did see out of memory exception when this first happened - the system had been running fine with not restarts for around 6 weeks when this happened and wasn't doing anything particularly "stressful" at the time this occurred.

After the OH restart I see the following exception -:

org.osgi.framework.ServiceException: Exception in org.apache.felix.scr.impl.manager.SingleComponentManager.getService()
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:230) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:118) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:48) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:547) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:534) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:660) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:88) ~[?:?]
	at org.apache.felix.scr.impl.inject.methods.BindMethod.getServiceObject(BindMethod.java:675) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2612) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.prebind(DependencyManager.java:434) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1830) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1060) ~[?:?]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:955) ~[?:?]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:920) ~[?:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:220) ~[org.eclipse.osgi-3.18.0.jar:?]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:217) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:118) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:48) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:547) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:534) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:660) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:88) ~[?:?]
	at org.apache.felix.scr.impl.inject.methods.BindMethod.getServiceObject(BindMethod.java:675) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2612) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:2078) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager.invokeBindMethod(DependencyManager.java:2061) ~[?:?]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeBindMethod(SingleComponentManager.java:443) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:336) ~[?:?]
	at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:304) ~[?:?]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232) ~[?:?]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152) ~[?:?]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959) ~[?:?]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895) ~[?:?]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184) ~[?:?]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116) ~[?:?]
	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:123) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:961) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:937) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:874) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:141) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:262) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:500) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915) ~[?:?]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:437) ~[?:?]
	at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:671) ~[?:?]
	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:310) ~[?:?]
	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:593) ~[?:?]
	at org.apache.felix.scr.impl.Activator.access$200(Activator.java:74) ~[?:?]
	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:460) ~[?:?]
	at org.apache.felix.scr.impl.AbstractExtender.createExtension(AbstractExtender.java:196) ~[?:?]
	at org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:169) ~[?:?]
	at org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:49) ~[?:?]
	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:488) ~[osgi.core-8.0.0.jar:?]
	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:420) ~[osgi.core-8.0.0.jar:?]
	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232) ~[osgi.core-8.0.0.jar:?]
	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:450) ~[osgi.core-8.0.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:949) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:229) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:138) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:130) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:217) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.container.Module.publishEvent(Module.java:499) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.container.Module.start(Module.java:486) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:445) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:464) ~[org.eclipse.osgi-3.18.0.jar:?]
	at org.apache.karaf.features.internal.service.BundleInstallSupportImpl.startBundle(BundleInstallSupportImpl.java:165) ~[?:?]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:1160) ~[?:?]
	at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:1041) ~[?:?]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1069) ~[?:?]
	at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:1004) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.lang.IllegalStateException: Could not obtain lock
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainLock(AbstractComponentManager.java:242) ~[?:?]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.obtainStateLock(AbstractComponentManager.java:289) ~[?:?]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:908) ~[?:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:220) ~[org.eclipse.osgi-3.18.0.jar:?]
	at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:217) ~[org.eclipse.osgi-3.18.0.jar:?]
	... 83 more

I didn't clear the cache - I just restarted the PC and it resolved the issue. This exception appears to point to a lock issue - possibly a lockfile (??). I'm not sure why a PC restart would be any different to an OH restart though in this respect.

Again - not sure if this is exactly the same issue as @mstormi is seeing, but the symptoms sound similar.

[mstormi]

It was only OH restarts. I'm on an RPi 3, but I have seen this on some other Raspis, too, both 32 and 64 openHABian.
I'd be very surprised if it was OS related. Rebooting the box should not make a difference, should it.
(well unless maybe this is a race condition thing and we accidentially-luckily get a different startup order)

I have gotten it to fully start after I juggled forth and back with some with startup level definitions.

The 'missing reference' message to me indicate it's related to Karaf dependencies and not to file locking, I have not seen any message on that like you did so I'm inclined to believe we're not seeing the same issue.

[cdjackson]

Rebooting the box should not make a difference, should it.

I agree in principle, but as I said, I restarted OH twice and that didn't resolve it, so I think I did a soft restart (ie via the OS shutdown command), but I might have done a power cycle - I forget now. Either way, that restored the system.

I have not seen any message on that like you did so I'm inclined to believe we're not seeing the same issue.

Ok, useful to know. I guess the bottom line is it's a similar outcome, so I'm interested to hear the answer to your question on how to diagnose. If it happens again I'll dig a bit deeper...

[mstormi]

@splatch I read on some occasion in the forum that you know how to translate those missing references.

Could you please have a look and explain to us how to decode and to debug this dependency mess?

[J-N-K]

IIRC the references have the same order as the parameters of the constructor of the service that failed, starting with 0 for the first one.

[mstormi]

As it complains about stuff depending on "marketplace", how do I find out which bundle I'm actually from the marketplace and not from openhab-addons package i.e. the co-installed 4.3.2 .kar file?

[J-N-K]

@mstormi I guess the ItemUIRegistry is missing, that's reference $002 for the renderer classes in Basic UI and reference $001 in the RRD4j chart servlet. Can you run diag on org.openhab.core.ui?

[mstormi]

I have another instance of the problem.
After the machine rebooted, all bundles were active but the rule engine wasn't running
So I did bundle:restart org.openhab.core.automation. That now resulted in the same bindings as in post #1 to be stuck in 'Waiting'

I tried to diag here but don't get anything meaningful out.

openhab> diag org.openhab.core.ui
openhab> bundle:list|grep -i  ui
 40 x Active  x  80 x 7.0.0                 x guice
192 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: UI REST Interface
223 x Active  x  80 x 4.3.2                 x openHAB Core :: Bundles :: UI
224 x Active  x  80 x 4.3.2                 x openHAB Core :: Bundles :: UI Icon Support
226 x Active  x  80 x 4.3.2                 x openHAB UI :: Bundles :: Main UI
227 x Active  x  80 x 4.3.2                 x openHAB UI :: Bundles :: Icon Set :: Classic
291 x Waiting x  80 x 4.3.2                 x openHAB UI :: Bundles :: Basic UI
openhab> bundle:diag 223
openhab>


openhab> bundle:list|grep -v Act
START LEVEL 100 , List Threshold: 50
 ID x State   x Lvl x Version               x Name
qqqqnqqqqqqqqqnqqqqqnqqqqqqqqqqqqqqqqqqqqqqqnqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
158 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: Marketplace Add-on Services
188 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: Sitemap REST Interface
192 x Waiting x  80 x 4.3.2                 x openHAB Core :: Bundles :: UI REST Interface
286 x Waiting x  80 x 4.3.2                 x openHAB Add-ons :: Bundles :: Persistence Service :: RRD4j
291 x Waiting x  80 x 4.3.2                 x openHAB UI :: Bundles :: Basic UI

[mstormi]

how can the marketplace bundle be 'waiting' when it is not installed?

openhab> bundle:list|grep -v Act
START LEVEL 100 , List Threshold: 50
 ID x State    x Lvl x Version               x Name
158 x Resolved x  80 x 4.3.3                 x openHAB Core :: Bundles :: Marketplace Add-on Services
188 x Waiting  x  80 x 4.3.3                 x openHAB Core :: Bundles :: Sitemap REST Interface
192 x Waiting  x  80 x 4.3.3                 x openHAB Core :: Bundles :: UI REST Interface
286 x Waiting  x  80 x 4.3.3                 x openHAB Add-ons :: Bundles :: Persistence Service :: RRD4j
299 x Resolved x  80 x 4.3.3                 x openHAB UI :: Bundles :: Basic UI
openhab> feature:uninstall openhab-ui-basic
openhab> feature:list |grep -i market
openhab-core-addon-marketplace                    x 4.3.3            x          x Started     x distro-4.3.3             x
openhab> feature:uninstall openhab-core-addon-marketplace
Error executing command: Feature named 'openhab-core-addon-marketplace/0' is not installed
openhab> diag 158
openHAB Core :: Bundles :: Marketplace Add-on Services (158)
------------------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.core.addon.marketplace.internal.community.CommunityMarketplaceAddonService (434)
org.openhab.core.addon.marketplace.internal.community.CommunityBlockLibaryAddonHandler (432)
  missing references: $000
org.openhab.core.addon.marketplace.internal.automation.MarketplaceRuleTemplateProvider (431)
org.openhab.core.addon.marketplace.internal.community.CommunityBundleAddonHandler (433)
org.openhab.core.addon.marketplace.internal.community.CommunityRuleTemplateAddonHandler (435)
org.openhab.core.addon.marketplace.internal.json.JsonAddonService (438)
org.openhab.core.addon.marketplace.internal.community.CommunityTransformationAddonHandler (436)
org.openhab.core.addon.marketplace.internal.community.CommunityUIWidgetAddonHandler (437)
  missing references: $000

[mstormi]

I guess the ItemUIRegistry is missing, that's reference $002 for the renderer classes in Basic UI and reference $001 in the RRD4j chart servlet. Can you run diag on org.openhab.core.ui?

No output on that, so I think that it was not.

openhab> diag org.openhab.core.ui
openhab>

Could some core dev in the know please give me guidance how to debug that ?
ping @wborn

openhab> diag org.openhab.core.ui
openhab> diag org.openhab.persistence.rrd4j
openHAB Add-ons :: Bundles :: Persistence Service :: RRD4j (286)
----------------------------------------------------------------
Status: Waiting
Declarative Services
org.openhab.persistence.rrd4j.internal.console.RRD4jCommandExtension (305)
org.openhab.persistence.rrd4j.internal.charts.RRD4jChartServlet (304)
  missing references: $001
org.openhab.persistence.rrd4j.internal.RRD4jPersistenceService (303)
openhab> diag org.openhab.ui.basic
openHAB UI :: Bundles :: Basic UI (303)
---------------------------------------
Status: Waiting
Declarative Services
org.openhab.ui.basic.internal.render.SetpointRenderer (318)
  missing references: $002
org.openhab.ui.basic.internal.render.GroupRenderer (312)
  missing references: $002
org.openhab.ui.basic.internal.render.ChartRenderer (308)
  missing references: $002
org.openhab.ui.basic.internal.render.VideoRenderer (322)
  missing references: $002
org.openhab.ui.basic.internal.servlet.ManifestServlet (325)
  missing references: $000
org.openhab.ui.basic.internal.render.ButtongridRenderer (307)
  missing references: $002
org.openhab.ui.basic.internal.render.SelectionRenderer (317)
  missing references: $002
org.openhab.ui.basic.internal.render.FrameRenderer (311)
  missing references: $002
org.openhab.ui.basic.internal.render.ImageRenderer (313)
  missing references: $002
org.openhab.ui.basic.internal.render.TextRenderer (321)
  missing references: $002
org.openhab.ui.basic.internal.render.SwitchRenderer (320)
  missing references: $002
org.openhab.ui.basic.internal.render.PageRenderer (316)
  missing references: $002
org.openhab.ui.basic.internal.render.WebviewRenderer (323)
  missing references: $002
org.openhab.ui.basic.internal.servlet.CmdServlet (324)
org.openhab.ui.basic.internal.BasicUITile (306)
org.openhab.ui.basic.internal.render.ColorpickerRenderer (309)
  missing references: $002
org.openhab.ui.basic.internal.render.ColortemppickerRenderer (310)
  missing references: $002
org.openhab.ui.basic.internal.servlet.WebAppServlet (326)
  missing references: $000, $001
org.openhab.ui.basic.internal.render.InputRenderer (314)
  missing references: $002
org.openhab.ui.basic.internal.render.SliderRenderer (319)
  missing references: $002
org.openhab.ui.basic.internal.render.MapviewRenderer (315)
  missing references: $002

openhab>

[mstormi]

I noticed Karaf 4.4.7 fixes "avoid race condition at startup between FileInstall and ConfigAdmin" and would love to test if that's related to the issue here.

@holgerfriedrich was Karaf 4.4.7 backported into OH 4.3.x? I did not see that in the release notes.
Coincidence or not, I saw first occurences of this bundle issue starting around 4.2, older instances appear to be unaffected as far as I can tell. Karaf was upgraded to 4.4.6 in May, around 4.2 release time.

Plus I guess backporting is well worth it (regardless of this issue) given we need a stable release as not everybody will happily jump onto OH5 train quick,

[holgerfriedrich]

@mstormi No, Karaf changes did not get backported. Karaf upgrades touch a lot of dependencies and require quite an amount of testing. It may break plugins compiled before the merge, due to the changing dependencies. For me this is something that goes far beyond the scope of "bugfixes" we typically port back to the last release.

Maybe you want to start a discussion about the future of the 4.3.x branch in the light of OH5 breaking changes outside if this issue.

Back to this issue: this is something I cannot reproduce. I run two productive 4.3.x instances (w/o using marketplace) and do not encounter the problems described above. I have the core and addons package installed and never clear the caches manually, just let apt do the upgrades.

[mstormi]

Maybe you want to start a discussion about the future of the 4.3.x branch in the light of OH5 breaking changes outside if this issue.

Err, no. Patching 4.x up to latest Karaf has nothing to do with "the future" or any breaking changes.
It's a matter of sanity and quality management i.e. proper maintenance of our only product release.
Please seriously reconsider backporting your PR (even if it should turn out to be unrelated to this issue).

Back to this issue: this is something I cannot reproduce. I run two productive 4.3.x instances

That's not helpful either. OH needs to work on several tens of thousands of instances, and already I myself have more than you that suffer from this problem. And it's not some luxury exotic issue thing. To the best of my knowledge, I don't use anything unusual - actually I am even not aware of using anything from the marketplace - but the issue affects main components like UI and API that everybody needs.

Thing is, I have no idea how to diagnose it.