openHAB sometimes doesn't start properly after upgrades

[mstormi]

I noticed on various occasions that openHAB didn't start properly after a package upgrade.

On every of these, I had to manually clean the cache to make it finally start (or to proceed after start, actually - it starts in fact but keeps hanging).

@BClark09 I'd prefer if we sort this out before OH 4.3 release Dec 15.

[BClark09]

When you mention Karaf cache, do you mean cache and tmp directories inside ${OPENHAB_USERDIR}? or just the .karaf dir?

If cache and tmp, these are indeed removed in the distro's update scripts without condition. (Note that Linuxpkg does this in a similar way). Have you tried completing an upgrade without openHAB running and then looking to see if the files still exist?

If .karaf, what is stored here? I thought it was just the console history?

[mstormi]

did not manually remove anything. Used `openhab-cli clean-cache`, can you compare what that does to those dirs you named? Am 14. November 2024 19:40:44 MEZ schrieb Ben Clark ***@***.***>:

…

When you mention Karaf cache, do you mean `cache` and `tmp` directories inside `${OPENHAB_USERDIR}`? or just the `karaf` dir? If `cache` and `tmp`, these are indeed removed in the [distro's update scripts](https://github.com/openhab/openhab-distro/blob/main/distributions/openhab/src/main/resources/bin/update#L321C1-L324C43) without condition. (Note that [Linuxpkg does this in a similar way](https://github.com/openhab/openhab-linuxpkg/blob/main/resources/control-runtime/preinst#L84)). Have you tried completing an upgrade without openHAB running and then looking to see if the files still exist? If `.karaf`, what is stored here? I thought it was just the console history?

[mike-bike]

Not sure if it helps. I do see sometimes start-up issues (Bindings not all initializing, etc) on the first start after an upgrade. Most of the time I do a stop, clean-cache, restart. However, I was also able to get a clean start after a stop and start without cache clean. Maybe there are sometimes timing issues in the first start after an upgrade. Unfortunately, this seems not to be reproducible.

[andrewfg]

Most of the time I do a stop, clean-cache, restart

Me too. This should be automated!

[BClark09]

I think we need to understand what the problem is and where it's coming from.

From what I understand, there seems to be a problem on first launch of an upgrade for some people and this issue is resolved by cleaning the cache. That does not mean that the cache isn't cleaned on an upgrade:

ben@smarthome:~ $ sudo systemctl stop openhab
ben@smarthome:~ $ du -sh ${OPENHAB_USERDATA}/*
4.0K    /var/lib/openhab/backups
140M    /var/lib/openhab/cache
292K    /var/lib/openhab/config
304K    /var/lib/openhab/etc
3.7M    /var/lib/openhab/jsondb
81M     /var/lib/openhab/persistence
162M    /var/lib/openhab/piper
8.0K    /var/lib/openhab/secrets
90M     /var/lib/openhab/tmp
4.0K    /var/lib/openhab/zigbee
144K    /var/lib/openhab/zwave
ben@smarthome:~ $ sudo apt install openhab
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  openhab
1 upgraded, 0 newly installed, 0 to remove and 59 not upgraded.
Need to get 111 MB of archives.

[...]

[openHAB] JSON Database upgrade completed.
[openHAB] openHAB was not running so will not start after upgrade.
[openHAB] Please use the command:
            sudo /bin/systemctl start openhab.service

ben@smarthome:~ $ du -sh ${OPENHAB_USERDATA}/*
4.0K    /var/lib/openhab/backups
288K    /var/lib/openhab/config
304K    /var/lib/openhab/etc
3.7M    /var/lib/openhab/jsondb
81M     /var/lib/openhab/persistence
162M    /var/lib/openhab/piper
8.0K    /var/lib/openhab/secrets
4.0K    /var/lib/openhab/tmp
4.0K    /var/lib/openhab/zigbee
144K    /var/lib/openhab/zwave

You can see from above that the cache and tmp directories have indeed been cleaned during the upgrade process. This will be the same if you're using the distributions update scripts.

[andrewfg]

I suppose that if the cached files are in memory then deleting the disk images does not automatically clear the memory image. Also probably a service restart won't clear the memory images either. I suppose that only a full shutdown and reboot would do that.

[BClark09]

The openHAB service is shutdown during an upgrade (or in the case of the distribution scripts, the update wont run at all until openHAB is stopped), so there should be no instance in memory.

[mstormi]

You can see from above that the cache and tmp directories have indeed been cleaned during the upgrade process. This will be the same if you're using the distributions update scripts.

Did you check if exactly the same will happen when I manually run openhab-cli clean-cache ?

[BClark09]

I only have my real-home system to test with at the moment, so won't manually run the command myself. But as far as I can see there's no functional difference in the commands it uses vs the update scripts:

https://github.com/openhab/openhab-linuxpkg/blob/main/resources/usr/bin/openhab-cli#L99-L101

[andrewfg]

When you mention Karaf cache, do you mean cache and tmp directories

From your script above it looks like the update script also cleans /marketplace as well as /tmp and /cache directories.

[BClark09]

Yep, this was necessary because of openhab/openhab-core#4019

[wborn]

I noticed on various occasions that openHAB didn't start properly after a package upgrade.

Please add some logging etc. of when this happens. If you can reproduce it maybe you can also add debug logging of openhab/karaf as well.