[BUG] Container does not start after update

[Tom7320]

Describe the bug
After an automatic update last night to the current version the container does not start anymore.

photon-docker configuration

services:
    photon:
        image: rtuszik/photon-docker:latest
        container_name: photon
        environment:
            - UPDATE_STRATEGY=PARALLEL
            - UPDATE_INTERVAL=720h # Check for updates every 30 days
            # - REGION=andorra # Optional: specific region (continent, country, or planet)
            # - APPRISE_URLS=pover://user@token  # Optional: notifications
        volumes:
            - /mnt/vmdata/photon:/photon/data
        restart: unless-stopped
        ports:
            - "2322:2322"

Debug Logs

[photon] 2026-02-16T05:08:58.213535141Z    Building photon-docker @ file:///photon
[photon] 2026-02-16T05:08:58.388343445Z       Built photon-docker @ file:///photon
[photon] 2026-02-16T05:08:58.389567032Z Uninstalled 1 package in 0.33ms
[photon] 2026-02-16T05:08:58.392127009Z Installed 1 package in 0.75ms
[photon] 2026-02-16T05:08:58.670117416Z 2026-02-16 05:08:58,669 - root - INFO - Photon Manager starting...
[photon] 2026-02-16T05:08:58.671770250Z 2026-02-16 05:08:58,671 - root - INFO - Existing index found, skipping initial setup
[photon] 2026-02-16T05:08:58.672636818Z 2026-02-16 05:08:58,672 - root - INFO - Starting Photon (attempt 1/3)...
[photon] 2026-02-16T05:08:58.679945994Z 2026-02-16 05:08:58,679 - root - INFO - Photon started with PID: 28
[photon] 2026-02-16T05:08:58.683083415Z 2026-02-16 05:08:58,680 - root - INFO - Waiting for Photon to become ready...
[photon] 2026-02-16T05:08:58.687112315Z 2026-02-16 05:08:58,687 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:08:58.769032257Z WARNING: Using incubator modules: jdk.incubator.vector
[photon] 2026-02-16T05:08:59.487933127Z 2026-02-16 05:08:59,485 [main] INFO  de.komoot.photon.App - Start up database cluster, this might take some time.
[photon] 2026-02-16T05:08:59.547644329Z Cluster Name: photon
[photon] 2026-02-16T05:08:59.547958532Z Base Path:    /photon/data/photon_data
[photon] 2026-02-16T05:08:59.551090895Z Num Of Node:  1
[photon] 2026-02-16T05:08:59.571148580Z Node Name:      Node 1
[photon] 2026-02-16T05:08:59.571321367Z HTTP Port:      9201
[photon] 2026-02-16T05:08:59.571350305Z Data Directory: /photon/data/photon_data/node_1/data
[photon] 2026-02-16T05:08:59.571422864Z Log Directory:  /photon/data/photon_data/node_1/logs
[photon] 2026-02-16T05:08:59.970859282Z [2026-02-16T05:08:59,970][INFO ][o.o.n.Node               ] version[3.4.0], pid[28], build[unknown/unknown/unknown], OS[Linux/6.8.0-100-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/21.0.9/21.0.9+10-LTS]
[photon] 2026-02-16T05:08:59.972462257Z [2026-02-16T05:08:59,972][INFO ][o.o.n.Node               ] JVM home [/opt/java/openjdk]
[photon] 2026-02-16T05:08:59.979601821Z [2026-02-16T05:08:59,979][INFO ][o.o.n.Node               ] JVM arguments [--add-modules=jdk.incubator.vector, --enable-native-access=ALL-UNNAMED, -Des.gateway.auto_import_dangling_indices=true, -Des.cluster.routing.allocation.batch_mode=true, -Dlog4j2.disable.jmx=true]
[photon] 2026-02-16T05:08:59.990387489Z [2026-02-16T05:08:59,990][INFO ][o.o.p.PluginsService     ] no modules loaded
[photon] 2026-02-16T05:08:59.991490228Z [2026-02-16T05:08:59,991][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.analysis.common.CommonAnalysisModulePlugin]
[photon] 2026-02-16T05:08:59.992443942Z [2026-02-16T05:08:59,992][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.geo.GeoModulePlugin]
[photon] 2026-02-16T05:08:59.992946605Z [2026-02-16T05:08:59,992][INFO ][o.o.p.PluginsService     ] loaded plugin [org.opensearch.transport.Netty4ModulePlugin]
[photon] 2026-02-16T05:09:00.134837964Z [2026-02-16T05:09:00,128][INFO ][o.o.e.NodeEnvironment    ] using [1] data paths, mounts [[/photon/data (vmdata)]], net usable_space [1.3tb], net total_space [1.4tb], types [9p]
[photon] 2026-02-16T05:09:00.138190027Z [2026-02-16T05:09:00,137][INFO ][o.o.e.NodeEnvironment    ] heap size [962mb], compressed ordinary object pointers [true]
[photon] 2026-02-16T05:09:00.327266060Z [2026-02-16T05:09:00,327][INFO ][o.a.l.i.v.PanamaVectorizationProvider] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
[photon] 2026-02-16T05:09:00.395685437Z [2026-02-16T05:09:00,395][INFO ][o.o.n.Node               ] node name [Node 1], node ID [wx2gm_jKSzy4xksY28hDPA], cluster name [photon], roles [data, cluster_manager]
[photon] 2026-02-16T05:09:00.426813161Z [2026-02-16T05:09:00,426][INFO ][o.o.e.ExtensionsManager  ] ExtensionsManager initialized
[photon] 2026-02-16T05:09:00.666143946Z [2026-02-16T05:09:00,665][INFO ][o.o.i.MergeSchedulerConfig] Updating autoThrottle for index _na_ from [false] to [true]
[photon] 2026-02-16T05:09:00.666701777Z [2026-02-16T05:09:00,666][INFO ][o.o.i.MergeSchedulerConfig] Updating maxThreadCount from [0] to [1] and maxMergeCount from [0] to [6] for index _na_.
[photon] 2026-02-16T05:09:00.667284785Z [2026-02-16T05:09:00,667][INFO ][o.o.i.MergeSchedulerConfig] Initialized index _na_ with maxMergeCount=6, maxThreadCount=1, autoThrottleEnabled=true
[photon] 2026-02-16T05:09:00.824811651Z [2026-02-16T05:09:00,824][INFO ][o.o.i.MergeSchedulerConfig] Updating autoThrottle for index _na_ from [false] to [true]
[photon] 2026-02-16T05:09:00.827269931Z [2026-02-16T05:09:00,827][INFO ][o.o.i.MergeSchedulerConfig] Updating maxThreadCount from [0] to [1] and maxMergeCount from [0] to [6] for index _na_.
[photon] 2026-02-16T05:09:00.828090354Z [2026-02-16T05:09:00,828][INFO ][o.o.i.MergeSchedulerConfig] Initialized index _na_ with maxMergeCount=6, maxThreadCount=1, autoThrottleEnabled=true
[photon] 2026-02-16T05:09:01.705199719Z [2026-02-16T05:09:01,704][INFO ][o.o.t.NettyAllocator     ] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=962mb}]
[photon] 2026-02-16T05:09:01.758979272Z [2026-02-16T05:09:01,757][INFO ][o.o.d.DiscoveryModule    ] using discovery type [single-node] and seed hosts providers [settings]
[photon] 2026-02-16T05:09:02.058898347Z [2026-02-16T05:09:02,058][WARN ][o.o.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[photon] 2026-02-16T05:09:02.301422508Z [2026-02-16T05:09:02,301][INFO ][o.o.n.Node               ] initialized
[photon] 2026-02-16T05:09:02.303087934Z [2026-02-16T05:09:02,302][INFO ][o.o.n.Node               ] starting ...
[photon] 2026-02-16T05:09:02.372056078Z [2026-02-16T05:09:02,368][INFO ][o.o.t.TransportService   ] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[photon] 2026-02-16T05:09:02.651265540Z [2026-02-16T05:09:02,651][INFO ][o.o.c.c.Coordinator      ] cluster UUID [Opc5VjX9QMK2WEqZnBsWLA]
[photon] 2026-02-16T05:09:02.801085168Z [2026-02-16T05:09:02,800][INFO ][o.o.c.s.ClusterManagerService] Tasks batched with key: org.opensearch.cluster.coordination.JoinHelper, count:3 and sample tasks: elected-as-cluster-manager ([1] nodes joined)[{Node 1}{wx2gm_jKSzy4xksY28hDPA}{llSAws6KSKywGVJF5rn0vg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 80, version: 799, delta: cluster-manager node changed {previous [], current [{Node 1}{wx2gm_jKSzy4xksY28hDPA}{llSAws6KSKywGVJF5rn0vg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}
[photon] 2026-02-16T05:09:02.958498812Z [2026-02-16T05:09:02,958][INFO ][o.o.c.s.ClusterApplierService] cluster-manager node changed {previous [], current [{Node 1}{wx2gm_jKSzy4xksY28hDPA}{llSAws6KSKywGVJF5rn0vg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]}, term: 80, version: 799, reason: Publication{term=80, version=799}
[photon] 2026-02-16T05:09:02.969833192Z [2026-02-16T05:09:02,969][INFO ][o.o.d.PeerFinder         ] setting findPeersInterval to [1s] as node commission status = [true] for local node [{Node 1}{wx2gm_jKSzy4xksY28hDPA}{llSAws6KSKywGVJF5rn0vg}{127.0.0.1}{127.0.0.1:9300}{dm}{shard_indexing_pressure_enabled=true}]
[photon] 2026-02-16T05:09:02.974933221Z [2026-02-16T05:09:02,974][INFO ][o.o.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[photon] 2026-02-16T05:09:02.978081798Z [2026-02-16T05:09:02,977][INFO ][o.o.n.Node               ] started
[photon] 2026-02-16T05:09:03.046287710Z [2026-02-16T05:09:03,045][INFO ][o.o.g.GatewayService     ] recovered [1] indices into cluster_state
[photon] 2026-02-16T05:09:03.689960531Z 2026-02-16 05:09:03,689 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:03.785104990Z [2026-02-16T05:09:03,784][INFO ][o.o.i.MergeSchedulerConfig] Updating autoThrottle for index photon from [false] to [true]
[photon] 2026-02-16T05:09:03.786612445Z [2026-02-16T05:09:03,786][INFO ][o.o.i.MergeSchedulerConfig] Updating maxThreadCount from [0] to [1] and maxMergeCount from [0] to [6] for index photon.
[photon] 2026-02-16T05:09:03.787170514Z [2026-02-16T05:09:03,787][INFO ][o.o.i.MergeSchedulerConfig] Initialized index photon with maxMergeCount=6, maxThreadCount=1, autoThrottleEnabled=true
[photon] 2026-02-16T05:09:03.799909651Z [2026-02-16T05:09:03,799][INFO ][o.o.p.PluginsService     ] PluginService:onIndexModule index:[photon/_qgY0cuQTheANxav_G3A1w]
[photon] 2026-02-16T05:09:03.884218558Z [2026-02-16T05:09:03,884][INFO ][o.o.i.MergeSchedulerConfig] Updating autoThrottle for index photon from [true] to [true]
[photon] 2026-02-16T05:09:06.448698671Z [2026-02-16T05:09:06,448][INFO ][o.o.c.r.a.AllocationService] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[photon][1]]]).
[photon] 2026-02-16T05:09:07.129115788Z [2026-02-16T05:09:07,128][INFO ][d.k.p.App                ] Database cluster is now ready.
[photon] 2026-02-16T05:09:07.235640355Z [2026-02-16T05:09:07,235][ERROR][d.k.p.App                ] Cannot access property meta data. Database too old?
[photon] 2026-02-16T05:09:07.236729561Z [2026-02-16T05:09:07,236][INFO ][d.k.p.App                ] Shutting down the service.
[photon] 2026-02-16T05:09:07.237571225Z [2026-02-16T05:09:07,237][INFO ][d.k.p.Server             ] Shutting down OpenSearch runner
[photon] 2026-02-16T05:09:07.238144923Z [2026-02-16T05:09:07,238][INFO ][o.o.n.Node               ] stopping ...
[photon] 2026-02-16T05:09:07.373455179Z [2026-02-16T05:09:07,373][INFO ][o.o.n.Node               ] stopped
[photon] 2026-02-16T05:09:07.374417040Z [2026-02-16T05:09:07,374][INFO ][o.o.n.Node               ] closing ...
[photon] 2026-02-16T05:09:07.380803009Z [2026-02-16T05:09:07,380][INFO ][o.o.n.Node               ] closed
[photon] 2026-02-16T05:09:07.381710745Z [2026-02-16T05:09:07,381][INFO ][stdout                   ] Closed all nodes.
[photon] 2026-02-16T05:09:08.694228840Z 2026-02-16 05:09:08,694 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:13.696124202Z 2026-02-16 05:09:13,695 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:18.700198844Z 2026-02-16 05:09:18,700 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:23.702089312Z 2026-02-16 05:09:23,701 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:28.705247498Z 2026-02-16 05:09:28,704 - root - ERROR - Photon health check failed after 1 attempts
[photon] 2026-02-16T05:09:33.709653063Z 2026-02-16 05:09:33,709 - root - ERROR - Photon health check failed after 1 attempts

Help would be appreciated. Thx a lot! :)

[tedpearson]

Experiencing the same behavior. You can't downgrade without some sort of cleanup of the data volume after running 2.0.0. I ended up deleting the data volume to get 1.3.0 to start up.

[rtuszik]

It looks like the check for an outdated index isn’t working as I expected it to.
You will have to clear the docker volume containing the old index in order to have it re-download the recent one.

[Tom7320]

It looks like the check for an outdated index isn’t working as I expected it to. You will have to clear the docker volume containing the old index in order to have it re-download the recent one.

That's exactly what I did. Thanks a lot! I just asked here because I wanted to avoid downloading the whole index again...

[tedpearson]

It looks like the check for an outdated index isn’t working as I expected it to. You will have to clear the docker volume containing the old index in order to have it re-download the recent one.

Thanks, removing the docker volume worked for me for the upgrade also.

[pax0707]

Downloading the new data now.
Hope it helps.

[pax0707]

gain, after the update to v2.1.1:

2026-03-10T11:54:22.698482016Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][0], node[quiGz7whSrOkOXcOfnnDoQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=FtPi9GugQBeiJPWaPjuHtg], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[fetching_shard_data]]
2026-03-10T11:54:22.698557102Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][1], node[quiGz7whSrOkOXcOfnnDoQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=mWc_fKwHSYmXrvbIq-fd9A], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[fetching_shard_data]]
2026-03-10T11:54:22.698617716Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][3], node[quiGz7whSrOkOXcOfnnDoQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=qpdi6HtOQFiANrb1HgsLbQ], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[fetching_shard_data]]
2026-03-10T11:54:22.698687002Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][4], node[quiGz7whSrOkOXcOfnnDoQ], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[INITIALIZING], a[id=Qbats_18TrSxl6qN8R9rbA], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[fetching_shard_data]]
2026-03-10T11:54:22.698739225Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] ---- unassigned
2026-03-10T11:54:22.698798660Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][0], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[no_attempt]]
2026-03-10T11:54:22.698862195Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[no_attempt]]
2026-03-10T11:54:22.698931440Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][2], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[deciders_throttled]]
2026-03-10T11:54:22.699008276Z [2026-03-10T11:54:22,698][WARN ][stderr                   ] --------[photon][2], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[no_attempt]]
2026-03-10T11:54:22.699067800Z [2026-03-10T11:54:22,699][WARN ][stderr                   ] --------[photon][3], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[no_attempt]]
2026-03-10T11:54:22.699149686Z [2026-03-10T11:54:22,699][WARN ][stderr                   ] --------[photon][4], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2026-03-10T11:53:51.726Z], delayed=false, allocation_status[no_attempt]]
2026-03-10T11:54:22.699196630Z [2026-03-10T11:54:22,699][WARN ][stderr                   ] tasks: (0):
2026-03-10T11:54:22.699749711Z [2026-03-10T11:54:22,699][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.onFailure(OpenSearchRunner.java:1442)
2026-03-10T11:54:22.700012361Z [2026-03-10T11:54:22,699][WARN ][stderr                   ] 	at org.codelibs.opensearch.runner.OpenSearchRunner.ensureYellow(OpenSearchRunner.java:838)
2026-03-10T11:54:22.700083496Z [2026-03-10T11:54:22,700][WARN ][stderr                   ] 	at de.komoot.photon.Server.startInternal(Server.java:107)
2026-03-10T11:54:22.700141441Z [2026-03-10T11:54:22,700][WARN ][stderr                   ] 	at de.komoot.photon.Server.<init>(Server.java:65)
2026-03-10T11:54:22.700204265Z [2026-03-10T11:54:22,700][WARN ][stderr                   ] 	at de.komoot.photon.App.runPhoton(App.java:83)
2026-03-10T11:54:22.700255069Z [2026-03-10T11:54:22,700][WARN ][stderr                   ] 	at de.komoot.photon.App.main(App.java:49)
2026-03-10T11:54:22.834415772Z 2026-03-10 11:54:22,834 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:27.835627807Z 2026-03-10 11:54:27,835 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:32.836784028Z 2026-03-10 11:54:32,836 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:37.839642826Z 2026-03-10 11:54:37,839 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:42.840739653Z 2026-03-10 11:54:42,840 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:47.841801136Z 2026-03-10 11:54:47,841 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:52.842760263Z 2026-03-10 11:54:52,842 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:54:57.843755531Z 2026-03-10 11:54:57,843 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:02.844877180Z 2026-03-10 11:55:02,844 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:07.846229146Z 2026-03-10 11:55:07,846 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:12.847294060Z 2026-03-10 11:55:12,847 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:17.854663138Z 2026-03-10 11:55:17,854 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:22.855678918Z 2026-03-10 11:55:22,855 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:27.858250526Z 2026-03-10 11:55:27,858 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:32.859298438Z 2026-03-10 11:55:32,859 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:37.860685587Z 2026-03-10 11:55:37,860 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:42.861771973Z 2026-03-10 11:55:42,861 - root - ERROR - Photon health check failed after 1 attempts
2026-03-10T11:55:47.862022666Z 2026-03-10 11:55:47,861 - root - ERROR - Photon failed to become ready within 120 seconds
2026-03-10T11:55:47.862104162Z 2026-03-10 11:55:47,862 - root - ERROR - Photon health check failed on attempt 1
2026-03-10T11:55:47.863712572Z 2026-03-10 11:55:47,863 - root - INFO - Stopping Photon...