refactor(docker): supervise the container with s6-overlay

[gantoine]

Description

^{Explain the changes or enhancements you are proposing with this pull request.}

Supersedes #3990, which fixed the unbounded shutdown wait in docker/init_scripts/init. This takes the alternative route discussed there: rather than bounding the wait, replace the hand-rolled supervisor with one that already solves this class of problem.

init was ~400 lines of process supervision: PID files, /proc polling, a 5s watchdog loop, and a trap-based shutdown whose wait was unbounded. A process that refused to exit stranded PID 1, and the container then sat Up serving nothing, which no restart policy can detect because policies only react to exits.

s6-overlay replaces it. Ordering and readiness gates that were sequential bash become a declared dependency graph, and the failure modes become configuration rather than code:

romm-init ─ romm-valkey ─ romm-valkey-ready ─ romm-migrations ─ romm-startup ─┬─ romm-gunicorn ─ romm-gunicorn-ready ─ romm-nginx
                                                                              ├─ romm-rq-worker
                                                                              ├─ romm-rq-scheduler
                                                                              ├─ romm-watcher
                                                                              └─ romm-sync-watcher

docker-entrypoint.sh is unchanged and still execs /init, which is now s6's. Every process, environment variable and conditional from the old script carried over.

Changes

• timeout-kill (10s) bounds SIGTERM before SIGKILL, and flag-timeout-killpg escalates to the whole process group. No process can hold PID 1 open.
• S6_BEHAVIOUR_IF_STAGE2_FAILS=2 exits the container when a startup step fails, so the restart policy retries. This is the self-healing path for the reported case, where the daemon's restart policy starts containers at host boot without honouring compose's depends_on and RomM comes up before its database.
• valkey gets --shutdown-on-sigterm force, so it stops refusing to exit when the snapshot it writes on shutdown cannot be written. This is the root cause of the reported hang, and unlike setting REDIS_SAVE_POLICY empty it costs no persistence.
• A crashed service is restarted on its own instead of taking the whole container down. Its leftover children are swept first, see the note below.
• s6-overlay v3.2.3.2 is pinned by version and SHA256 per architecture, for linux/amd64 and linux/arm64.

One regression found and fixed during testing

Hard-killing nginx's master leaves its workers orphaned and still holding :8080, so every replacement fails to bind. The first version of this branch restart-looped forever on that.

master is not immune either: I tested it, and the same kill takes the whole container down (exit 1, after 20 Address in use errors), relying on the restart policy to clear the orphans. Each longrun now has a finish script that sweeps the dead service's process group before s6 brings it back, which makes this strictly better than today: the service recovers and the container stays up.

Behaviour change for read-only root filesystems

/run must now be mounted exec, since s6 executes its generated stage scripts from there, and Docker's --tmpfs /run defaults to noexec. Documented in docs/BACKEND_ARCHITECTURE.md.

Worth saying plainly that a read-only root filesystem does not fully work on master today either: it fails earlier, on /etc/nginx/conf.d/default.conf, before s6 is involved. I did not try to fix that pre-existing part here.

Testing

Built the image and exercised it against a real MariaDB. All results below are from the built image, not the fast iteration image:

Scenario	Result
Startup, heartbeat	HTTP 200 after 10s, no warnings or deprecations in the log
Read-only /redis-data, docker stop -t 60	5.6s, exit 0 (master: 60.3s, exit 137, SIGKILLed by docker)
Database unreachable at boot	exits 1 after 9s, restart policy retries (master: wedges)
Service ignoring SIGTERM, with children	15.4s, exit 0, PID 1 never stranded
nginx, gunicorn and rq worker each kill -9ed	all respawn, heartbeat 200, 0 bind failures, container stays up
Optional services off, then on	stay down / come up, each started exactly once, no restart loop
External Redis (REDIS_HOST set)	internal valkey stays down, 0 local valkey-server processes
Non-root (--user 1000:1000)	heartbeat 200
Read-only rootfs (/run mounted exec)	heartbeat 200

Also shellcheck and bash -n clean across all 21 shell scripts, and trunk fmt && trunk check clean.

Draft, because

• The compose example is untouched. fix(docker): bound the shutdown wait so a stuck process cannot strand PID 1 #3990 added a healthcheck to make a hung PID 1 observable; PID 1 can no longer hang, so whether that is still wanted is a separate call.
• No CI run has exercised the multi-arch build yet. The linux/arm64 path is built locally, the linux/amd64 tarball is pinned but unbuilt here.
• Worth a second opinion on whether replacing the supervisor wholesale is the direction you want, versus the narrower fix in fix(docker): bound the shutdown wait so a stuck process cannot strand PID 1 #3990.

Checklist

^{Please check all that apply.}

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

_{There is no shell test harness in the repo. The verification above is manual, against real containers, and reproducible from the table.}

AI assistance disclosure

Per CONTRIBUTING.md: this change was written with AI assistance (Claude Code). The AI wrote the s6 service tree, the Dockerfile changes and the docs, and ran the verification above. Reviewed before opening.

[XenuIsWatching]

Heads up: romm-rq-scheduler/run carries over the same four-flag gate as init, and that condition is the root cause of a bug I just opened #4033 for.

if [[ ${ENABLE_SCHEDULED_RESCAN:-false} != "true" &&
	${ENABLE_SCHEDULED_UPDATE_SWITCH_TITLEDB:-false} != "true" &&
	${ENABLE_SCHEDULED_UPDATE_LAUNCHBOX_METADATA:-false} != "true" &&
	${ENABLE_SCHEDULED_CLEANUP_ORPHANED_RESOURCES:-false} != "true" ]]; then
	debug_log "No scheduled tasks are enabled, not starting the RQ scheduler"
	disable_service

Nothing else respects that condition, in three separate ways:

• startup.py registers cleanup_netplay, cleanup_upload_tmp and cleanup_zip_cache unconditionally, all enabled=True with cron strings. On a default install none of them ever run.
• ENABLE_SCHEDULED_CONVERT_IMAGES_TO_WEBP, ENABLE_SCHEDULED_RETROACHIEVEMENTS_PROGRESS_SYNC and ENABLE_SYNC_PUSH_PULL gate periodic tasks but are absent from the list, so enabling any one alone silently does nothing.
• The filesystem watcher defers its rescans through the scheduler (watcher.py:214), and ENABLE_RESCAN_ON_FILESYSTEM_CHANGE is not in the list either. That is the reported failure: the watcher accumulates delayed scan_platforms entries nothing will ever execute, fix(scan): refuse concurrent scans and stop queued ones #3974's guard reads them as a queued scan, and every manual scan is refused permanently with "A scan is already in progress".

Worth noting entrypoint.sh starts the scheduler unconditionally and always has, so the two entry paths already disagree about this on master. #4033 drops the condition in init to match.

Not a blocker for this PR, just flagging so it does not get re-introduced. Whichever of the two lands second will need to carry the change across, and in the s6 layout that means dropping the disable_service branch from romm-rq-scheduler/run entirely.

Separately, and much smaller: this drops the romm service healthcheck that was in #3990. I have no objection, the bounded shutdown is the part that mattered and --shutdown-on-sigterm force is a better fix than what I had. Just noting it since a hung PID 1 stays unobservable without one.

I verified the valkey change against 5.1.0 on my own hardware, read-only /redis-data with snapshots configured:

shutdown-on-sigterm=default  ->  26s, exit 137   (SIGKILLed by docker)
shutdown-on-sigterm=force    ->   1s, exit 0

Nice.