refactor(BA-7321): emit a conventional compose deployment from the DOCKER install mode

[kyujin-cho]

Summary

Converges the DOCKER install mode onto the same deployment convention as a hand-written compose file, removing the installer-specific dialect (whole-directory mounts + command:/working_dir overrides + all-host networking):

• Per-file configs, default commands: every service mounts its generated config read-only at the image's default /etc/backend.ai/* path and runs the image's default command. The manager's entrypoint therefore performs its designed keypair bootstrap, exchanging the RPC keypair through <install-dir>/fixtures mounted at /app/fixtures.
• Host networking only where required (manager, agent, and the manager-cli one-off tool). The webserver, storage-proxy, and app-proxy trio move to the project bridge network with published ports matching the installer's fixed ServiceConfig ports. A new _fixup_bridge_service_addresses() rewrites their generated configs: outbound addresses (halfstack, manager API, coordinator API, default OTLP endpoint) → host.docker.internal (via extra_hosts: host-gateway), bind addresses → 0.0.0.0; announce/advertised addresses are untouched. The storage-proxy's working-dir-relative paths (vfolder root, TLS material, IPC dir) become absolute.
• storage-proxy runs as the installing user (user: "<uid>:<gid>" substituted at render time), matching PACKAGE-mode vfolder file ownership; its IPC dir moves under the pre-created, user-owned install tree.
• Parity mounts only where load-bearing: agent and storage-proxy (paths handed to the host Docker daemon), and manager-cli (which also keeps working_dir for install-dir-relative CLI invocations like mgr generate-rpc-keypair fixtures/manager).
• Ordering fixes this design exposed: configure_appproxy() now runs before install_appproxy_db() (a docker compose run against a service whose per-file mount source doesn't exist yet would make the daemon create a root-owned directory in its place), and the copied alembic-appproxy.ini gets a localhost → host.docker.internal rewrite via a new base-class hook, since the schema one-off now runs on the bridge network.
• Tests assert the full convention: networking split, per-service config mounts at default-command targets, published port sets, parity-mount allowlist, storage-proxy user, and no command/working_dir overrides outside manager-cli. docker/README.md's DOCKER-mode contract and deployment-layout notes are updated to match.

Needs live re-verification of the full DOCKER install flow on a Linux Docker host before undrafting (bridge reachability via host-gateway, appproxy schema one-off, keypair bootstrap via the manager entrypoint, storage-proxy as non-root) — folds into the BA-7270 E2E scope.

Merge order note: stacked on the bake CI PR — last PR of the BA-7264 chain; retarget to main after the base PR merges.

Part of the BA-7264 epic (#13582).

[kyujin-cho]

Second commit (50f777a) addresses the two-compose-files finding from live testing: the deployment now runs as ONE compose project. docker-compose.services.yml is the single entry file — it include:s docker-compose.halfstack.current.yml (kept as a separate file since dev tooling regenerates it), so docker compose -f docker-compose.services.yml ps/down/up covers halfstack + services together.

Consequences wired in:

• Halfstack starts under the unified backendai-services project (_start_halfstack override, scoped to the halfstack members via config --services); the services compose file is generated before it.
• Bridge services now address halfstack and the coordinator by compose service DNS with container-side ports (backendai-half-db:5432, backendai-half-redis:6379, backendai-half-etcd:2379, appproxy-coordinator:10200); extra_hosts: host-gateway remains only on the webserver (manager API on the host network). The alembic-appproxy.ini rewrite also targets backendai-half-db:5432 now.
• depends_on health gating on db/redis/etcd mirrors the reference hand-written deployment.
• Requires Docker Compose >= 2.20 (include).

Live retest points on top of the previous list: single-project ps output, halfstack health-gated startup, service-DNS reachability from the bridge services.

[kyujin-cho]

Third commit (bea28dc), after live testing showed two files still on disk and exposed a real bug: under include, the halfstack services sit on their named half network while the bridge services sat on the project default network — service-DNS resolution between them would never have worked.

Both fixed by merging instead of including:

• generate_services_compose() folds the prepared halfstack YAML into the rendered services document (merge_halfstack_into_services, loud failure on name collisions) and deletes docker-compose.halfstack.current.yml — the install directory ends up with exactly ONE compose file.
• The five bridge services declare networks: [half], joining the halfstack network; merge integrity (all network refs + depends_on targets) is validated against the real halfstack template in tests and a full-merge simulation (23 services).
• install_halfstack override: prepare files → generate merged compose → start only the halfstack members (recorded at merge time, profile-filtered — the config --services subprocess is gone).
• The Compose ≥ 2.20 requirement disappears with include; stale template header comments rewritten.

Retest: single docker-compose.services.yml with backendai-half-* inline; docker compose -f docker-compose.services.yml ps covers the whole deployment; DNS reachability webserver/storage/appproxy → halfstack now goes through the shared half network.

[kyujin-cho]

Fourth commit (7e01296): per review direction, the vfolder root moves to the fixed /vfroot/local/volume1 and with it ALL daemon-visible state leaves the install directory — no running service bind-mounts the home-directory-resident install dir anymore.

• Agent parity mounts are now exactly /var/lib/backend.ai (scratches, agent/ var state, commit/, and the krunner share — its separate mount is gone), /tmp/backend.ai (kernel IPC), and /vfroot/local. agent.toml fixups and hydrate_install_info write these fixed paths.
• storage-proxy mounts only its config (ro), /vfroot/local, and <install-dir>/configs/storage-proxy (ro TLS material); its IPC dir became container-local.
• Ownership bootstrap: the installer (unprivileged) prepares /vfroot/local/volume1 via a docker compose run --user 0 storage-proxy one-off after image pull — the daemon auto-creates the bind source, the one-off chowns the volume dir to the installing uid:gid, then the host-side version.txt write works. prepare_local_vfolder_host was split into file-prep (overridden) + DB-grant halves.
• The install() pre-create loop is gone; only manager-cli (installer tool) and the manager's narrow fixtures/ mount still reference the install directory.
• Tests assert the per-service system-mount table and that no other service touches /var/lib/backend.ai or /vfroot; README parity table + fragment updated.

Retest notes: fresh runs need no manual prep (/vfroot, /var/lib/backend.ai, /tmp/backend.ai are daemon-created); a truly clean slate additionally means sudo rm -rf /vfroot/local/volume1 /var/lib/backend.ai /tmp/backend.ai after docker compose down. Wiping ~/backendai between runs no longer breaks running containers' mounts.