feat(BA-7321): add DOCKER install mode deploying the published service images

changes/13681.feature.md

@@ -0,0 +1 @@
+Add a DOCKER install mode to the installer that deploys the published `lablup/backend.ai-*` service images with docker compose, including full bootstrap of the database schema, fixtures, etcd configuration, and kernel images

docker/README.md

@@ -60,6 +60,12 @@ path the image's default command reads:
 | appproxy-coordinator | `/etc/backend.ai/proxy-coordinator.toml` | |
 | appproxy-worker | `/etc/backend.ai/proxy-worker.toml` | one container per worker: each needs its OWN toml with a unique `authority`, protocol (`http`/`tcp`), `api_bind_addr` port, and a non-overlapping `[proxy_worker.port_proxy] bind_port_range` (port-based frontends only) — and the compose port mappings must match |
 
+The `/etc/backend.ai/*` targets are what the images' **default commands** read.
+The DOCKER install mode (see the reference compose file below) instead keeps
+every config in the parity-mounted install directory and overrides each
+service's `command:` to point there — either layout works; pick one per
+deployment.
+
 Shared prerequisites:
 
 | Item | Used by | Why |
@@ -79,7 +85,7 @@ together they amount to root-equivalent control of the host.
 | Requirement | manager | agent | Why |
 |---|---|---|---|
 | `network_mode: host` | optional | ✅ | Agent: kernel↔agent ZMQ/service ports and agent RPC are advertised on host addresses; kernels spawned on the host network must reach them. Manager: convenience only — the bridge alternative works via the `announce-addr` / `announce-internal-addr` knobs |
-| `privileged: true` | — | ✅ | Agent: container/device management against the host daemon; sysfs reads for metrics. The manager does not need it — the Docker socket alone suffices for its (conditional) Docker use |
+| `privileged: true` | default | ✅ | Agent: container/device management against the host daemon; sysfs reads for metrics. The manager does not strictly need it — the Docker socket alone suffices for its (conditional) Docker use — but the DOCKER install mode's generated compose grants it by default; remove the flag for a least-privilege deployment |
 | `/var/run/docker.sock` bind mount | conditional | ✅ | DooD: containers are created by talking to the **host** Docker daemon. Manager: only when the `local` container registry is used |
 | `pid: host` | — | ✅ | Host PID namespace visibility: the agent inspects and signals kernel processes by host PID |
 | `cgroup: host` (host cgroup namespace) | — | ✅ | **Required, not optional** — see below |
@@ -110,15 +116,21 @@ the same absolute path on both sides. The paths are set by `agent.toml` —
 **every one of them must be an absolute path**, bind-mounted host↔container at
 the identical location:
 
-| Config knob (`agent.toml`) | Reference value | Used for |
+The values below are the defaults the DOCKER install mode writes
+(`<install-dir>` is the install target directory); a hand-rolled deployment
+may choose any absolute paths as long as the parity rule holds.
+
+| Config knob (`agent.toml`) | DOCKER-mode default | Used for |
 |---|---|---|
-| `[container] scratch-root` | `/var/lib/backend.ai/scratches` | Scratch roots of kernel containers |
-| `[agent] ipc-base-path` | `/tmp/backend.ai/ipc` (this document's chosen value; `configs/agent/sample.toml` ships `/var/run/backend.ai/ipc`) | Agent↔kernel IPC sockets |
-| `[agent] var-base-path` | `/var/lib/backend.ai` | Plugin state bind-mounted into kernels (e.g. accelerator hook caches) |
-| env `BACKENDAI_KRUNNER_SHARED` | `/var/lib/backend.ai/krunner` | Kernel-runner files: the image entrypoint copies them here so the host daemon can mount them into kernels. Covered automatically by the `/var/lib/backend.ai` parity mount; the entrypoint **refuses to start** without it — override the path via the `BACKENDAI_KRUNNER_SHARED` env var |
+| `[container] scratch-root` | `<install-dir>/scratches` | Scratch roots of kernel containers |
+| `[agent] mount-path` | `<install-dir>/vfolder/local` | Vfolder tree whose subdirectories become kernel bind-mount sources |
+| `[agent] ipc-base-path` | `<install-dir>/ipc/agent` | Agent↔kernel IPC sockets |
+| `[agent] var-base-path` | `<install-dir>/var/agent` | Plugin state bind-mounted into kernels (e.g. accelerator hook caches) |
+| `[agent] image-commit-path` | `<install-dir>/tmp/backend.ai/commit` | Session image-commit tarballs written by the host daemon |
+| env `BACKENDAI_KRUNNER_SHARED` | `/var/lib/backend.ai/krunner` | Kernel-runner files: the image entrypoint copies them here so the host daemon can mount them into kernels. Mounted as its **own** bind mount (the Docker daemon creates the host directory on first start); the entrypoint **refuses to start** without it |
 
-With the reference values, two parity mounts cover everything:
-`/var/lib/backend.ai` and `/tmp/backend.ai`.
+With these defaults, two mounts cover everything: the `<install-dir>` parity
+mount and the fixed `/var/lib/backend.ai/krunner` krunner share.
 
 Vfolder roots (e.g. `/vfroot/local/volume1`) follow the same rule on the
 **storage-proxy**: mount each volume at the identical absolute path on host and
@@ -135,20 +147,48 @@ namespace is invisible to the host daemon — use `hostdir`.
 
 `docker-compose.monorepo.yml` at the repository root is a **partial, legacy
 example** — it uses different image names, includes no agent or storage-proxy,
-and runs on a bridge network. The fragment below is the authoritative
-reference for the two elevated services, verified against a live deployment.
-Replace `<version>` with a tag from the tagging scheme above. The `cgroup:`
-key requires Docker Compose v2.15+.
+and runs on a bridge network. The authoritative reference is the compose file
+the **DOCKER install mode** of `backend.ai-installer` generates at
+`<install-dir>/docker-compose.services.yml` (rendered from
+`src/ai/backend/install/configs/docker-compose.services.yml`). Its contract:
+
+- Every service runs on the host network, so the generated configs use the
+  same `127.0.0.1` addressing as a package-based install.
+- The install directory is bind-mounted into every container at the identical
+  absolute path, and each service's `command:` reads its config from there —
+  no `/etc/backend.ai` mounts.
+- `/etc/machine-id` is passed through read-only to the manager and agent so
+  anything deriving a stable host identity sees the host's, not the
+  container's.
+- All images are pinned to the installer's own version (the event-bus
+  version-skew rule above).
+- The compose project name is fixed (`backendai-services`) so the file never
+  shares a project with the halfstack file the installer places in the same
+  directory.
+- One-off management commands run via `docker compose run` on a dedicated
+  non-privileged `manager-cli` twin of the manager (no Docker socket, no
+  restart policy; its `cli` profile keeps `up -d` from starting it).
+- No agent-watcher container ships in this mode, and the app-proxy data plane
+  runs as an `appproxy-worker` / `appproxy-worker-tcp` pair.
+
+The fragment below reproduces the two elevated services. Replace `<version>`
+with a tag from the tagging scheme above and `<install-dir>` with the install
+target directory. The `cgroup:` key requires Docker Compose v2.15+.
 
 ```yaml
+name: backendai-services
 services:
   manager:
     image: lablup/backend.ai-manager:<version>
     network_mode: host    # optional — bridge works via the announce-addr knobs
+    privileged: true      # installer default; the socket alone suffices (see the matrix) — remove for least privilege
+    working_dir: <install-dir>
+    command: ["python", "-m", "ai.backend.manager.server", "--config", "<install-dir>/manager.toml"]
     volumes:
-      - /var/run/docker.sock:/var/run/docker.sock   # only when the `local` container registry is used
-      - ./manager.toml:/etc/backend.ai/manager.toml:ro
-      - ./fixtures:/app/fixtures    # read-write: the entrypoint writes the RPC keypair here
+      - /var/run/docker.sock:/var/run/docker.sock   # needed only when the `local` container registry is used
+      - /etc/machine-id:/etc/machine-id:ro
+      # parity mount: configs, fixtures/ (RPC keypair, written relative to working_dir), vfolder/
+      - <install-dir>:<install-dir>
     restart: unless-stopped
 
   agent:
@@ -157,20 +197,21 @@ services:
     privileged: true
     pid: host
     cgroup: host          # REQUIRED on cgroup v2 hosts; Docker defaults to private
-    deploy:               # GPU nodes only
-      resources:
+    deploy:               # GPU nodes only (the installer currently rejects --accelerator
+      resources:          # until the published images bundle the accelerator plugins)
         reservations:
           devices:
             - driver: nvidia
               count: all
               capabilities: [gpu]
+    working_dir: <install-dir>
+    command: ["python", "-m", "ai.backend.agent.server", "-f", "<install-dir>/agent.toml"]
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
-      - ./agent.toml:/etc/backend.ai/agent.toml:ro
+      - /etc/machine-id:/etc/machine-id:ro
       # path-parity mounts: host path == container path
-      # (the krunner share /var/lib/backend.ai/krunner is covered by the /var/lib/backend.ai mount)
-      - /var/lib/backend.ai:/var/lib/backend.ai
-      - /tmp/backend.ai:/tmp/backend.ai
+      - <install-dir>:<install-dir>
+      - /var/lib/backend.ai/krunner:/var/lib/backend.ai/krunner   # created by the Docker daemon on first start
     restart: unless-stopped
 ```
 

src/ai/backend/install/app.py

@@ -36,7 +36,7 @@
 from ai.backend.plugin.entrypoint import find_build_root
 
 from .common import detect_os
-from .context import DevContext, PackageContext, current_log
+from .context import DevContext, DockerContext, PackageContext, current_log
 from .types import (
     Accelerator,
     CliArgs,
@@ -195,6 +195,132 @@ async def install(self, dist_info: DistInfo, install_variable: InstallVariable)
             current_log.reset(_log_token)
 
 
+class DockerInstallReport(Static):
+    _install_info: InstallInfo
+
+    def __init__(self, install_info: InstallInfo, **kwargs: Any) -> None:
+        super().__init__(**kwargs)
+        self._install_info = install_info
+
+    @override
+    def compose(self) -> ComposeResult:
+        install_info = self._install_info
+        service = install_info.service_config
+        base_path = install_info.base_path
+        yield Markdown(
+            textwrap.dedent(
+                f"""
+        All Backend.AI services are now running as containers of the
+        `lablup/backend.ai-*:{install_info.version}` images.
+
+        Connect to `http://{service.webserver_addr.face.host}:{service.webserver_addr.face.port}`
+        with the admin credentials:
+        - Username: `admin@lablup.com`
+        - Password: `wJalrXUt`
+
+        The deployment lives in `{base_path}`:
+        - `docker-compose.services.yml` — the service containers
+          (inspect with `docker compose -f docker-compose.services.yml ps`)
+        - `docker-compose.halfstack.current.yml` — PostgreSQL / Redis / etcd
+        - `*.toml` / `webserver.conf` — the per-service configurations,
+          bind-mounted into the containers
+
+        To stop or restart everything:
+        ```console
+        $ cd {base_path}
+        $ docker compose -f docker-compose.services.yml down
+        $ docker compose -f docker-compose.services.yml up -d
+        ```
+
+        To see this guide again, run './backendai-install-<platform> install --show-guide'.
+        """
+            )
+        )
+
+
+class DockerSetup(Static):
+    _non_interactive: bool
+
+    def __init__(self, *, non_interactive: bool = False, **kwargs: Any) -> None:
+        super().__init__(**kwargs)
+        self._non_interactive = non_interactive
+
+    @override
+    def compose(self) -> ComposeResult:
+        yield Label("Docker Compose Setup", classes="mode-title")
+        with TabbedContent():
+            with TabPane("Install Log", id="tab-docker-log"):
+                yield SetupLog(
+                    wrap=True,
+                    classes="log",
+                )
+            with TabPane("Install Report", id="tab-docker-report"):
+                yield Label("Installation is not complete.")
+
+    def begin_install(self, dist_info: DistInfo, install_variable: InstallVariable) -> None:
+        self.query_one("SetupLog.log").focus()
+        top_tasks.add(asyncio.create_task(self.install(dist_info, install_variable)))
+
+    async def install(self, dist_info: DistInfo, install_variable: InstallVariable) -> None:
+        _log = self.query_one(".log", SetupLog)
+        _log_token = current_log.set(_log)
+        # prerequisites
+        if self._non_interactive:
+            if dist_info.target_path is None:
+                raise ValueError("Target path must be specified in non-interactive mode")
+        else:
+            if dist_info.target_path.exists():
+                input_box = InputDialog(
+                    f"The target path {dist_info.target_path} already exists. "
+                    "Please set a different target path below, or "
+                    "leave the box as blank to overwrite the folder.",
+                    str(dist_info.target_path),
+                    allow_cancel=False,
+                )
+                _log.mount(input_box)
+                value = await input_box.wait()
+                if value is None:
+                    raise ValueError("Target path input was cancelled")
+                dist_info.target_path = Path(value)
+        ctx = DockerContext(
+            dist_info,
+            install_variable,
+            cast(App[None], self.app),
+            non_interactive=self._non_interactive,
+        )
+        try:
+            await ctx.check_prerequisites()
+            # install
+            await ctx.install()
+            # configure
+            await ctx.configure()
+            # post-setup
+            await ctx.populate_images()
+            await ctx.start_services()
+            await ctx.dump_install_info()
+            install_report = DockerInstallReport(ctx.install_info, id="install-report")
+            self.query_one("TabPane#tab-docker-report Label").remove()
+            self.query_one("TabPane#tab-docker-report").mount(install_report)
+            self.query_one("TabbedContent", TabbedContent).active = "tab-docker-report"
+        except asyncio.CancelledError:
+            _log.write(Text.from_markup("[red]Interrupted!"))
+            await asyncio.sleep(1)
+            raise
+        except PrerequisiteError as e:
+            _log.write(Text.from_markup("[red]:warning: A prerequisite check has failed."))
+            _log.write(e)
+        except Exception as e:
+            _log.write(Text.from_markup("[red]:warning: Unexpected error!"))
+            _log.write(e)
+            _log.write(Traceback())
+        finally:
+            _log.write("")
+            _log.write(Text.from_markup("[bright_cyan]All tasks finished. Press q/Q to exit."))
+            if self._non_interactive:
+                self.app.post_message(Key("q", "q"))
+            current_log.reset(_log_token)
+
+
 class PackageTypeMenu(Static):
     """Sub-menu for selecting package deployment type."""
 
@@ -538,6 +664,7 @@ def __init__(
             self._dist_info = DistInfo()
         self._enabled_menus = set()
         self._enabled_menus.add(InstallModes.PACKAGE)
+        self._enabled_menus.add(InstallModes.DOCKER)
         self._non_interactive = args.non_interactive
         if args.target_path is not None and args.target_path != str(Path.home() / "backendai"):
             self._dist_info.target_path = Path(args.target_path)
@@ -601,9 +728,11 @@ def compose(self) -> ComposeResult:
             # maintain_desc = "Could not find an existing setup (missing INSTALL-INFO)"
             maintain_desc = "Coming soon!"
         configure_desc = "Configure setup variables before installation."
+        docker_desc = "Deploy the published service container images with docker compose"
         mode_desc: dict[InstallModes, str] = {
             InstallModes.DEVELOP: develop_desc,
             InstallModes.PACKAGE: package_desc,
+            InstallModes.DOCKER: docker_desc,
             InstallModes.MAINTAIN: maintain_desc,
             InstallModes.CONFIGURE: configure_desc,
         }
@@ -675,6 +804,16 @@ def start_package_mode(self) -> None:
             li = self.app.query_one("#pkg-type-release", ListItem)
             lv.post_message(ListView.Selected(lv, li, list(lv.children).index(li)))
 
+    @on(ListView.Selected, "#mode-list", item="#mode-docker")
+    def start_docker_mode(self) -> None:
+        if InstallModes.DOCKER not in self._enabled_menus:
+            return
+        self.app.sub_title = "Docker Compose Setup"
+        switcher = self.app.query_one("#top", ContentSwitcher)
+        switcher.current = "docker-setup"
+        docker_setup = self.app.query_one("#docker-setup", DockerSetup)
+        self.app.call_later(docker_setup.begin_install, self._dist_info, self.install_variable)
+
     @on(ListView.Selected, "#mode-list", item="#mode-maintain")
     def start_maintain_mode(self) -> None:
         if InstallModes.MAINTAIN not in self._enabled_menus:
@@ -751,6 +890,7 @@ def compose(self) -> ComposeResult:
                     id="pkg-type-menu",
                 )
                 yield PackageSetup(id="pkg-setup", non_interactive=self._args.non_interactive)
+                yield DockerSetup(id="docker-setup", non_interactive=self._args.non_interactive)
                 yield Configure(id="configure")
         yield Footer()