doc(BA-7272): correct privilege matrix, fixtures mount, and krunner share location per review

Author: kyujin-cho

docker/README.md

@@ -3,8 +3,8 @@
 The `backend.ai-*` dockerfiles in this directory are built and published to
 Docker Hub for every release tag by `.github/workflows/docker-images.yml`
 (matrix from `scripts/list-dockerfiles.sh --service`), multi-arch
-(`linux/amd64` + `linux/arm64`). The remaining dockerfiles are internal build
-tooling and are not published.
+(`linux/amd64` + `linux/arm64`). The remaining dockerfiles are runtime helper
+images and are not published.
 
 ## Published images
 
@@ -23,7 +23,7 @@ tooling and are not published.
 | Tag | Meaning |
 |---|---|
 | `<version>` (e.g. `26.9.0`, `26.9.0rc1`) | The normalized package version of the release tag that built the image |
-| `latest` | The most recent **final** release only — never moved by rc/alpha/beta releases |
+| `latest` | Applied to **any** final (non-prerelease) release — never moved by rc/alpha/beta releases. This includes hotfixes cut from older release branches, so `latest` can move *backwards*; pin explicit versions in production |
 
 All images take the same build-arg contract: `PYTHON_VERSION` (from
 `pants.toml`) and `PKGVER` (normalized `VERSION`), and install the release
@@ -37,6 +37,9 @@ crashes on the `triggered_user` metadata field added in 26.8).
 
 ## Infra images (not published)
 
+Runtime helper images loaded on demand by the agent from bundled archives —
+not published to Docker Hub.
+
 | Dockerfile | Role |
 |---|---|
 | `krunner-extractor.dockerfile` | Extracts kernel-runner archives during agent operation |
@@ -50,33 +53,34 @@ path the image's default command reads:
 
 | Service | Config mount target | Notes |
 |---|---|---|
-| manager | `/etc/backend.ai/manager.toml` | also mount `fixtures/` at `/app/fixtures:ro` (initial DB fixtures) |
+| manager | `/etc/backend.ai/manager.toml` | also mount `fixtures/` at `/app/fixtures` **read-write** — manager RPC keypair (auto-generated at first start) + DB fixtures |
 | agent | `/etc/backend.ai/agent.toml` | see the privilege and path-parity sections below |
 | webserver | `/etc/backend.ai/webserver.conf` | note the `.conf` target name, not `.toml` |
-| storage-proxy | `/etc/backend.ai/storage-proxy.toml` | runs unprivileged — set `user:` to the vfroot owner UID:GID; mount TLS material at `/app/ssl:ro` if enabled |
+| storage-proxy | `/etc/backend.ai/storage-proxy.toml` | to run unprivileged, set the `user`/`group` knobs in `storage-proxy.toml` (the daemon drops privileges itself) rather than compose `user:` — the chown watcher requires *starting* as root; if TLS is enabled, mount the cert material read-only at whatever path `ssl-cert`/`ssl-privkey` point to |
 | appproxy-coordinator | `/etc/backend.ai/proxy-coordinator.toml` | |
-| appproxy-worker | `/etc/backend.ai/proxy-worker.toml` | one container per worker: each needs its OWN toml with a unique `authority`, protocol (`http`/`tcp`), `api_bind_addr` port, and non-overlapping `bind_port_range` — and the compose port mappings must match |
+| appproxy-worker | `/etc/backend.ai/proxy-worker.toml` | one container per worker: each needs its OWN toml with a unique `authority`, protocol (`http`/`tcp`), `api_bind_addr` port, and a non-overlapping `[proxy_worker.port_proxy] bind_port_range` (port-based frontends only) — and the compose port mappings must match |
 
 Shared prerequisites:
 
 | Item | Used by | Why |
 |---|---|---|
 | halfstack services (PostgreSQL, Valkey/Redis, etcd) | all | the reference definitions live in `docker-compose.halfstack-main.yml` |
 | `supergraph.graphql` + a GraphQL gateway (e.g. `ghcr.io/graphql-hive/gateway`) | GraphQL federation | the supergraph schema is generated per release (`scripts/generate-graphql-schema.sh`); the gateway composes manager subgraphs |
-| `/etc/machine-id` bind mount | manager, agent | stable node identity |
-| `wheelhouse/` mount at `/app/wheelhouse` (optional) | manager, agent | staging directory for extra plugin wheels (e.g. accelerator plugins) installed into the container on top of the base image |
+| RPC auth key distribution | manager, agent | the agent needs the manager's RPC **public** key to authenticate RPC calls — e.g. share the parity-mounted fixtures directory across nodes, or mount a common key directory at `/etc/backend.ai/keys:ro` |
+| `wheelhouse/` mount at `/app/wheelhouse` (optional) | manager, agent | an operator convention only — nothing in the images consumes it automatically; to add extra plugin wheels (e.g. accelerator plugins), the operator must `docker exec <container> pip install /app/wheelhouse/*.whl` or build a derived image |
 
 ## Container privileges
 
 Most services run fine with compose defaults (bridge network, config file
-bind-mounted read-only). The manager and the agent need more; grant each item
-consciously — together they amount to root-equivalent control of the host.
+bind-mounted read-only). Only the **agent** needs real host privileges; the
+manager needs at most the Docker socket. Grant each item consciously —
+together they amount to root-equivalent control of the host.
 
 | Requirement | manager | agent | Why |
 |---|---|---|---|
-| `network_mode: host` | ✅ | ✅ | Kernel↔agent ZMQ/service ports and agent RPC are advertised on host addresses; kernels spawned on the host network must reach them |
-| `privileged: true` | ✅ | ✅ | Container/device management against the host daemon; sysfs reads for metrics |
-| `/var/run/docker.sock` bind mount | ✅ | ✅ | DooD: containers are created by talking to the **host** Docker daemon |
+| `network_mode: host` | optional | ✅ | Agent: kernel↔agent ZMQ/service ports and agent RPC are advertised on host addresses; kernels spawned on the host network must reach them. Manager: convenience only — the bridge alternative works via the `announce-addr` / `announce-internal-addr` knobs |
+| `privileged: true` | — | ✅ | Agent: container/device management against the host daemon; sysfs reads for metrics. The manager does not need it — the Docker socket alone suffices for its (conditional) Docker use |
+| `/var/run/docker.sock` bind mount | conditional | ✅ | DooD: containers are created by talking to the **host** Docker daemon. Manager: only when the `local` container registry is used |
 | `pid: host` | — | ✅ | Host PID namespace visibility: the agent inspects and signals kernel processes by host PID |
 | `cgroup: host` (host cgroup namespace) | — | ✅ | **Required, not optional** — see below |
 | Host `/sys` visibility | — | ✅ | Container resource metrics are read from the host cgroupfs/sysfs (follows automatically from the host cgroup namespace) |
@@ -89,6 +93,7 @@ The agent's host-PID→container-PID translation
 (`host_pid_to_container_pid` in `src/ai/backend/agent/utils.py`, via
 `src/ai/backend/common/cgroup.py`) parses `/proc/<pid>/cgroup` expecting
 **host-rooted** paths (`docker/<id>` or `system.slice/docker-<id>.scope`) and
+reads hardcoded `/sys/fs/cgroup/...` paths; the metrics path separately
 resolves the cgroupfs mount point from `/proc/mounts`. In a private cgroup
 namespace, sibling-container paths are not host-rooted and the mounted cgroupfs
 is namespaced — PID translation and sysfs metrics both break.
@@ -108,40 +113,46 @@ the identical location:
 | Config knob (`agent.toml`) | Reference value | Used for |
 |---|---|---|
 | `[container] scratch-root` | `/var/lib/backend.ai/scratches` | Scratch roots of kernel containers |
-| `[agent] ipc-base-path` | `/tmp/backend.ai/ipc` | Agent↔kernel IPC sockets |
+| `[agent] ipc-base-path` | `/tmp/backend.ai/ipc` (this document's chosen value; `configs/agent/sample.toml` ships `/var/run/backend.ai/ipc`) | Agent↔kernel IPC sockets |
 | `[agent] var-base-path` | `/var/lib/backend.ai` | Plugin state bind-mounted into kernels (e.g. accelerator hook caches) |
-| — (fixed path) | `/tmp/backend-ai-krunner` | Kernel-runner files: the image entrypoint copies them here so the host daemon can mount them into kernels; **without this mount kernel creation fails** with `bind source path does not exist` (the entrypoint logs a warning at startup) |
+| env `BACKENDAI_KRUNNER_SHARED` | `/var/lib/backend.ai/krunner` | Kernel-runner files: the image entrypoint copies them here so the host daemon can mount them into kernels. Covered automatically by the `/var/lib/backend.ai` parity mount; the entrypoint **refuses to start** without it — override the path via the `BACKENDAI_KRUNNER_SHARED` env var |
 
-With the reference values, three parity mounts cover everything:
-`/var/lib/backend.ai`, `/tmp/backend.ai`, and `/tmp/backend-ai-krunner`.
+With the reference values, two parity mounts cover everything:
+`/var/lib/backend.ai` and `/tmp/backend.ai`.
 
 Vfolder roots (e.g. `/vfroot/local/volume1`) follow the same rule on the
 **storage-proxy**: mount each volume at the identical absolute path on host and
 in the storage-proxy container, so the kernel bind-mount sources it reports
-resolve on the host. The agent container itself does not need the vfroot mount.
+resolve on the host. The agent container itself does not need the vfroot mount
+— unless the agent itself performs the volume mounting
+(`cohabiting-storage-proxy = false`), in which case its `mount-path` must be a
+**shared-propagation** bind mount (`bind-propagation: rshared`) so host-side
+mounts become visible inside the container. Also, `scratch-type = "memory"` is
+unsupported in the containerized agent — a tmpfs mounted inside the container's
+namespace is invisible to the host daemon — use `hostdir`.
 
 ## Reference compose file
 
-`docker-compose.monorepo.yml` at the repository root composes the service
-images and assumes the halfstack dependencies from
-`docker-compose.halfstack-main.yml`. The fragment below shows the full working
-shape of the two elevated services, verified against a live deployment:
+`docker-compose.monorepo.yml` at the repository root is a **partial, legacy
+example** — it uses different image names, includes no agent or storage-proxy,
+and runs on a bridge network. The fragment below is the authoritative
+reference for the two elevated services, verified against a live deployment.
+Replace `<version>` with a tag from the tagging scheme above. The `cgroup:`
+key requires Docker Compose v2.15+.
 
 ```yaml
 services:
   manager:
-    image: lablup/backend.ai-manager:26.4.4
-    network_mode: host
-    privileged: true
+    image: lablup/backend.ai-manager:<version>
+    network_mode: host    # optional — bridge works via the announce-addr knobs
     volumes:
-      - /var/run/docker.sock:/var/run/docker.sock
-      - /etc/machine-id:/etc/machine-id
+      - /var/run/docker.sock:/var/run/docker.sock   # only when the `local` container registry is used
       - ./manager.toml:/etc/backend.ai/manager.toml:ro
-      - ./fixtures:/app/fixtures:ro
+      - ./fixtures:/app/fixtures    # read-write: the entrypoint writes the RPC keypair here
     restart: unless-stopped
 
   agent:
-    image: lablup/backend.ai-agent:26.4.4
+    image: lablup/backend.ai-agent:<version>
     network_mode: host
     privileged: true
     pid: host
@@ -155,16 +166,15 @@ services:
               capabilities: [gpu]
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
-      - /etc/machine-id:/etc/machine-id
       - ./agent.toml:/etc/backend.ai/agent.toml:ro
       # path-parity mounts: host path == container path
+      # (the krunner share /var/lib/backend.ai/krunner is covered by the /var/lib/backend.ai mount)
       - /var/lib/backend.ai:/var/lib/backend.ai
       - /tmp/backend.ai:/tmp/backend.ai
-      - /tmp/backend-ai-krunner:/tmp/backend-ai-krunner
     restart: unless-stopped
 ```
 
-The agent entrypoint builds its krunner symlink farm **at container start**, so
-after adding or changing the parity mounts, recreate the container
-(`docker compose up -d --force-recreate agent`) — a restart of the old
-container is not enough.
+Bind mounts are fixed at container **creation**, so after adding or changing
+the parity mounts, recreate the container
+(`docker compose up -d --force-recreate agent`) — the entrypoint does re-run
+on a plain restart, but the old container's mounts cannot change.