fix(#50): durable vLLM logs that survive restart + model logs

CHANGELOG.md

@@ -4,6 +4,35 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.23.0] - 2026-06-20
+
+### Added
+
+- **Durable vLLM logs that survive restart/recreate (#50).** When a vLLM
+  container restarted, its `docker logs` — and any EngineCore crash trace — were
+  lost, which blocked root-causing #50 for lack of data. `model init` now
+  scaffolds `mg-logwrap.sh`, bind-mounted as each vLLM service's entrypoint: it
+  tees stdout+stderr to a per-boot file `<service>-<boot>.log` under a
+  host-mounted log dir (`${MODEL_GEAR_LOG_DIR:-<deploy>/logs}` → `/logs/model-gear`),
+  then `exec`s the real command so vLLM stays the signal target (graceful
+  shutdown) and the exit code (and `restart:` policy) are unchanged. Teeing at the
+  process-I/O level captures **both** Python tracebacks and native CUDA/C++ aborts;
+  if logging can't be set up it falls back to a plain `exec` and never blocks
+  serving. The crash boot is preserved as its own file. Wired into the single-model
+  and fleet (`primary`/`embed`/`rerank`) compose templates. See
+  `docs/durable-logs.md`.
+- **`model logs`** — new read-only verb to list/tail the durable logs, reading the
+  host files directly so it works even after the crashed container is gone:
+  `model logs` (list boots), `model logs <service>` (tail latest), and
+  `model logs <service> --previous` (tail the boot that crashed, after a restart).
+
+### Changed
+
+- `model init` / `model serve` / `model fleet up` pre-create the host log dir
+  (user-owned) before compose bind-mounts it, so logs are never root-owned.
+
+### Fixed
+
 ## [0.22.1] - 2026-06-19
 
 ### Fixed

CLAUDE.md

@@ -67,13 +67,13 @@ model_gear/                 # Python package (pip install model-gear)
     ├── _runtime_ops.py     # shared glue (deployment dir, port, compose_check)
     └── _commands/          # one module per verb: register(sub) + handler
         ├── switch.py serve.py stop.py status.py assess.py benchmark.py init.py fleet.py
-        └── tunnel.py whoami.py learn.py explain.py overview.py doctor.py cli.py
+        └── logs.py tunnel.py whoami.py learn.py explain.py overview.py doctor.py cli.py
 ```
 
 **Mutation safety:** write verbs (`switch`, `serve`, `stop`, `init`, `tunnel`) default to
 **dry-run**; require `--apply` to commit. Agents call CLIs in loops, so
 safe-by-default is mandatory. The read-only verbs (`status`, `assess`,
-`benchmark`, `overview`, `whoami`, `explain`, `doctor`) never change the world.
+`benchmark`, `logs`, `overview`, `whoami`, `explain`, `doctor`) never change the world.
 
 ## Build / test / publish
 

docs/durable-logs.md

@@ -0,0 +1,97 @@
+# Durable logs: never lose a crash trace again
+
+When a vLLM container restarts or is recreated, its `docker logs` are gone. That
+is exactly how the EngineCore crash trace in
+[issue #50](https://github.com/agentculture/model-gear/issues/50) vanished before
+anyone could read it: the server 500'd on a tool-calling request, the engine went
+down, and by the time the box was looked at, a restart had wiped the logs — so
+the root cause could not be investigated for lack of data.
+
+model-gear fixes the **observability gap**, not (yet) the crash itself: it makes
+each vLLM service's output durable so the *next* crash leaves a trace you can
+read. Pinning the EngineCore root cause (MTP speculative decoding + tools vs FP4)
+needs a controlled repro **with that durable trace in hand** — durable logs is
+the prerequisite that unblocks it.
+
+## How it works
+
+`model init` scaffolds **`mg-logwrap.sh`** next to `docker-compose.yml`. Each
+vLLM service bind-mounts it as the container **entrypoint**, so the real
+`command:` (the `vllm serve …` arg list, unchanged) arrives at the wrapper as
+`"$@"`. The wrapper:
+
+1. opens a **per-boot** file `<service>-<ISO8601>.log` under the host-mounted log
+   dir and points `<service>-latest.log` at it;
+2. tees `stdout`+`stderr` to that file **and** passes them through to the console
+   (so `docker logs` keeps working too);
+3. `exec`s the real command, so **vLLM stays the signal target** — `docker stop`
+   still drains it gracefully (SIGTERM reaches vLLM, not a shell), and the
+   container's exit code is vLLM's (so `restart: unless-stopped` is unaffected).
+
+Because it tees at the process-I/O level — below Python's logging — it captures
+**both** Python tracebacks **and** native `stderr` aborts (CUDA / C++ / OOM),
+which is the class of crash that most needs investigating and which Python-level
+file logging would miss. If anything about logging fails (no log dir, read-only
+mount, no `bash`), the wrapper falls back to a plain `exec "$@"` — logging can
+never stop the model from serving.
+
+### Paths
+
+| | Host | In container |
+|---|---|---|
+| Log dir | `${MODEL_GEAR_LOG_DIR:-<deploy>/logs}` | `/logs/model-gear` |
+| Single model | `…/logs/vllm-<boot>.log` | `/logs/model-gear/vllm-<boot>.log` |
+| Fleet gears | `…/logs/{primary,embed,rerank}-<boot>.log` | `/logs/model-gear/<svc>-<boot>.log` |
+
+The host dir is created (user-owned) by `model init`, `model serve`, and
+`model fleet up` before compose bind-mounts it, so the logs are never
+root-owned. Per-boot files mean the **crash boot is preserved as its own file**
+and never overwritten by the restart that follows it.
+
+## Reading the logs — `model logs`
+
+`model logs` is read-only and reads the **host** files directly, so it works even
+after the crashed container is gone (`docker logs` would not):
+
+```text
+model logs                 # list per-boot files (newest first) + the log dir
+model logs vllm            # tail the latest boot for a service
+model logs vllm --previous # tail the boot BEFORE the latest — i.e. the crashed
+                           #   boot, after a restart created a fresh healthy one
+model logs primary -n 200  # more lines (fleet service)
+model logs --json          # structured listing
+```
+
+The `--previous` flag is the #50 investigation path: after a crash+restart, the
+latest boot is the healthy one — `--previous` tails the boot that actually
+crashed.
+
+### Pruning
+
+Per-boot files accumulate across restarts. They are plain files under the host
+log dir; prune old ones with a one-liner, e.g. keep the newest 20 per service:
+
+```bash
+ls -1t <deploy>/logs/vllm-*.log | tail -n +21 | xargs -r rm
+```
+
+## Why not OTEL?
+
+OpenTelemetry was considered first. vLLM's OTEL support is **traces-only**
+(`--otlp-traces-endpoint`, request spans) — it has **no native OTLP log export**,
+and a crash traceback is not a span (the engine dies), so OTEL tracing would not
+capture the very thing #50 needs. Capturing logs via OTEL would require an OTEL
+Collector + `filelog` receiver sidecar reading the same `stderr` plus a backend
+to store it — significant new infrastructure for no gain over a host file. So
+crash durability is done at the file level; OTEL **traces** remain a future
+opt-in for *request* observability (latency/token spans), a separate concern from
+crash logs.
+
+## Scope
+
+- Wrapped: the vLLM generate/embed/rerank services (single-model `vllm`, fleet
+  `primary` / `embed` / `rerank`).
+- Not changed: the model's serving flags/behaviour, the `restart:` policy, or the
+  healthcheck. No docker-socket mounts, no new runtime dependencies. Auto-restart
+  / autoheal is intentionally **out of scope** here (see #50) — this PR makes the
+  crash investigable; recovery is a separate decision.

docs/gateway-fleet.md

@@ -52,6 +52,11 @@ The backends are reachable only on the compose network (`http://vllm-primary:800
 gateway needs no Docker socket access — compose owns the lifecycle; the gateway
 only routes.
 
+Each vLLM gear runs through `mg-logwrap` so its output (and any crash trace)
+persists to per-boot files under the host log dir and **survives restart/recreate** —
+read them with `model logs {primary,embed,rerank}` even after a container is gone.
+See [docs/durable-logs.md](durable-logs.md) (issue #50).
+
 ### Adding a fallback
 
 The gateway adds a second backend **only** when `FALLBACK_URL` or

model_gear/cli/__init__.py

@@ -69,6 +69,7 @@ def _build_parser() -> argparse.ArgumentParser:
     from model_gear.cli._commands import fleet as _fleet_cmd
     from model_gear.cli._commands import init as _init_cmd
     from model_gear.cli._commands import learn as _learn_cmd
+    from model_gear.cli._commands import logs as _logs_cmd
     from model_gear.cli._commands import overview as _overview_cmd
     from model_gear.cli._commands import serve as _serve_cmd
     from model_gear.cli._commands import status as _status_cmd
@@ -99,6 +100,7 @@ def _build_parser() -> argparse.ArgumentParser:
     _benchmark_cmd.register(sub)
     _init_cmd.register(sub)
     _fleet_cmd.register(sub)
+    _logs_cmd.register(sub)
     _tunnel_cmd.register(sub)
 
     # Agent-first / introspection verbs (sibling rubric).

model_gear/cli/_commands/fleet.py

@@ -24,7 +24,7 @@
 from model_gear import assess
 from model_gear.cli import _runtime_ops
 from model_gear.cli._output import emit_diagnostic, emit_result
-from model_gear.runtime import _compose, _health
+from model_gear.runtime import _compose, _env, _health
 
 _UNSET = "(unset)"
 _JSON_HELP = "Emit structured JSON."
@@ -58,6 +58,8 @@ def cmd_fleet_up(args: argparse.Namespace) -> int:
         emit_result(payload if json_mode else msg, json_mode=json_mode)
     else:
         emit_diagnostic(f">> building + starting the fleet in {deploy_dir}")
+        # Ensure the durable-log dir exists (user-owned) before compose bind-mounts it.
+        _compose.ensure_log_dir(deploy_dir, _env.read_env(env_path, _compose.LOG_DIR_ENV) or None)
         _runtime_ops.compose_check(
             _compose.compose_up_build(deploy_dir), "docker compose up -d --build"
         )

model_gear/cli/_commands/init.py

@@ -55,6 +55,10 @@ def _emit_dry_run(target: Path, fleet: bool, audio: bool, json_mode: bool) -> No
 
 def _emit_apply(target: Path, fleet: bool, audio: bool, force: bool, json_mode: bool) -> None:
     written = _compose.write_scaffold(target, force=force, templates=_templates(fleet, audio))
+    # Create the durable-log dir now (as the invoking user) so the compose bind-mount
+    # source exists before `model serve` / `fleet up` — otherwise Docker makes it
+    # root-owned. The mg-logwrap entrypoint writes per-boot logs here (issue #50).
+    _compose.ensure_log_dir(target)
     if fleet:
         # Pin the gateway image to the model-gear release that scaffolded this.
         _env.set_env(target / _compose.ENV_FILE, "MODEL_GEAR_VERSION", __version__)