feat: self-healing watchdog, Claude login from the dashboard, claude-code diagnostics

Auto-heal: hub watchdog health-checks every agent. On a down agent it auto-restarts it (rate-limited,
never starts services that were never up); if the restart fails it pulls the log tail and asks a
healthy agent (opencode, else brain) to diagnose the root cause. Events at GET /api/health_events and
in Status -> Self-healing log. Config: watchdog.enabled/interval. Hub-triggered restarts set
AIOS_NO_UPDATE_CHECK=1 so a heal never triggers a git pull.

Claude login in the dashboard: hub runs the Claude CLI login as an interactive session (pty on
Linux/WSL, pipes on Windows), streams its output, extracts the authorize URL, and accepts the code
back. Endpoints GET/POST /api/claude_login (start|input|stop). Settings now has a two-step flow with
a live auth status line.

claude-code fix: it was working all along — the auth probe used a 45s timeout while the claude CLI
cold-starts slowly (it waits 3s on stdin), so it falsely reported 'not logged in'. Timeout 45s->180s,
plus GET /api/claude_status reporting service_up/version/logged_in/model/error.

Verified on Windows: killed crewai's real listener -> watchdog logged 'down', restarted it, logged
'healed', service back up. Login session spawns/streams/exits (2.1.202) and stops. claude_status
returns logged_in:true, model claude-sonnet-4-5, and a precise error when claudecode is stopped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: ZD Studios

README.md

@@ -167,7 +167,8 @@ Plus two integrations that glue it together:
 - **✦ Team** — one assistant that orchestrates the whole team: the Brain plans, delegates subtasks to the specialist agents, and synthesizes one answer (the practical "merge").
 - **Configure openclaw AND hermes fully inside the hub** — both control-UIs are embedded via frame-stripping proxies, so you get channels, **connectors**, model providers, **MCP servers**, skills, plugins, **automations/cron**, and sessions right in the hub (they normally block embedding).
 - **Automations** — schedule prompts to run against any agent every N minutes (daily digests, checks).
-- **Connect your Claude account (Pro/Max)** — first run `aios claude-login` (opens the browser OAuth for your subscription), then one click in **Settings → "Use my Claude subscription"** routes the Brain/Team/crews through **claude-code** (no API key, no per-token cost). The button tests a real completion and tells you if you still need to log in.
+- **Log in to Claude from the dashboard** — **Settings → "1 · Log in to Claude"** runs the Claude CLI login *through the hub*: it shows the authorize link, you approve in the browser and paste the code back, all in the UI. Then **"2 · Use my Claude subscription"** routes the Brain/Team/crews through **claude-code** (no API key, no per-token cost). A live status line shows whether claude-code is up and actually authenticated. (Terminal equivalent: `aios claude-login`.)
+- **🛡️ Self-healing agents** — a watchdog in the hub health-checks every agent. If one stops responding it's **automatically restarted**; if the restart fails, a **healthy agent reads its logs and diagnoses the cause**. See the incident log in **Status → Self-healing log** (`watchdog.enabled` in `aios.config.yaml`).
 - **Settings** — edit provider/key/model, channel tokens, and `aios.config.yaml`; it re-renders into every agent, no terminal needed.
 - **Themes** — Light, Dark, Midnight, Slate, Rose. Chat renders markdown (headers, bullets, code blocks).
 

aios.config.example.yaml

@@ -54,6 +54,10 @@ updates:
   check_on_start: true     # on `aios start`, check the repo for updates and notify
   auto_update: true        # on `aios start`, auto git-pull + reinstall if the repo changed
 
+watchdog:
+  enabled: true            # if an agent stops responding, auto-restart it
+  interval: 45             # seconds between health sweeps; a healthy agent diagnoses failures
+
 # Health-check URLs aios waits on during `start` and checks in `status`/`smoke`.
 health:
   opencode: http://127.0.0.1:4096/

aios.py

@@ -467,6 +467,9 @@ def service_specs(cfg: dict) -> dict:
             "AIOS_HUB_PORT": str(hport),
             "AIOS_OPENCODE_DIR": str(oc / "packages" / "opencode") if oc else "",
             "AIOS_BUN": find_tool("bun") or "",
+            # watchdog: auto-restart downed agents, then have an agent diagnose failures
+            "AIOS_WATCHDOG": "1" if cfg_get(cfg, "watchdog.enabled", True) else "0",
+            "AIOS_WATCHDOG_INTERVAL": str(cfg_get(cfg, "watchdog.interval", 45)),
         },
     }
     return specs
@@ -1203,12 +1206,13 @@ def wire_openclaw_os(quiet=False):
 def cmd_start(args):
     cfg = load_config()
     secrets = load_env(ENV_PATH)
-    if cfg_get(cfg, "updates.auto_update", False):
+    _no_upd = os.environ.get("AIOS_NO_UPDATE_CHECK") == "1"  # set by the hub's watchdog
+    if not _no_upd and cfg_get(cfg, "updates.auto_update", False):
         if _git_pull() is True:  # new commits arrived → refresh deps/config
             _install_all(cfg)
             render_native(cfg, secrets)
             cfg = load_config()
-    elif cfg_get(cfg, "updates.check_on_start", True):
+    elif not _no_upd and cfg_get(cfg, "updates.check_on_start", True):
         _check_updates_notice()
     if not secrets.get("AIOS_LLM_API_KEY") and not any(secrets.get(v) for v in PROVIDER_VAR.values()):
         warn("no model API key set — the stack runs, but agents need a key to answer "

aios_hub.py

@@ -73,6 +73,7 @@ def run_aios(*a, background=False):
     env = dict(os.environ)
     for k in ("AIOS_LLM_PROVIDER", "AIOS_LLM_API_KEY", "AIOS_LLM_BASE_URL", "AIOS_DEFAULT_MODEL"):
         env.pop(k, None)
+    env["AIOS_NO_UPDATE_CHECK"] = "1"  # never git-pull during a hub-triggered restart
     if background:
         threading.Thread(target=lambda: subprocess.run(cmd, cwd=str(ROOT), env=env,
                          capture_output=True), daemon=True).start()
@@ -392,6 +393,196 @@ def dry_run(target: str, message: str, history: list | None = None) -> dict:
             "free": free, "priced": priced, "est_cost_usd": round(cost, 6), "note": note}
 
 
+# --------------------------------------------------------------------------- #
+# Watchdog — if an agent goes down, restart it; if that fails, an agent debugs #
+# --------------------------------------------------------------------------- #
+HEALTH_FILE = ROOT / ".aios" / "health_events.json"
+WATCH = ["opencode", "hermes", "openclaw", "crewai", "claudecode"]
+_wd = {"seen_up": {}, "last_restart": {}}
+
+
+def _log_event(kind: str, svc: str, msg: str):
+    ev = _load_json(HEALTH_FILE, [])
+    ev.insert(0, {"ts": time.time(), "kind": kind, "service": svc, "message": str(msg)[:2000]})
+    _save_json(HEALTH_FILE, ev[:60])
+
+
+def _service_logs(svc: str, n: int = 40) -> str:
+    return (run_aios("logs", svc, "-n", str(n)).get("out") or "")[-3000:]
+
+
+def _diagnose(svc: str, logtail: str) -> str:
+    """Ask a healthy agent to debug the crashed one."""
+    q = (f"The AI OS service '{svc}' stopped responding and an automatic restart did not fix it.\n"
+         f"Log tail:\n\n{logtail}\n\n"
+         "In at most 3 bullets: the likely root cause, and the exact command to fix it.")
+    try:
+        helper = "opencode" if ping(PEERS.get("opencode", "")) else "brain"
+        return route(helper, q)
+    except Exception as e:
+        return f"(diagnosis unavailable: {e})"
+
+
+def _watchdog_loop():
+    if os.environ.get("AIOS_WATCHDOG", "1") != "1":
+        return
+    interval = int(os.environ.get("AIOS_WATCHDOG_INTERVAL", "45"))
+    time.sleep(20)  # let the stack finish booting before we judge it
+    while True:
+        try:
+            for svc in WATCH:
+                url = PEERS.get(svc)
+                if not url:
+                    continue
+                if ping(url):
+                    _wd["seen_up"][svc] = time.time()
+                    continue
+                # Only heal services we've actually seen alive (never auto-start disabled ones)
+                if not _wd["seen_up"].get(svc):
+                    continue
+                if time.time() - _wd["last_restart"].get(svc, 0) < 180:
+                    continue  # rate-limit: no restart storms
+                _wd["last_restart"][svc] = time.time()
+                _log_event("down", svc, f"{svc} stopped responding — auto-restarting…")
+                run_aios("restart", svc)
+                time.sleep(20)
+                if ping(url):
+                    _log_event("healed", svc, f"{svc} is back up (automatic restart).")
+                else:
+                    diag = _diagnose(svc, _service_logs(svc))
+                    _log_event("failed", svc, f"Restart didn't fix {svc}. Agent diagnosis:\n{diag}")
+        except Exception:
+            pass
+        time.sleep(interval)
+
+
+# --------------------------------------------------------------------------- #
+# Claude login from the dashboard (interactive CLI session over HTTP)          #
+# --------------------------------------------------------------------------- #
+class _LoginSession:
+    proc = None
+    master = None
+    out = ""
+    lock = threading.Lock()
+
+
+_login = _LoginSession()
+_ANSI = __import__("re").compile(r"\x1b\[[0-9;?]*[A-Za-z]|\x1b\][^\x07]*\x07|\r")
+
+
+def _claude_bin():
+    return shutil.which("claude") or shutil.which("claude.cmd")
+
+
+def claude_login_start(mode: str = "setup-token") -> dict:
+    claude = _claude_bin()
+    if not claude:
+        return {"ok": False, "error": "`claude` CLI not found on PATH."}
+    if _login.proc and _login.proc.poll() is None:
+        return {"ok": True, "already": True}
+    with _login.lock:
+        _login.out = ""
+    args = [claude] + ([mode] if mode else [])
+    try:
+        if os.name != "nt":
+            import pty
+            m, s = pty.openpty()
+            _login.master = m
+            _login.proc = subprocess.Popen(args, stdin=s, stdout=s, stderr=s,
+                                           env=dict(os.environ), close_fds=True)
+            os.close(s)
+
+            def _rd():
+                while True:
+                    try:
+                        d = os.read(m, 4096)
+                    except OSError:
+                        break
+                    if not d:
+                        break
+                    with _login.lock:
+                        _login.out += d.decode(errors="replace")
+            threading.Thread(target=_rd, daemon=True).start()
+        else:
+            _login.master = None
+            _login.proc = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                                           stderr=subprocess.STDOUT, text=True, bufsize=1,
+                                           env=dict(os.environ))
+
+            def _rd():
+                for line in _login.proc.stdout:
+                    with _login.lock:
+                        _login.out += line
+            threading.Thread(target=_rd, daemon=True).start()
+        return {"ok": True}
+    except Exception as e:
+        return {"ok": False, "error": str(e)}
+
+
+def claude_login_input(text: str) -> dict:
+    if not (_login.proc and _login.proc.poll() is None):
+        return {"ok": False, "error": "no login session running"}
+    try:
+        data = (text or "") + "\n"
+        if _login.master is not None:
+            os.write(_login.master, data.encode())
+        else:
+            _login.proc.stdin.write(data)
+            _login.proc.stdin.flush()
+        return {"ok": True}
+    except Exception as e:
+        return {"ok": False, "error": str(e)}
+
+
+def claude_login_state() -> dict:
+    with _login.lock:
+        raw = _login.out[-8000:]
+    running = bool(_login.proc and _login.proc.poll() is None)
+    urls = __import__("re").findall(r"https?://[^\s\"'<>]+", raw)
+    return {"output": _ANSI.sub("", raw), "running": running,
+            "exit": (_login.proc.poll() if _login.proc else None),
+            "url": urls[-1] if urls else ""}
+
+
+def claude_login_stop() -> dict:
+    if _login.proc and _login.proc.poll() is None:
+        try:
+            _login.proc.kill()
+        except Exception:
+            pass
+    return {"ok": True}
+
+
+def claude_status() -> dict:
+    """Is claude-code-api up, and is the Claude CLI actually authenticated?"""
+    st = {"service_up": False, "version": "", "logged_in": False, "model": "", "error": ""}
+    try:
+        h = json.loads(urllib.request.urlopen(CLAUDECODE + "/health", timeout=6).read())
+        st["service_up"] = True
+        st["version"] = h.get("claude_version", "")
+    except Exception as e:
+        st["error"] = f"claude-code-api not reachable — run `aios start claudecode` ({e})"
+        return st
+    model = "claude-sonnet-4-5"
+    try:
+        ids = _models_from(CLAUDECODE + "/v1")
+        if ids:
+            model = next((i for i in ids if "sonnet" in i.lower()), ids[0])
+    except Exception:
+        pass
+    st["model"] = model
+    try:  # cold start of the claude CLI is slow — be generous
+        b = json.dumps({"model": model, "messages": [{"role": "user", "content": "hi"}],
+                        "max_tokens": 16}).encode()
+        rq = urllib.request.Request(CLAUDECODE + "/v1/chat/completions", data=b, method="POST",
+                                    headers={"Content-Type": "application/json"})
+        urllib.request.urlopen(rq, timeout=180)
+        st["logged_in"] = True
+    except Exception as e:
+        st["error"] = f"Claude CLI not authenticated (or timed out): {str(e)[:200]}"
+    return st
+
+
 # --------------------------------------------------------------------------- #
 # "Team" — a single agent that orchestrates the others (the practical merge)   #
 # --------------------------------------------------------------------------- #
@@ -560,6 +751,12 @@ def do_GET(self):
             self._send(200, {"usage": u, "total": sum(u.values()), "free_requests": free})
         elif self.path == "/api/ui":
             self._send(200, {"widgets": _load_json(WIDGETS_FILE, [])})
+        elif self.path == "/api/health_events":
+            self._send(200, {"events": _load_json(HEALTH_FILE, [])})
+        elif self.path == "/api/claude_status":
+            self._send(200, claude_status())
+        elif self.path == "/api/claude_login":
+            self._send(200, claude_login_state())
         elif self.path in ("/v1/models", "/api/v1/models"):
             # AIOS as an OpenAI-compatible API: its "models" are the chat targets.
             now = int(time.time())
@@ -639,7 +836,7 @@ def do_POST(self):
                     rq = urllib.request.Request(CLAUDECODE + "/v1/chat/completions", data=b, method="POST",
                                                 headers={"Content-Type": "application/json",
                                                          "Authorization": "Bearer sk-aios-claudecode"})
-                    urllib.request.urlopen(rq, timeout=45)
+                    urllib.request.urlopen(rq, timeout=180)  # claude CLI cold start is slow
                     logged_in = True
                 except Exception as e:
                     detail = str(e)[:160]
@@ -649,6 +846,16 @@ def do_POST(self):
                          "Pointed at claude-code, but the Claude CLI isn't logged in yet. Run "
                          "`aios claude-login` (or `claude setup-token`) in a terminal to authorize your "
                          "Pro/Max account, then chat.")})
+        elif self.path == "/api/claude_login":
+            op = payload.get("op", "start")
+            if op == "start":
+                self._send(200, claude_login_start(payload.get("mode", "setup-token")))
+            elif op == "input":
+                self._send(200, claude_login_input(payload.get("text", "")))
+            elif op == "stop":
+                self._send(200, claude_login_stop())
+            else:
+                self._send(400, {"error": "unknown op"})
         elif self.path == "/api/dryrun":
             self._send(200, dry_run(payload.get("target", "brain"), payload.get("message", ""),
                                     payload.get("history", [])))
@@ -752,6 +959,7 @@ def do_POST(self):
 
 def main():
     threading.Thread(target=_scheduler_loop, daemon=True).start()
+    threading.Thread(target=_watchdog_loop, daemon=True).start()  # auto-heal downed agents
     # Bind all interfaces by default so the Windows browser can reach it over WSL.
     host = os.environ.get("AIOS_HUB_HOST", "0.0.0.0")
     srv = ThreadingHTTPServer((host, PORT), Handler)