Skip to content

Commit 917efb0

Browse files
authored
Device heartbeat reads TensorFold's /health and /metrics (#143)
IAK now reads the live status of each TensorFold server in the model registry and puts it on the device card under one `tensorfold` key, so CodeWatch can show a card per machine. Opt in with INTENT_TENSORFOLD_REGISTRY=<path to models.json> on uik-daemon or `ide-agent-kit intent daemon`; INTENT_TENSORFOLD_SCOPE=local (default) or all. Read with GET /health, GET /metrics and GET /v1/slide/graph only, never a completion. Runs on its own timer at the heartbeat interval and the beat reads the last snapshot, so a dead server costs a stale card and never a late heartbeat. Unreachable is reachable:false with the reason; an unreadable registry is an error, never an empty list. tok/s comes from generation_tokens_total deltas between reads. Living Weights publishes fact counts only, never fact text. version, backend and device are read from /health when present and null otherwise. Registry: `engine` ("tensorfold" marks an entry, any other name skips it, absent means auto-detect from /metrics or the /health shape). A key is sent only to an entry marked TensorFold that names its own keyFile, only on the two gated routes; the fleet-wide LLM_API_KEY fallbacks are not used for a status read (resolveEntryToken gains entryOnly). New `ide-agent-kit tensorfold-status [--json]` prints the same object. Requests carry Connection: close: a CLI run that exited with a kept-alive socket left a ConnectionResetError traceback in the 0.6.0 server log. Tests: 25 new, fixtures recorded from the live 0.6.0 server on 8091, plus 401 /metrics, 404 slide graph, unreachable, hung socket, redirect and heartbeat payload cases. Co-authored-by: ThinkOffApp <233579950+ThinkOffApp@users.noreply.github.com>
1 parent ffe31a7 commit 917efb0

16 files changed

Lines changed: 1684 additions & 5 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
## Unreleased
44

55
### Added
6+
- **TensorFold status on the device heartbeat**: `uik-daemon` and `ide-agent-kit intent daemon` can publish the live status of each TensorFold server in the model registry under the device's `tensorfold` key, so CodeWatch can show a card per machine. Opt in with `INTENT_TENSORFOLD_REGISTRY=<path to models.json>`; `INTENT_TENSORFOLD_SCOPE=local` (default, loopback entries only) or `all`. Read with GET `/health`, `/metrics` and `/v1/slide/graph` only, never a completion. Publishes model, warming, version/backend/device when the server reports them, memory, running and waiting requests, token counters, tok/s from counter deltas between reads, draft acceptance, TTFT p50, KV cache use, Living Weights fact counts (never fact text) and the metrics auth state. An unreachable server is `reachable: false`; an unreadable registry is an `error`, never an empty list. New `ide-agent-kit tensorfold-status [--json]` prints the same object. The registry gains `engine` (`"tensorfold"` marks an entry, any other name skips it, absent means auto-detect). A key is sent only to an entry marked TensorFold that names its own `keyFile`, and only on the two gated routes; the fleet-wide `LLM_API_KEY` fallbacks are never used for a status read.
67
- **What is still waiting on you** (`bin/iak-pending.mjs`): asks every daemon the fleet knows about - this machine's plus every `watchdog-roster.json` entry with a `gate` - for its pending intents and prints them oldest first with the age in the leading column. Motivation: confirmation cards post to the room correctly and then scroll away; on 2026-09-20 four approval requests sat pending on one machine for up to 9 hours 12 minutes and the owner asked twice in one night what was waiting on him, because no surface anywhere could answer it. A host that could not be reached is reported under its own `COULD NOT ASK` heading and is never folded into "nothing pending"; a gate naming another machine by LAN IP is refused rather than probed and reported the same way. A daemon whose rows carry no `status` this version understands is a shape mismatch rather than an empty queue, and a fleet roster that exists but cannot be parsed or read exits `2` instead of silently shrinking the fleet to this machine (only a *missing* roster means "single-machine install"). An age that cannot be believed - missing, future-dated, or epoch-seconds - renders as `age unknown`, sorts to the top and survives `--older-than`, rather than being clamped to "fresh". Exit codes: `0` all clear, `10` pending, `11` some hosts unreachable, `12` no host reachable, `2` cannot run, precedence `12 > 11 > 10 > 0`; `--older-than` narrows what is printed and announced but can never produce a `0`. `--json` for machines. `--room` requires `--older-than` and announces each item at most once per escalation (threshold, then each doubling of its age), keeping a ledger at `~/.iak/pending-announced.json` that is written atomically before the post, carries forward entries for hosts it could not ask, rolls back when a post fails, and degrades to silence - never a storm - when it is missing, corrupt or unwritable. `examples/pending-launchd.plist` wires it to a timer. The gate token is resolved by the existing `resolveGateToken` helper, is sent only to hosts that helper already trusts, and never appears in output or argv.
78
- **Session degradation watch** (`bin/iak-degradation-watch.mjs`): a per-machine daemon that tails watched agents' Claude Code session transcripts and posts a room alert (pushed to the owner's phone via CodeWatch) the moment a session's `message.model` leaves the configured accepted set or its `permissionMode` differs from the expected one. Motivation: on 2026-07-18 an agent's session silently slipped to a small fallback model with manual permission prompts while the owner was away; the degraded agent kept posting — looping on stale tasks and reposting completed instructions — and cost an evening to diagnose. Alerts fire immediately on first detection (a wrong model in a transcript is a fact, not a flap) with a 30-minute per-signal cooldown and a recovery note. Config: `degradation_watch` block (`interval_sec`, `alert_room`, `agents[]` with `transcript_dir`, `accepted_models` prefix list, `expected_permission_mode`); API key reuses `poller.api_key`. `--once` and `--dry-run` flags for testing. Run one instance per machine under process supervision.
89
- **Send-to-session**: the :8788 daemon gains `POST /sessions/send {agent, text, from?}` — deliver text as user input into any named agent's live session — plus `GET /sessions/agents` for the target picker. Four delivery adapters, configured per agent under `mcp.confirmations.sessions.agents`: `wake` (spawn a wake/injection script), `gui` (same, chained behind a human-idle guard with `--wait` so it never types over the user), `tmux` (literal `send-keys` + Enter), and `forward` (relay to the owning machine's daemon, provenance passed through so the `[from …]` prefix is applied exactly once). Deliveries are accepted-async (202) and every attempt is receipted. This is the backend half of the CodeWatch send box; it drives any agent uniformly, including headless sessions the provider apps cannot reach.

‎README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -486,6 +486,28 @@ npx uik-daemon # standalone equivalent, INTENT_* env vars
486486

487487
The IDE Agent Kit and User Intent Kit are designed to be deployed together: IAK consumes the intent state for gating, UIK publishes your own heartbeats. Without the daemon your slot goes stale and other agents will treat you as offline.
488488

489+
#### TensorFold status on the device card
490+
491+
IAK, TensorFold, CodeWatch and GroupMind are meant to work as one system. If
492+
you run a [TensorFold](https://github.com/ThinkOffApp/TensorFold) server, the
493+
heartbeat can carry its live status (model, memory, requests, tok/s, draft
494+
acceptance, time to first token, KV cache use, Living Weights) so CodeWatch
495+
can show a card for it on each machine.
496+
497+
1. List the server in `config/models.json` with `"engine": "tensorfold"` (or
498+
leave `engine` out and let it be auto-detected). See
499+
`config/models.README.md`.
500+
2. Check what the card will say: `ide-agent-kit tensorfold-status` (add
501+
`--json` for the exact object).
502+
3. Turn it on for the publisher: set
503+
`INTENT_TENSORFOLD_REGISTRY=/path/to/config/models.json` in the
504+
environment of `uik-daemon` or `ide-agent-kit intent daemon`, and restart it.
505+
506+
It is read with GET requests only (`/health`, `/metrics`, `/v1/slide/graph`),
507+
never a completion, and an unreachable server shows as unreachable instead of
508+
stopping the heartbeat. The payload shape is in
509+
`packages/user-intent-kit/README.md`.
510+
489511
UIK v0.2.2 or later recommended for the `uik-daemon` bin. v0.2.0 introduced the daemon, v0.2.1 fixed a silent-exit in the example file, and v0.2.2 added agent-status republish on the heartbeat cadence so the agent slot does not expire while the daemon is running. Pre-0.2.2 daemons publish device state correctly but the agent slot still goes stale after its TTL.
490512

491513
### Enrichment Configuration
@@ -1185,6 +1207,7 @@ ide-agent-kit gateway <health|agents|trigger|wake> [options]
11851207
ide-agent-kit memory <list|get|set|append|delete|search> [options]
11861208
ide-agent-kit rooms archive [--backfill] [--max-pages N] [--room R]
11871209
ide-agent-kit rooms search "words" [--from handle] [--since 2026-09-20] [--regex] [--limit N]
1210+
ide-agent-kit tensorfold-status [--json] [--registry <path>] [--local] [--timeout <ms>]
11881211
ide-agent-kit init [--ide <claude-code|codex|cursor|vscode|gemini>] [--profile <balanced|low-friction>]
11891212
ide-agent-kit acp <spawn|list|status|send|close> [options]
11901213
ide-agent-kit keepalive <start|stop|status> [--pid-file <path>] [--heartbeat-sec <sec>]

‎bin/cli.mjs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,11 @@ Usage:
100100
state: overall_state + reachability + background gate summary
101101
patch: update this device's slot: intent patch key=value [key=value ...]
102102
daemon: long-running publisher (desktop state + agent heartbeat) [--interval <sec>]
103+
Add INTENT_TENSORFOLD_REGISTRY=<models.json> to publish TensorFold status on each beat.
104+
105+
ide-agent-kit tensorfold-status [--json] [--registry <path>] [--local] [--timeout <ms>] [--config <path>]
106+
Live status of every TensorFold server in the model registry, read with GET only
107+
(/health, /metrics, /v1/slide/graph). Same object the device heartbeat publishes.
103108
104109
ide-agent-kit poll --rooms <room1,room2> --api-key <key> --handle <@handle> [--interval <sec>] [--config <path>]
105110
(Legacy) Poll GroupMind rooms with explicit CLI args. Prefer "rooms watch".
@@ -562,6 +567,22 @@ async function main() {
562567
process.exit(1);
563568
}
564569

570+
// ── TensorFold status (GET only) ───────────────────────
571+
if (command === 'tensorfold-status') {
572+
const opts = parseKV(args, 'tensorfold-status');
573+
const config = loadConfig(opts.config);
574+
const { resolveTensorFoldRegistryPath, runTensorFoldStatus } = await import('../src/tensorfold-status.mjs');
575+
const registryPath = resolveTensorFoldRegistryPath({ cliPath: typeof opts.registry === 'string' ? opts.registry : undefined, config });
576+
const timeoutMs = opts.timeout ? Number(opts.timeout) : undefined;
577+
if (timeoutMs !== undefined && (!Number.isFinite(timeoutMs) || timeoutMs <= 0)) {
578+
console.error('Error: --timeout must be a positive number of milliseconds');
579+
process.exit(2);
580+
}
581+
const result = await runTensorFoldStatus({ registryPath, scope: opts.local ? 'local' : 'all', json: !!opts.json, timeoutMs });
582+
(result.exitCode === 0 ? console.log : console.error)(result.text);
583+
process.exit(result.exitCode);
584+
}
585+
565586
// ── Rooms ──────────────────────────────────────────────
566587
if (command === 'rooms') {
567588
const opts = parseKV(args, subcommand || 'rooms');

‎config/models.README.md‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Copy the example to `config/models.json` and edit it; `config/models.json` is
55
the file `bin/model-capacity.mjs` reads by default.
66

77
```json
8-
{ "models": [ { "id": ..., "host": ..., "port": ..., "kind": ..., "box": ..., "sharing": ..., "owner": ..., "auth": ..., "keyFile": ... } ] }
8+
{ "models": [ { "id": ..., "host": ..., "port": ..., "kind": ..., "engine": ..., "box": ..., "sharing": ..., "owner": ..., "auth": ..., "keyFile": ... } ] }
99
```
1010

1111
A bare top-level array is also accepted.
@@ -18,6 +18,7 @@ A bare top-level array is also accepted.
1818
| `host` | yes | **Tailnet MagicDNS name** (`asus1`, `mini`) or tailnet IP (`100.x.y.z`). See the rule below. |
1919
| `port` | yes | Integer 1-65535. The OpenAI-style HTTP port, not the ssh port. |
2020
| `kind` | yes | `openai` \| `vllm` \| `lmstudio`. Decides which list endpoint is called: `lmstudio` uses `/api/v1/models` and counts only loaded instances, the other two use `/v1/models`. |
21+
| `engine` | no | Which server software answers on this port. `tensorfold` turns on the TensorFold status read-out for this entry (see below). Any other short name (`vllm`, `llama.cpp`, `ollama`) says "not TensorFold" and skips auto-detection. Omit it to let the status reader auto-detect. |
2122
| `box` | no | Free text for humans: which machine, where it lives. Shown in `--json`, never probed. |
2223
| `sharing` | no | `exclusive` \| `shared`, default `shared`. Advisory metadata for the switcher's ranking. The probe applies the same BUSY rules to both - it is documented in `docs/model-capacity.md` rather than silently inferred. |
2324
| `owner` | no | Handle of whoever to ask before taking the box. |
@@ -64,6 +65,42 @@ two hosts behind the same router were in opposite path states to the same
6465
Berlin box at the same moment. A value written into this shared file is wrong
6566
for every caller except the one who wrote it.
6667

68+
## TensorFold entries
69+
70+
`ide-agent-kit tensorfold-status` and the device heartbeat (when
71+
`INTENT_TENSORFOLD_REGISTRY` points at this file) read the live status of
72+
every TensorFold server listed here: model, warming, memory, requests, token
73+
counters, tok/s, draft acceptance, time to first token, KV cache use and
74+
Living Weights. They send **GET /health, GET /metrics and GET /v1/slide/graph
75+
only**, never a completion.
76+
77+
Which entries count as TensorFold:
78+
79+
* `"engine": "tensorfold"`: always, reachable or not.
80+
* no `engine`: auto-detected when `/metrics` carries `tensorfold:` series or
81+
`/health` has TensorFold's shape. A server started with an API key hides
82+
both, so mark a keyed server `"engine": "tensorfold"` explicitly.
83+
* any other `engine`: never asked.
84+
85+
**Keys.** A TensorFold server may gate `/metrics` and `/v1/slide/graph`
86+
behind a key (`/health` stays open, and says only `{"status": "ok"}` while
87+
keys are on). The status reader sends a key only to an entry marked
88+
`"engine": "tensorfold"` that names its own `keyFile`, and only on those two
89+
routes. It never uses `LLM_API_KEY` / `LLM_API_KEY_FILE`: those belong to some
90+
other server as far as a status read knows. Without a usable key it reports
91+
`metrics: "auth_required"` and the rest of the card reads unknown.
92+
93+
```json
94+
{
95+
"id": "gb10-tensorfold",
96+
"host": "asus3",
97+
"port": 8080,
98+
"kind": "openai",
99+
"engine": "tensorfold",
100+
"keyFile": "~/.iak/tensorfold_gb10_api_key.txt"
101+
}
102+
```
103+
67104
## The rule: tailnet names, never LAN IPs
68105

69106
`host` must not be a private LAN address. The loader rejects `10.x.x.x`,

‎config/models.example.json‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,28 @@
2828
"box": "Mac mini (always-on)",
2929
"sharing": "shared",
3030
"owner": "@myagent"
31+
},
32+
{
33+
"id": "mb-tensorfold",
34+
"host": "localhost",
35+
"port": 8091,
36+
"kind": "openai",
37+
"engine": "tensorfold",
38+
"box": "This Mac, TensorFold serving one model (loopback only)",
39+
"sharing": "shared",
40+
"owner": "@myagent"
41+
},
42+
{
43+
"id": "gb10-tensorfold",
44+
"host": "asus3",
45+
"port": 8080,
46+
"kind": "openai",
47+
"engine": "tensorfold",
48+
"box": "ASUS Ascent GX10 (GB10), native TensorFold started with an API key file",
49+
"sharing": "exclusive",
50+
"owner": "@myagent",
51+
"auth": "bearer",
52+
"keyFile": "~/.iak/example_tensorfold_api_key.txt"
3153
}
3254
]
3355
}

‎packages/user-intent-kit/README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,89 @@ export INTENT_MODEL_ENDPOINT=127.0.0.1:8080
123123
as a last resort for a box with no endpoint to ask; a server that answers
124124
always outranks it.
125125

126+
### TensorFold servers on the device card
127+
128+
Point the daemon at a model registry (ide-agent-kit's `config/models.json`)
129+
and each TensorFold server in it gets its live status on this machine's
130+
device card:
131+
132+
```bash
133+
export INTENT_TENSORFOLD_REGISTRY=~/ide-agent-kit/config/models.json
134+
export INTENT_TENSORFOLD_SCOPE=local # default; `all` also reads remote entries
135+
```
136+
137+
- **GET only.** `/health`, `/metrics` and `/v1/slide/graph`, never a
138+
completion, so a status read cannot interfere with a Living Weights lesson.
139+
- **Off unless set.** Nothing is read or published without the registry path.
140+
- **Not on the heartbeat's path.** It reads on its own timer at the heartbeat
141+
interval; the beat publishes the last reading and never waits on a socket.
142+
- **One key, replaced whole.** Everything goes under `tensorfold`, so a server
143+
removed from the registry disappears from the card on the next beat.
144+
- **Unreachable is a reading.** A server that does not answer is published as
145+
`reachable: false` with the reason; it never takes the heartbeat down.
146+
- **Which entries:** marked `"engine": "tensorfold"`, or auto-detected from
147+
their `/metrics`. See `config/models.README.md` in ide-agent-kit, including
148+
how a keyed server names its key.
149+
- **TensorFold's default port is 8080, and so is `INTENT_MODEL_ENDPOINT`'s.**
150+
The served-model probe above may send one short generation request to a
151+
loopback endpoint to prove what is loaded. On a machine where TensorFold
152+
listens on 8080, set `INTENT_MODEL_GENERATE=0` so the heartbeat never asks it
153+
for a token.
154+
155+
The payload, per device:
156+
157+
```json
158+
"tensorfold": {
159+
"schema": 1,
160+
"checked_at": "2026-10-14T09:00:00.000Z",
161+
"scope": "local",
162+
"error": null,
163+
"servers": [{
164+
"id": "mb-tensorfold", "host": "localhost", "port": 8091, "local": true,
165+
"source": "registry", "reachable": true, "reason": null,
166+
"model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit", "warming": false,
167+
"version": null, "backend": null, "device": null,
168+
"memory": { "used_gb": 18.7, "peak_gb": 40.7, "budget_gb": 96.2 },
169+
"requests": { "running": 0, "waiting": 0 },
170+
"tokens": { "prompt_total": 0, "generation_total": 0 },
171+
"tok_s": null, "tok_s_window_s": null, "decode_tok_s": null,
172+
"draft": { "drafted": 0, "accepted": 0, "acceptance": null },
173+
"ttft_p50_s": null, "kv_cache_usage": 0,
174+
"living_weights": { "available": false, "facts": null, "learning": null, "learned": null },
175+
"metrics": "ok",
176+
"checked_at": "2026-10-14T09:00:00.000Z"
177+
}]
178+
}
179+
```
180+
181+
A value that could not be read is `null`, never `0`. `error` is set (and
182+
`servers` empty) when the registry itself could not be read, so "no servers"
183+
and "could not tell" never look the same.
184+
185+
What the numbers mean:
186+
187+
- `local`: the server runs on this machine (loopback entry). `source` is
188+
`registry` (marked `"engine": "tensorfold"`) or `detected`.
189+
- `memory`: GB, base 10, one decimal, like `mem_total_gb`. `used` is the
190+
engine's active memory, `budget` its configured budget.
191+
- `tokens`: counters since the server started, for finished requests.
192+
- `tok_s`: generated tokens of requests that finished between two reads,
193+
divided by the seconds between them (`tok_s_window_s`). `null` on the first
194+
read and after a server restart. `decode_tok_s` is the server's own live
195+
decode rate, where the server reports one (the native build does).
196+
- `draft.acceptance`: accepted over drafted MTP tokens since start, `null`
197+
until something was drafted.
198+
- `ttft_p50_s`: median time to first token since start, estimated from the
199+
histogram buckets.
200+
- `kv_cache_usage`: the fullest stream's cache use, 0 to 1.
201+
- `living_weights.available`: `/v1/slide/graph` answered (`false` on a 404,
202+
`null` when it could not be read). Counts only: the fact text stays on the
203+
machine. The native build answers the route even without `--slide`, so
204+
there `facts` is the better signal.
205+
- `metrics`: `ok`, `auth_required`, `auth_rejected` or `unavailable`.
206+
- `version`, `backend`, `device`: read from `/health` when the server reports
207+
them (newer builds), `null` otherwise.
208+
126209
> Note: The older `examples/iak-integration.js` is a one-shot demo and will exit once its setInterval timer is unref'd. Use `uik-daemon` for long-running deployments.
127210
128211
## Quick start

0 commit comments

Comments
 (0)