Skip to content

Latest commit

 

History

History
416 lines (330 loc) · 17.8 KB

File metadata and controls

416 lines (330 loc) · 17.8 KB

heimdall-cli — Programmatic & AI-Friendly Fleet Access

heimdall-cli is a one-shot, JSON-emitting client for a running hub. It subscribes like a dashboard, gathers the current fleet state, prints JSON to stdout, and exits — so shell scripts, CI/CD, and AI agent harnesses can consume the fleet without the TUI. (The same tool is also reachable as heimdall-hub cli.)

It is read-only: it only subscribes; it never mutates the hub or any host.

Install

# one binary, like the others
curl -fsSL https://github.com/kinncj/Heimdall/releases/latest/download/heimdall-cli_linux_amd64 -o heimdall-cli
chmod +x heimdall-cli && sudo mv heimdall-cli /usr/local/bin/
# or: paru -S heimdall-cli-bin     # Arch (AUR)

Usage

heimdall-cli [--hub addr] [--token t] [--tls …] [--wait 800ms] <command> [args]

Defaults to --hub localhost:9090. Output is JSON on stdout; errors are JSON on stderr with a non-zero exit.

Command Output
fleet summary: total, counts by_state, host_ids
hosts array of hosts: state, labels, metrics, has_logs/has_processes, log_sources
host <id> one host in full, including processes
top <id> the host's latest process table
logs <id> [source] the host's buffered log lines (optionally one source)
run <id> <cmd> [args] run an allow-listed, read-only command on a host (v2)

run commands (read-only, allow-listed, audited on the daemon): process.list, disk.df, uptime, os.info, dir.list <dir> (bounded to safe roots), plus the privileged dmesg and journal.tail. They are hub-mediated — the dashboard/CLI asks the hub, which routes the request down the host's outbound stream; the daemon runs unprivileged commands itself and delegates privileged ones to the root helper over the local socket (a host with no helper returns insufficient_permission). Anything off the list is refused and never executed. Requires the daemon to be started with --allow-commands.

heimdall-cli --hub "$HUB" run web-01 disk.df  | jq -r .stdout
heimdall-cli --hub "$HUB" run web-01 os.info   | jq -r .stdout
# exit non-zero if a command was refused
test "$(heimdall-cli --hub "$HUB" run web-01 uptime | jq -r .status)" = ok

top/logs show data only for hosts that push it (heimdall-daemon --process-interval / --log-source). On a quiet fleet, raise --wait (e.g. --wait 3s) so the CLI catches the next push.

Bash parsing (jq)

HUB=hub.internal:9090
export HEIMDALL_TOKEN=…   # if the hub requires a token

# ids of every offline host
heimdall-cli --hub "$HUB" hosts | jq -r '.[] | select(.state=="offline").id'

# one host's CPU%
heimdall-cli --hub "$HUB" host web-01 | jq -r '.metrics["cpu.util"]'

# performance vs efficiency cores (standardized P/E/LP, works on Apple/Intel/AMD)
heimdall-cli --hub "$HUB" host web-01 | jq '.core_groups'
# -> [{"type":"E","cores":[0,1,2,3],"util_pct":[...]},{"type":"P","cores":[4,...],...}]
# absent on uniform CPUs; the human summary is also in .details["cpu.topology"]

# top 5 processes by CPU on a host
heimdall-cli --hub "$HUB" top dgx-spark | jq '.processes | sort_by(-.cpu_pct)[:5]'

# exit non-zero if any host is offline (handy as a health gate)
test "$(heimdall-cli --hub "$HUB" fleet | jq '.by_state.offline // 0')" -eq 0

Because every value is typed JSON, you never screen-scrape a table.

CI/CD — wait for a host to come online after a release

A common deploy gate: after shipping, block until the target host re-registers and reports online, then continue (or fail the job if it never does).

# .github/workflows/verify-online.yml
name: Verify host online after deploy
on:
  release:
    types: [published]

jobs:
  verify:
    runs-on: ubuntu-latest
    env:
      HUB: ${{ secrets.HEIMDALL_HUB }}            # e.g. hub.internal:9090 (reachable from CI)
      HEIMDALL_TOKEN: ${{ secrets.HEIMDALL_TOKEN }}
      HOST: web-01
    steps:
      - name: Install heimdall-cli
        run: |
          curl -fsSL https://github.com/kinncj/Heimdall/releases/latest/download/heimdall-cli_linux_amd64 -o heimdall-cli
          chmod +x heimdall-cli && sudo mv heimdall-cli /usr/local/bin/

      - name: Wait until ${{ env.HOST }} is online
        run: |
          for i in $(seq 1 60); do
            state=$(heimdall-cli --hub "$HUB" host "$HOST" 2>/dev/null | jq -r '.state // "unknown"')
            echo "attempt $i — $HOST: $state"
            if [ "$state" = "online" ]; then
              echo "::notice::$HOST is online"; exit 0
            fi
            sleep 5
          done
          echo "::error::$HOST did not come online within 5 minutes"; exit 1

The same loop works in GitLab CI, a Jenkins stage, or a plain cron — it's just a binary and jq.

Pipe logs to Datadog (or anything)

Stream a host's error lines and forward them as Datadog events. This uses the dog CLI from datadogpy (pip install datadog); swap in datadog-ci, curl to the Events API, or your log shipper.

HUB=hub.internal:9090

heimdall-cli --hub "$HUB" logs web-01 app \
  | jq -r '.lines[] | select(.line | test("error|ERROR|panic")) | .line' \
  | while IFS= read -r line; do
      dog event post "heimdall web-01 error" "$line" \
        --tags "service:web,host:web-01,source:heimdall" --alert_type error
    done

Raw API variant (no extra CLI):

heimdall-cli --hub "$HUB" logs web-01 app \
  | jq -c '.lines[] | {ddsource:"heimdall", host:.host, service:.source, message:.line}' \
  | while IFS= read -r evt; do
      curl -sS -X POST "https://http-intake.logs.datadoghq.com/api/v2/logs" \
        -H "DD-API-KEY: $DD_API_KEY" -H "Content-Type: application/json" -d "$evt" >/dev/null
    done

Run it on a timer (cron/systemd) to tail continuously; each invocation forwards the hub's buffered window.


Use it from an AI agent / harness (copy-paste)

The files below make an AI harness query the fleet out-of-the-box. The Claude Code trio (AGENT/SKILL/COMMAND) comes first; equivalents for GitHub Copilot and any other harness (Hermes, OpenAI-compatible, custom) follow. Copy each to the path shown; they assume heimdall-cli is on $PATH and read HEIMDALL_HUB / HEIMDALL_TOKEN from the environment.

AGENT — .claude/agents/fleet.md

---
name: fleet
description: >-
  Read-only Heimdall fleet inspector. Use to answer questions about host
  health, state (online/stale/offline), per-host metrics, top processes, or
  recent logs. Calls the heimdall-cli binary; never guesses.
tools: Bash
---

You answer questions about a Heimdall fleet using the `heimdall-cli` binary, which
prints JSON from a running hub. Always run the CLI — never invent values.

Connection (read from the environment, with fallbacks):
- hub: `${HEIMDALL_HUB:-localhost:9090}`
- token: `$HEIMDALL_TOKEN` (omit `--token` if unset)

Commands (read-only, JSON on stdout):
- `heimdall-cli --hub "$HEIMDALL_HUB" fleet`            — counts by state
- `heimdall-cli --hub "$HEIMDALL_HUB" hosts`            — every host + metrics + capabilities
- `heimdall-cli --hub "$HEIMDALL_HUB" host <id>`        — one host in full
- `heimdall-cli --hub "$HEIMDALL_HUB" top <id>`         — process table
- `heimdall-cli --hub "$HEIMDALL_HUB" logs <id> [src]`  — recent log lines
- `heimdall-cli --hub "$HEIMDALL_HUB" run <id> <cmd>`   — allow-listed diagnostic
  (process.list | disk.df | uptime | os.info | dir.list <dir>); read-only, audited

Parse with `jq`. Be concise and factual: name host ids and states explicitly.
If a host is missing, say so — do not assume it exists. For "is X healthy?",
check `.state` and the relevant `.metrics` (cpu.util, mem.used, disk.used) and any
`.alerts`.

SKILL — .claude/skills/fleet/SKILL.md

---
name: fleet
description: >-
  Query a Heimdall fleet (host state, metrics, top processes, logs) via the
  heimdall-cli JSON client. Use when asked about fleet or host health, what is
  online/offline, what is using CPU, or to read a host's recent logs.
---

# SKILL: fleet

Run `heimdall-cli` (JSON, read-only) against the hub at
`${HEIMDALL_HUB:-localhost:9090}` (token: `$HEIMDALL_TOKEN`).

## Recipes

- Health snapshot: `heimdall-cli --hub "$HEIMDALL_HUB" fleet`
- Offline hosts: `heimdall-cli --hub "$HEIMDALL_HUB" hosts | jq -r '.[]|select(.state=="offline").id'`
- Hot processes: `heimdall-cli --hub "$HEIMDALL_HUB" top <id> | jq '.processes|sort_by(-.cpu_pct)[:5]'`
- Errors in logs: `heimdall-cli --hub "$HEIMDALL_HUB" logs <id> <src> | jq -r '.lines[]|select(.line|test("error";"i")).line'`
- Run a diagnostic: `heimdall-cli --hub "$HEIMDALL_HUB" run <id> disk.df | jq -r .stdout`
  (allow-listed read-only only: process.list, disk.df, uptime, os.info, dir.list)

Always parse the JSON; report host ids, states, and concrete numbers.

COMMAND — .claude/commands/fleet.md

---
description: Summarize the Heimdall fleet, or one host if an id is given.
---

Inspect the Heimdall fleet with `heimdall-cli` (hub `${HEIMDALL_HUB:-localhost:9090}`).

- If `$ARGUMENTS` is empty: run `heimdall-cli --hub "$HEIMDALL_HUB" fleet` and
  `heimdall-cli --hub "$HEIMDALL_HUB" hosts`, then report total/online/stale/offline
  and call out any host with alerts or high cpu/mem/disk.
- If `$ARGUMENTS` is a host id: run `heimdall-cli --hub "$HEIMDALL_HUB" host
  $ARGUMENTS` and summarize its state, key metrics, and any alerts.

Keep it to a short, factual brief with explicit host ids and numbers.

Copilot — .github/copilot-instructions.md

GitHub Copilot Chat reads repo-wide custom instructions from this file. Append the block below (it coexists with any existing instructions):

## Heimdall fleet (read-only)

When asked about host or fleet health, run the `heimdall-cli` binary — it prints
JSON from a running hub. Never invent values; always call the CLI and parse with `jq`.

- Hub: `${HEIMDALL_HUB:-localhost:9090}` · token: `$HEIMDALL_TOKEN` (omit `--token` if unset)
- `heimdall-cli --hub "$HEIMDALL_HUB" fleet` — counts by state
- `heimdall-cli --hub "$HEIMDALL_HUB" hosts` — every host + metrics + capabilities
- `heimdall-cli --hub "$HEIMDALL_HUB" host <id>` — one host in full
- `heimdall-cli --hub "$HEIMDALL_HUB" top <id>` — process table
- `heimdall-cli --hub "$HEIMDALL_HUB" logs <id> [src]` — recent log lines
- `heimdall-cli --hub "$HEIMDALL_HUB" run <id> <cmd>` — allow-listed read-only
  diagnostic (process.list | disk.df | uptime | os.info | dir.list <dir>)

Report host ids and states explicitly; if a host is missing, say so.

Any other harness (Hermes, OpenAI-compatible, custom)

There's no standard file, so do two things: (1) drop the AGENT text above into the harness's system / developer message verbatim — it is the system prompt; and (2) for tool-calling models, register one tool that shells out to the CLI.

A portable wrapper (works from any language that can exec a process):

#!/usr/bin/env sh
# heimdall-tool <subcommand> [args...]  ->  JSON on stdout, JSON error + non-zero on failure
exec heimdall-cli --hub "${HEIMDALL_HUB:-localhost:9090}" ${HEIMDALL_TOKEN:+--token "$HEIMDALL_TOKEN"} "$@"

A matching tool/function schema (OpenAI-style; trim to your harness):

{
  "name": "heimdall_fleet",
  "description": "Query a Heimdall fleet (read-only JSON): host state, metrics, top processes, logs, allow-listed diagnostics.",
  "parameters": {
    "type": "object",
    "properties": {
      "command": {
        "type": "string",
        "enum": ["fleet", "hosts", "host", "top", "logs", "run"],
        "description": "fleet|hosts need no args; host|top|logs|run take a host id (and logs a source, run a command)."
      },
      "args": { "type": "array", "items": { "type": "string" }, "description": "e.g. [\"web-01\"] or [\"web-01\", \"disk.df\"]" }
    },
    "required": ["command"]
  }
}

Wire the tool handler to run heimdall-tool "$command" "${args[@]}" and return its stdout. The model gets the same read-only, JSON-only surface every other harness uses.

That's the whole loop: a human pipes JSON through jq, CI gates on a host's state, a log shipper consumes logs, and an agent — Claude, Copilot, Hermes, or your own — answers fleet questions, all from the same read-only binary.

What it looks like in practice

Once the files above are in place, you talk to your fleet in plain English. These are real, unedited sessions — the agent runs heimdall-cli, parses the JSON, and answers. Note what it does not do: it never invents a number, it names host ids and states explicitly, it surfaces observability gaps (a host with no logs or no process table), and it flags the wrong default hub when it finds the real one.

Free-form question → heimdall-cli hosts

$ claude -p "check my fleet"

Fleet Brief — hub my.fancy.hub:9090

4 hosts, all online. No alerts. (Note: the skill's default localhost:9090 was wrong — the real hub came from the running daemon's env.)

Host State CPU Mem Disk GPU Temp
workstation-01 (this box, strix-halo) online 8.2% 19.6% 45.4% 61.9°C
macbook-01.local online 27.5% 71.8% 91.4% 9.9% 51.9°C
mac-mini-01.local online 8.9% 69.0% 66.7% 21.7% 51.8°C
dgx-01 (hub) online 0.2% 2.4% 21.5% 0% 33°C

Worth watching:

  • macbook-01.local — disk at 91.4% used, the only host near a real threshold. Mem also high at 71.8%. Worth freeing disk space soon.
  • mac-mini-01.local — mem 69%, fine but the next-highest.
  • workstation-01 and dgx-01 are both relaxed. dgx-01 is essentially idle.

Only workstation-01 reports per-process data (has_processes: true); none have logs enabled. Want me to pull top processes on workstation-01 or dig into the MacBook's disk usage?

The /fleet slash command

The COMMAND file turns the whole brief into one keystroke:

$ claude -p "/fleet"

Fleet is healthy — all 4 hosts online, none stale or offline.

Fleet snapshot (hub dgx-01)

Host State CPU Mem Disk GPU Temp Uptime
workstation-01 (strix-halo, AI powerhouse) 🟢 online 1.3% 19.7% 45.4% 51.4°C ~4.7d
macbook-01.local 🟢 online 16.4% 70.6% 91.4% ⚠️ 11.2% 51.9°C ~21d
mac-mini-01.local 🟢 online 8.2% 68.5% 66.7% 23.0% 51.8°C ~12.7d
dgx-01 (hub) 🟢 online 0.2% 2.4% 21.5% 0% 33°C (GPU) ~11.3d

Notable:

  • All green — 4/4 online, last seen within ~1s of the snapshot.
  • ⚠️ macbook-01 disk is at 91.4% — the only metric worth watching. It's also the busiest node (CPU 16%, mem 71%, ~21 days uptime).
  • workstation-01 (the Strix Halo AI box) is idle and cool — plenty of headroom. Only host exposing process data (has_processes: true).
  • dgx-01 (the hub itself) is nearly idle — 0.2% CPU, 2.4% mem.
  • Network latency is low everywhere (4.9–15.8 ms); power draw is modest (workstation-01 pulls the most at ~17.4 W package).

Want me to drill into workstation-01's top processes, or keep an eye on that MacBook disk?

A targeted risk question → host + top

The agent pulls the full host record and process table only for the hosts you ask about, then reasons over what's there and what's missing.

$ claude -p "any risks for workstation-01 or the MacBook on my fleet?"

Both hosts are online and reachable. Here's the risk assessment:

workstation-01 — ✅ Healthy, no significant risks

  • Online, seen seconds ago. Uptime ~4.7 days.
  • CPU 1%, memory 20%, disk 45%, temp 49°C — all healthy.
  • Only notable consumer is a Firefox content process (Isolated Web Co, 14.6% CPU / 8.1% mem) — normal for an interactive workstation, not a leak or crash loop.

the MacBook (macbook-01.local) — ⚠️ Needs attention

  • Online, seen seconds ago. Uptime ~21 days.
  • 🔴 HIGH — Disk 91.4% full. The main actionable risk. Nearing capacity risks write failures, swap pressure, and app instability. Free space soon.
  • 🟡 MEDIUM — Memory 73.7%. Elevated; combined with the near-full disk, swap headroom is thin.
  • 🟢 LOW — CPU 28.8%, GPU 10%, temp 52°C — fine.
  • Observability gap: this host reports no process table and no logs (has_processes/has_logs both false), so the disk/memory usage can't be attributed and log-based crash detection is blind here. Worth enabling process + log collection on this agent.

Bottom line: workstation-01 is clean. The one real risk on your fleet is the MacBook Pro's nearly-full disk — clear space there, and consider turning on process/log telemetry so it's not a blind spot.

A few things to read out of these transcripts:

  • Hub discovery matters. The agent uses ${HEIMDALL_HUB:-localhost:9090}; when the default is wrong it says so and falls back to what it finds (here, mDNS via --hub auto). Set HEIMDALL_HUB in the agent's environment to skip the guesswork.
  • It reports capability gaps, not just metrics. has_processes / has_logs drive the "blind spot" callouts — enable process view and log streaming on a host to close them.
  • It stays read-only. Every answer above came from hosts / host / top — no mutation, nothing run on a host beyond the allow-listed diagnostics.

Notes

  • --wait (default 800ms) bounds how long the CLI gathers before printing; raise it on a slow link, a large fleet, or right after an idle period.
  • Data is whatever the hub holds: logs/processes appear only for hosts configured to push them. See process view and log streaming.

Next steps