feat(daemon,sdk): live sandbox resource metrics

[MDzaja]

Summary

Adds a live resource-usage snapshot for sandboxes. A new daemon endpoint GET /system/metrics returns current CPU, memory and disk usage, surfaced through all five SDKs as sandbox.getMetrics() (Python sync + async, TypeScript, Go, Ruby, Java).

Motivation

There's currently no first-class way for an SDK caller to read a sandbox's live resource usage. This exposes it directly from the sandbox daemon, so applications can monitor CPU/memory/disk, drive autoscaling decisions, or surface usage in dashboards.

What it returns

SystemMetrics:

Field	Type	Meaning
timestamp	string (RFC3339)	When the snapshot was taken
timestampUnix	int64	Same instant, Unix seconds
cpuCount	int	CPU cores available to the sandbox (ceil(cpuLimit))
cpuUsedPct	double	CPU usage as a % of the sandbox's CPU limit (a fully busy N-core sandbox with an N-core limit reads ~100%)
memUsed	int64	Memory used, in bytes (includes page cache)
memTotal	int64	Memory limit, in bytes
memCache	int64	Page cache, in bytes
diskUsed	int64	Disk used, in bytes
diskTotal	int64	Total disk size, in bytes
diskFree	int64	Available disk, in bytes

How it works

• CPU% is computed by a lightweight background sampler in the daemon: every 5s it reads the cgroup CPU-time counter and divides the delta by the wall-clock window and the CPU limit. The latest value is served from a mutex-guarded field. It reads 0 until the first sample window completes (cold start), and resets to 0 on a counter reset (cgroup recreated) or a backwards clock.
• Memory and disk are point-in-time reads taken on demand in the request handler (from cgroup memory.current/memory.stat and statfs), so they're always fresh.
• The sampler goroutine is started before the server begins serving and is cancelled on shutdown, so it adds no startup latency.

Changes by component

Daemon (Go)

• New apps/daemon/pkg/toolbox/system/ package: the SystemMetrics DTO, the background sampler, and the GET /system/metrics handler.
• server.go: starts the sampler and registers the route.
• Regenerated swagger. Byte fields are tagged format:"int64" (and cpuUsedPct format:"double") so strongly-typed generated clients use 64-bit types instead of overflow-prone 32-bit ones.

Shared telemetry (libs/common-go/pkg/telemetry)

• Exposes reusable cgroup helpers (GetContainerLimits, GetDiskStats, ReadCgroupMemUsageBytes, ReadCgroupMemCacheBytes, ReadCgroupCPUUsageNanos) and a shared CPUUsagePercent formula.
• The existing OTEL usage-collection path was refactored to use these helpers too, so the sampler and the OTEL exporter share one implementation (no duplicated cgroup-reading or CPU%-math logic).

Generated API clients (all languages)

• New SystemApi.getSystemMetrics() + SystemMetrics model.

SDKs (all five)

• New getMetrics() method on the sandbox, returning the live SystemMetrics. Mirrors the existing toolbox-API wiring (e.g. the info API) and is purely additive.

Backward compatibility

• Purely additive: a new endpoint, a new SystemMetrics model, and new SDK methods. No existing request/response contract, DTO, or SDK signature changed.
• New SDK getMetrics() requires the new daemon; against an older daemon the endpoint simply 404s.

Testing

• Daemon unit tests: the CPU% formula (table-driven) and the sampler state machine (cold start, normal delta, counter reset, idle, backwards clock, zero-limit) — run under -race.
• Per-SDK e2e tests: each SDK's e2e suite asserts the metrics shape and plausible bounds (memUsed ≤ memTotal, diskUsed ≤ diskTotal, cpuCount ≥ 1, etc.).
• Live-verified end-to-end on a running stack across all five SDKs, including CPU rising to ~100% under load on a 1-core sandbox.

Safety / performance notes

• The daemon code is race-free and leak-free: shared limits are immutable after construction, the only mutable sampler state is mutex-guarded, the sampler goroutine is bounded to the server context with a stopped ticker, and per-request/cgroup reads are tiny and not retained.
• No measurable impact on daemon startup time — the only synchronous addition is one cheap cgroup-limit read; all sampling runs in a background goroutine spawned before the server starts serving.

Field-semantics notes (for API docs / consumers)

• memUsed already includes page cache, so don't add memCache on top of it.
• cpuCount is ceil(cpuLimit), so a fractional CPU limit rounds up.
• diskUsed + diskFree may be slightly less than diskTotal (filesystem reserves blocks for root).

---

Summary by cubic

Add live sandbox resource metrics via GET /system/metrics and sandbox.getMetrics()/get_metrics() in all SDKs to read CPU, memory, and disk in real time. This enables monitoring, autoscaling, and dashboards.

• New Features

  • Daemon: GET /system/metrics returns a snapshot with timestamp, timestampUnix, cpuCount, cpuUsedPct, memUsed, memTotal, memCache, diskUsed, diskTotal, diskFree. CPU% is averaged over a 5s background sampler; memory and disk are read on demand.
  • SDKs: new metrics method in Python (sync/async), TypeScript, Go, Ruby, and Java; toolbox clients add SystemApi.getSystemMetrics() and a SystemMetrics model.
  • Telemetry: exported cgroup helpers and CPUUsagePercent; the OTEL path reuses them. Unit and e2e tests validate formulas and bounds.

• Migration

  • Additive change. Update to the latest daemon and SDKs; call sandbox.getMetrics()/get_metrics().
  • Older daemons will return 404 for this endpoint.

^{Written for commit c58dbe5. Summary will update on new commits.}

[nx-cloud]

View your CI Pipeline Execution ↗ for commit c58dbe5

Command	Status	Duration	Result
nx e2e:cleanup daytona-e2e	✅ Succeeded	<1s	View ↗
nx run sdk-typescript:test:runtime	✅ Succeeded	2m 56s	View ↗
nx run-many --target=test:e2e --all --nxBail=true	✅ Succeeded	6m 48s	View ↗
nx e2e daytona-e2e	✅ Succeeded	1m 16s	View ↗
nx run-many --target=build --projects=api,runne...	✅ Succeeded	10s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

---

☁️ Nx Cloud last updated this comment at 2026-06-22 18:53:21 UTC

[cubic-dev-ai]

4 issues found across 59 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>