Skip to content

HARVESTER.md

Latest commit

 

History

History
256 lines (210 loc) · 10.6 KB

HARVESTER.md

File metadata and controls

256 lines (210 loc) · 10.6 KB
Raw

GPU Harvester

TrainPulse already runs on every training node and already knows, tick by tick, whether each GPU is doing useful work. The harvester turns that existing signal into a way to lend idle GPUs to other work — batch jobs, inference, other teams' experiments — without taking anything away from the training job TrainPulse is there to protect.

It has two halves, matching where each decision actually needs to be made:

  • Per-node (internal/harvest, built into the trainpulse daemon): decides which of this machine's GPUs are idle, and is the only thing allowed to start or kill a process on one of them. This half must keep working even if every other machine in the fleet, and the coordinator below, disappears.
  • Cross-node (internal/coordinator, a separate trainpulse-harvester binary): polls every node's idle-GPU view, keeps a fleet-wide pool, and schedules submitted jobs onto it. It is best-effort by nature — it only ever knows what it last polled — and a coordinator outage must never weaken a node's own safety guarantees.

This mirrors the rest of TrainPulse: the daemon is the source of truth about its own machine; nothing external can be more authoritative about a node than the node itself.

Off by default

Harvesting is two separate opt-ins in the node's config, both false by default:

{
  "harvest": {
    "enabled": false,
    "allow_exec": false,
    "idle_utilization_pct": 5,
    "idle_window": "5m",
    "max_lease_ttl": "30m"
  },
  "auth_token": ""
}
  • enabled turns on idle-GPU detection: the read-only /v1/harvest/candidates and /v1/harvest/leases endpoints, and the trainpulse_harvest_* metrics. Nothing on the machine can be started or stopped because of this flag alone.
  • allow_exec turns on /v1/harvest/lease, which runs a caller-supplied command on the machine. This is remote code execution by design, scoped to whatever OS user runs the daemon. The daemon refuses to start with allow_exec: true unless auth_token is also set — see Security.

Run enabled alone for a while first. It costs nothing and tells you how much idle capacity actually exists before you decide whether to let anything claim it.

How a GPU becomes harvestable

Every tick, internal/harvest.Engine checks each GPU's utilization against idle_utilization_pct. The moment utilization rises above that threshold, the GPU's idle clock resets to zero. Once utilization has stayed at or below the threshold continuously for idle_window, the GPU is reported as harvestable (unless it is already leased).

GET /v1/harvest/candidates
{
  "enabled": true,
  "candidates": [
    {"gpu_index": 0, "utilization": 91.0, "idle_seconds": 0,   "harvestable": false},
    {"gpu_index": 1, "utilization": 0.0,  "idle_seconds": 612, "harvestable": true}
  ]
}

This is a utilization-only signal — it does not inspect what processes are running on the GPU. See Known limitations for what that trades away and how to close the gap later.

Leasing a GPU

With allow_exec: true and an auth_token set:

curl -X POST http://node:9876/v1/harvest/lease \
  -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"gpu_index": 1, "command": ["python", "batch_job.py"], "owner": "alice", "ttl_seconds": 1800}'

The daemon starts command with CUDA_VISIBLE_DEVICES pinned to the leased GPU, in its own process group, and returns a lease:

{"id": "lease_...", "gpu_index": 1, "owner": "alice", "status": "running",
 "claimed_at": "...", "expires_at": "..."}

gpu_index can be omitted (or null) to let the node pick any currently harvestable GPU. A lease ends one of four ways, all visible in GET /v1/harvest/leases via status:

status Why
released The command exited on its own, or a caller called /v1/harvest/release.
expired max_lease_ttl (or the caller's shorter ttl_seconds) passed.
preempted The real owner's workload resumed — see below.
failed The command could not be started, or exited with an error the daemon observed.

Giving the GPU back: preemption

TTL expiry alone is not enough — a training job that was paused (checkpoint, debugging, a scheduler hiccup) can resume mid-lease, and it must never have to wait behind a harvested job for the rest of the TTL.

TrainPulse already has a strong, existing signal for "the real owner is back": a training loop only calls POST /v1/training while it is actually running. So the moment a training sample arrives referencing a GPU with a running lease, that lease is killed immediately, tagged preempted / owner_resumed — no waiting for the next tick, no waiting for TTL.

If the sample carries per-rank GPU indices (ranks[].gpu_index), only those specific GPUs are preempted. If it doesn't (a bare single-process push, the common case for non-distributed training), every leased GPU on that node is preempted, because there is no way to tell which GPU the sample came from. This trades away some harvestable capacity on multi-GPU nodes running single-process training in exchange for never letting a harvested job block real training — see Known limitations for the follow-up that would narrow this.

The coordinator

trainpulse-harvester is a separate, stateless-ish process that polls a list of nodes and turns their individual /v1/harvest/candidates into one fleet-wide pool:

trainpulse-harvester -config coordinator.json
{
  "addr": "127.0.0.1:9877",
  "auth_token": "",
  "poll_interval": "10s",
  "nodes": [
    {"id": "gpu-node-01", "addr": "http://10.0.1.11:9876", "auth_token": "..."},
    {"id": "gpu-node-02", "addr": "http://10.0.1.12:9876", "auth_token": "..."}
  ]
}

API:

  • GET /v1/pool — every configured node, whether it's currently reachable, and its latest known candidates.
  • POST /v1/jobs — {"command": [...], "owner": "...", "ttl_seconds": N}. Scheduled onto the fleet's single most-idle harvestable GPU (see Scheduling); returns a Job.
  • GET /v1/jobs / GET /v1/jobs/status?id=... — list or fetch job state.
  • POST /v1/jobs/cancel — {"id": "...", "reason": "..."}, releases the underlying lease.

Each poll also reconciles every running job against its node's current lease list, so a job's status catches up to a TTL expiry or an owner-resumed preemption that happened on the node without the coordinator asking for it — the node never waits for the coordinator's permission to protect its own training job.

A coordinator outage does not compromise safety: nodes keep expiring and preempting leases on their own regardless of whether anything is polling them. It just means no new jobs get scheduled until the coordinator is back.

Scheduling

v1's scheduler is deliberately simple: one GPU per job, placed on whichever harvestable candidate across the whole fleet has been idle the longest. That heuristic isn't about node load or job size — it minimizes the chance the GPU's real owner comes back mid-job, since a GPU that has already sat idle the longest is statistically the least likely to be reclaimed in the next few minutes.

Security

Both harvest.allow_exec (node) and the coordinator's job-submission API are remote code execution surfaces by design — that is the feature. Treat them accordingly:

  • Always set auth_token on any node with allow_exec: true, and on the coordinator if it's reachable beyond localhost. The daemon refuses to start otherwise.
  • Never bind -addr to a non-loopback address without a token in front (the existing TrainPulse guidance in the main README applies doubly here).
  • The harvested command runs as whatever OS user runs trainpulse. Run the daemon as an unprivileged, dedicated user, and consider containerizing or otherwise sandboxing what allow_exec is permitted to run — TrainPulse itself does no sandboxing beyond CUDA_VISIBLE_DEVICES scoping and a process-group kill.
  • Prefer a narrow, reviewed set of harvestable commands over accepting arbitrary caller-supplied commands in production; the API does not currently support an allowlist (see below).

Known limitations

This is a first slice. Documented gaps, roughly in the order they'd matter for a production rollout:

  • Idle detection is utilization-only. It does not check whether any process is actually using the GPU (nvidia-smi --query-compute-apps), so a GPU that is allocated but momentarily quiet looks identical to one that is genuinely free. Adding a process-list check would let preemption target the exact GPU a new process appears on, instead of the current all-leases-on-this-node fallback.
  • One GPU per job. Multi-GPU jobs that need several leases claimed atomically (with rollback if only some succeed) are real distributed scheduling and are not implemented.
  • No command allowlist. allow_exec grants the caller to run whatever command they send; there's no per-node policy restricting which binaries or images are acceptable.
  • No isolation beyond CUDA_VISIBLE_DEVICES and a process group. A harvested job shares the host's filesystem, network namespace, and every other GPU's visibility is only blocked by the CUDA runtime honoring the env var. Container- or VM-level isolation is a natural next step.
  • Coordinator state is in-memory and single-instance. A coordinator restart loses job history (though not the underlying leases — those live on the nodes and keep expiring/preempting normally). No HA/leader election.
  • No Slurm/Kubernetes device-plugin integration. Both are natural consumers of /v1/harvest/candidates and /v1/pool but aren't wired up yet.

Reference

Node (trainpulse daemon) endpoints, all behind the existing bearer-auth middleware:

Endpoint Method Purpose
/v1/harvest/candidates GET Per-GPU idle/harvestable state
/v1/harvest/leases GET Every known lease, running and finished
/v1/harvest/lease POST Claim a GPU (allow_exec required)
/v1/harvest/release POST Force-stop a running lease

Coordinator (trainpulse-harvester) endpoints:

Endpoint Method Purpose
/v1/pool GET Fleet-wide node/candidate view
/v1/jobs GET / POST List jobs / submit a job
/v1/jobs/status?id= GET Fetch one job
/v1/jobs/cancel POST Cancel a running job

Metrics added to /metrics and /v1/metrics on the node: trainpulse_harvest_enabled, trainpulse_harvest_idle_gpus, trainpulse_harvest_harvestable_gpus, trainpulse_harvest_leased_gpus.