Skip to content

Refuse a build when the host has no CPU/RAM headroom for a second project #91

Description

@koenkooi

Part of #72. Needs #77 (M3) — parallel builds only become normal once projects have private volumes. Build-path sibling of #74 (M0).

Why this exists now

#72 settled that all three volumes are private per project by default and sharing is opt-in. That decision is right, but it has a consequence worth naming: under shared caches the one-VM rule serialised everything, so two projects could never contend for CPU or RAM. With private volumes, two projects building at once is legitimate — and nothing accounts for it.

The held-volume refusal (#74) covers volume contention. It says nothing about resource contention, and after M3 that becomes the more likely failure.

The problem, quantified

set_defaults() (mackas:631-637) sizes each VM from the host, independently:

MACKAS_CPUS   = host_cores - 2   (min 2)
MACKAS_MEMORY = host_mem * 2/3   (min 4g)

Every project's config gets that same default, seeded at pin time. So two projects building in parallel request 4/3 of host RAM and 2 × (cores − 2) CPUs. On a 32 GB Mac that is roughly 42 GB of guest memory against 32 GB of host. The comment at mackas:629 already says overcommitting "starves the host" — two concurrent projects do exactly that, structurally.

Validation cannot see it. check (mackas:2086-2099) compares one config's MACKAS_MEMORY against total host RAM (host_mem_gb(), mackas:529) and warns past 75%. Both projects pass that check individually and still oversubscribe together. This is the same shape as the disk-accounting gap #82 (M8) fixes — per-config validation with no machine-wide view — with one important difference: disk is a ceiling you approach slowly, whereas RAM oversubscription is swap-thrashing within minutes, and on a Mac it degrades the whole machine, not just the build.

What ships

A require_resource_headroom helper, sibling to the existing require_volumes_free() (mackas:1994), called from the same point in run_kas() that #74 adds the volume check to.

  • Walk running containers the way volume_in_use() already does (mackas:1971: container lscontainer inspect <id>), summing the CPU and memory each already reserves.
  • Add this build's own MACKAS_CPUS / MACKAS_MEMORY.
  • Refuse when the total exceeds host RAM, less a configurable margin — mirroring MACKAS_FREE_SPACE_MARGIN_GB (mackas:126, default 20) rather than inventing a second idiom.
  • Fail-closed on an inspect hiccup, matching require_volumes_free(). No --force.

Refusal, not degradation. Refusing when there is no headroom is the accepted behaviour. Auto-downsizing is explicitly rejected: a running VM's memory cannot be resized, so the only "graceful" alternative would be silently starting a smaller build than the config asked for — turning a clear refusal into a mysteriously slow build, with no message connecting cause to effect. That is the failure mode this project consistently designs against.

Graceful means the refusal is actionable, not that the build proceeds. The message must name the other project (not just a container id), both reservations, the host total, and the knob to change:

mackas: not enough headroom to start this build
  meta-qcom is building: 20 GiB / 10 cpus
  this build wants:      20 GiB / 10 cpus
  host has:              32 GiB / 12 cpus (margin 20 GiB)
  wait for it to finish, or lower MACKAS_MEMORY/MACKAS_CPUS for this project

Open question: can a reservation be read back?

Whether Apple's container inspect exposes the -c / -m a container was started with (kas_runtime_args() builds them at mackas:1470) is unverified — it needs a live Mac with the runtime, the same way #80's virtiofs question does.

If it does not, there is a fallback that costs nothing and blocks nothing: #72 already plans to reverse-grep the pinned configs to name which project holds a volume in the #74 refusal message. That same lookup yields the holder's MACKAS_CPUS / MACKAS_MEMORY. Slightly less precise — it reads intent from config rather than the running reservation, so a build started with an overridden value would be mis-attributed — but sufficient, and it makes the design independent of what inspect happens to expose.

Decide with data at implementation time; do not block on it.

Also worth doing

  • check should report the machine-wide picture alongside the existing single-config line, the way M8: Sharing groups, shared-volume safety, and machine-wide accounting #82 does for disk: sum the reservations of every pinned project and say plainly that N projects at these defaults cannot build concurrently. Informational, not a gate — the existing single-config check stays the pass/fail.
  • project add seeds new configs from currently-effective settings, which means the second project inherits host_mem * 2/3 and silently sets up the collision. Consider suggesting a lower value at pin time when other pinned projects already exist, in the same register as the existing legacy-volume nudge. Sizing is never changed silently.

Tests

  • Two mock running containers reserving more than host RAM: a third build refuses, and the message names the project, not just a container id.
  • Exactly at the margin boundary: allowed at the limit, refused one unit past it. Mutation-test the boundary — an off-by-one here either blocks a legitimate build or permits the thrash this exists to prevent.
  • Single project, nothing else running: unchanged behaviour, no new output. This is the guard for every existing user, since it must stay silent for them.
  • Inspect failure: fail-closed, matching require_volumes_free().
  • --dry-run performs no headroom query (same discipline as M0: Refuse a held volume on every build path #74; note that run_kas already calls ensure_container_running before the dry-run return, so assert the narrow property).

Size: M. Risk: medium — it is a build-path refusal, so a wrong threshold blocks legitimate work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions