fix(balloon): keep a guest reserve in the idle target instead of the floor (CORE-45) - #518
fix(balloon): keep a guest reserve in the idle target instead of the floor (CORE-45)#518AprilNEA wants to merge 4 commits into
Conversation
…floor (CORE-45) The idle target was `used + 256MB`, which for an idle guest (used ≈ 0) collapsed onto the 384MB floor. Measured 2026-07-29 on a 16 GB VM, the descent then asked for 15.8 GB of the guest's 15.96 GB available and held MemAvailable at literally 0 for ~98 s per cycle — an OOM hazard for any running container, and ~14 cores of reclaim spin to produce it. The module's own invariant says the target is never an unconditional constant; the formula violated it in exactly the idle case it exists for. The target is now the higher of two floors: `used + 1GiB`, so the guest keeps a working reserve, and `total - available/2`, so a single entry reclaims at most half the slack rather than walking the guest to the edge in one descent. A reclaim smaller than 512MB is not worth its guest-side cost and keeps the balloon instead. The 1GiB reserve strictly dominates the old 384MB floor, so the floor is removed rather than left as unreachable code — which also drops a latent `clamp(min, max)` panic for a configured size below 384MB. Policy is inert today (`reclaim_capable` is false on every macOS backend, arcbox#504); this is the precondition for re-enabling shrinking anywhere.
|
Run failed. View the logs →
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecced88fea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Greptile SummaryThe PR revises idle balloon sizing and strengthens failed-restore recovery.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the current code addresses the previously reported sizing-domain, stale-restore, active-retry, and test-oracle concerns.
|
| Filename | Overview |
|---|---|
| app/arcbox-core/src/vm_lifecycle/balloon/controller.rs | Adds sound active and idle retry paths for failed restores and strengthens controller regression tests. |
| app/arcbox-core/src/vm_lifecycle/balloon/mod.rs | Replaces the fixed floor policy with reserve- and share-bounded reclaim arithmetic in the configured-memory domain. |
Reviews (4): Last reviewed commit: "fix(balloon): also retry a stranded shri..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…le shrinks Three review findings, all verified against the code first. The guest's MemTotal sits below the configured size (kernel reservations: a 16384 MB VM reports 15.6 GiB, measured today). Deriving the target from MemTotal and applying it against the configured full size reclaimed that ~400MB gap on top of the intended share, eating into the very reserve the policy exists to protect. The policy is now expressed as a reclaim subtracted from full, with available used only as a delta — domain-free — and clamped to the guest's own total so an inconsistent reading cannot inflate it. A restore that exhausts its retries deliberately leaves the shrunk bookkeeping set, and its doc comment claimed the next idle entry would notice. Nothing ever read it, so a later entry deciding Keep stranded the guest at the stale target indefinitely. Entry is now the retrier: it restores before probing whenever a shrink is still applied, which the sizing invariant (balloon-empty stats) demanded regardless. The staged-descent test derives its expectation from the policy helpers, so it now also pins the sequence shape independently — first move is one SHRINK_STEP, every move is a full step but the clamping last, and it ends exactly at the policy target.
…entry A fail-open restore that exhausts its attempts lands in Active with the shrink still applied. Retrying only on the next idle entry is not a retry for an active VM, which may never go idle again — and that is the worst case to strand, since fail-open usually follows guest memory pressure: the guest is working, under pressure, and squeezed. Active now carries a 30s retry whenever a shrink is still applied, and no timer at all otherwise.
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
The new stale-shrink retrier only fires on idle re-entry, so a continuously busy VM never reaches it and stays squeezed for the daemon's lifetime.
Reviewed changes
- Re-expressed
idle_targetas a reclaim subtracted fromfull, so the configured and guest-visible memory domains can no longer be mixed —availableis used only as a delta and is clamped tomin(stats.total, full)so an inconsistent reading cannot inflate the reclaim. - Made
enter_idlethe retrier for a stale shrink: it restores before the stats probe wheneverappliedis set, which also keeps the sizing invariant (entry stats are only meaningful while the balloon is empty). - Pinned the derived descent oracle in
staged_descent_steps_on_settled_frameswith three shape assertions that hold independently of the helpers that produced the sequence. - Added
reclaim_ignores_the_memtotal_to_configured_gapandstale_shrink_is_restored_on_next_entry.
The domain rework is the right shape. Both guarantees in the doc comment now hold for every input, not just the ones the tests pass: the reclaim is capped at available - IDLE_MIN_RESERVE, so the guest always keeps its 1 GiB, and target >= full - available + reserve >= used + reserve keeps the target above usage. I swept adversarial inputs (available > total, available > full, full smaller than the reserve, zero available) looking for a violating case and there isn't one. The previous round's two concerns — MIN_WORTHWHILE_RECLAIM having no distinguishing test, and the reserve being short by full - MemTotal — are both closed by this commit.
ℹ️ Coverage sequencing
Nothing above unit tests exercises the descent. tests/e2e/tests/idle_balloon.rs is VZ-pinned and asserts the balloon never moves, which is correct forever for VZ (guest RAM lives in Apple's XPC process, so it can never become reclaim-capable) but means it can never cover a shrink. With reclaim_capable() still false on both backends, CORE-44's HV re-enable would be this formula's first execution against a real guest. Worth landing an HV-gated e2e alongside that flip rather than after it.
ℹ️ Nitpicks
The headroom/floor vocabulary outlives the constants this PR deleted:
controller.rs:127—PRESSURE_MIN_AVAILABLEis documented as "Safely below the entry headroom (256 MiB)". The margin it sits below is now the 1 GiB reserve, so the claim is more true than before, but it names a constant that no longer exists.controller.rs:384— the shrink log still says"idle balloon shrunk to guest usage + headroom".mod.rs:163— thestats.total == 0guard's comment reasons about"used + headroom"degenerating to "the floor". The guard is also now redundant:availableis clamped tostats.total, so a zero total gives a zero reclaim andentry_decisionreturnsKeepon the worthwhile check anyway. Keeping it as a cheap early-out is fine; the justification needs rewording either way.guest/arcbox-agent/src/memory_pressure.rs:11carries the same stale wording.controller.rs:760— the assertion message "every move is a full step except the clamping last one" describes a case the test never reaches.final_target()is exactlyFULL - 3 * SHRINK_STEP, so the sequence is[14, 12, 10]GiB and every move is a full step. The escape hatch in the condition is unexercised; a non-divisiblefinal_targetin the fixture would make it real.
Claude Opus ｜ 𝕏
fail_open notes activity, so the resulting ExitIdle arrives immediately after a failed restore. Retrying there turns a transient backend failure into a same-instant recovery instead of a wait for the 30s timer, which stays as the backstop for the case where no further edge ever arrives.
There was a problem hiding this comment.
Important
The new 30s retry closes the stranding hole, but it never terminates when the VM is stopped rather than deleted — the guest stays squeezed and the daemon logs 4 lines every 30s for the rest of its life.
Reviewed changes
- Added
STRANDED_RESTORE_RETRY(30s) and atokio::select!timer arm toMode::Active, armed only whenapplied.is_some()andstd::future::pending()otherwise, so an active VM that never goes idle again still retries. - Made
ExitIdleinMode::Activeretry the restore instead of being a no-op, so the activity edgefail_openitself emits recovers a transient failure immediately rather than after up to 30s. - Added
stranded_shrink_is_retried_while_activeandstranded_shrink_is_retried_on_the_activity_edge.
The topology is now complete in the sense the previous thread asked for: with the enter_idle restore from 4abf2ff, the ExitIdle edge, and the timer, there is no mode that can hold applied without a scheduled retry. I checked the two ways this could have been quietly non-functional and it is neither: Harness::settle() is 50 yield_now().awaits with no clock advance, so the activity-edge test genuinely cannot be rescued by the timer and the while-active test genuinely needs its explicit tokio::time::advance — both new tests are falsifiable. And the let stranded = self.applied.is_some() snapshot cannot go stale, since nothing mutates applied between the snapshot and either arm body within one loop iteration. Repeated EnterIdle/ExitIdle cannot starve the timer either, because both of those paths restore on their own.
What the retry lacks is a stopping condition. restore() clears applied on exactly two paths — the machine record being deleted, or set_balloon_target succeeding — and a stopped VM satisfies neither, so the loop runs forever. Details inline.
ℹ️ Nitpicks
controller.rs:252 — the Mode::IdleUnshrunk ExitIdle arm still comments "Nothing was shrunk; nothing to restore". Since 4abf2ff, enter_idle can return IdleUnshrunk with applied still set (that is exactly the failed-restore path), so the premise no longer holds. The arm is still correct, because the new Active timer picks the retry up, but for a different reason than the comment gives.
Claude Opus ｜ 𝕏
| std::future::pending::<()>().await; | ||
| } | ||
| } => { | ||
| self.restore("retrying a stranded shrink"); |
There was a problem hiding this comment.
This retry has no terminal condition when the VM is stopped rather than removed, and stop is the ordinary case.
restore() clears applied only when full_memory_bytes() returns None — i.e. the machine record was deleted — or when set_balloon_target succeeds. A stopped VM keeps its record, and set_balloon_target rejects a non-Running VM, so neither happens. The controller also never finds out about the stop: balloon commands fire only on idle edges, and a stranded controller is already in Mode::Active, so no EnterIdle/ExitIdle arrives to break the cycle.
The result is 3 warn! plus 1 error! every 30s — roughly 11.5k lines a day — for the remaining lifetime of the daemon, with applied permanently stuck. That is a regression in shape from the pre-commit behaviour, which was silent and one-shot. It also makes the reply on the previous thread only conditionally true: the retry re-applies full and clears applied once a VM is running again, but until then it cannot.
The fix wants to be "terminate when no running VM can accept the target", not "retry less often". Treating a not-running VM the same way restore() already treats a gone VM would do it, since a fresh boot starts with an empty balloon anyway — the guest is unsqueezed by the stop itself, so there is nothing left to give back. Clearing applied on the stop transition would work equally well and is arguably the more honest statement of the invariant.
Technical details
restore()(controller.rs:479-508): thelet Some(full) = self.deps.full_memory_bytes() elsearm is the only placeappliedis cleared without a successfulset_balloon_target, and it triggers on a missing machine record.full_memory_bytes()(actor.rs:197-202) ismachine_manager.get(&name).map(|info| info.memory_mb * MiB).MachineManager::get(machine.rs:1168) reads the map;stop(machine.rs:1118-1124) setsstate = Stoppedandcid = Nonebut keeps the entry; onlyremove()(machine.rs:1228) deletes it. Sofull_memory_bytes()staysSomeacross a stop.set_balloon_target(actor.rs:204-214→vm.rs:921-938) returnsErr(invalid_state)atvm.rs:928wheneverentry.info.state != MachineState::Running. Every attempt fails, all three times, forever.- Balloon commands are idle-edge-only (
actor.rs:431-448):EnterIdlewhenafter == Idle,ExitIdlewhenbefore == Idle. A stranded controller sits inMode::Activebecausefail_openput it there, so a stop emits neither. - The controller is per-actor-lifetime (
actor.rs:361-362, spawned once atactor.rs:373), not per-VM-boot, so "the rest of the daemon's lifetime" is the real bound.
Mitigating, as with the rest of this subsystem: reclaim_capable() is false on every backend today (actor.rs:193), so none of this executes until CORE-44 flips HV on.
| /// given back (a restore that silently "succeeds" in bookkeeping only | ||
| /// would leave the guest squeezed with nobody retrying). | ||
| /// would leave the guest squeezed with nobody retrying). `enter_idle` | ||
| /// is the retrier: it restores before probing whenever `applied` is set. |
There was a problem hiding this comment.
Two things in this doc block are now stale, both in ways that would mislead the next reader about where recovery happens.
The comment names enter_idle as "the retrier", but as of this commit there are three: enter_idle, the ExitIdle edge, and the Mode::Active timer. Worth saying that any mode holding applied has a scheduled retry, rather than naming one site that will drift again.
And the terminal tracing::error! a few lines down still says the guest "remains shrunk until the next idle cycle". The timer this commit adds is precisely what makes that false — an operator reading that line would conclude no recovery is scheduled and go looking for an idle transition that is no longer required.
1 participant
Problem
idle_target = used + 256MBdegenerates for an idle guest:MemAvailable ≈ total⇒used ≈ 0⇒ the target collapses onto the 384MB floor. The module's own invariant says the target is "never an unconditional constant" — the formula violated it in exactly the idle case it exists for.Measured 2026-07-29 (VZ, 16 GB VM, before #504 disabled the balloon): the descent asked the guest for 15.8 GB of its 15.96 GB available, drove
MemAvailableto literally 0 for ~98 s per cycle, then pressure-restored and repeated every ~8.5 min — ~2.8 cores averaged while "idle". During the zero-available window any running container is an OOM candidate.Change
The target is now the higher of two floors:
used + IDLE_MIN_RESERVE(1 GiB) — the guest keeps a working reserve;total - available / IDLE_RECLAIM_DIVISOR(2) — one entry reclaims at most half the slack, so no single descent walks the guest to the edge.A reclaim below
MIN_WORTHWHILE_RECLAIM(512 MB) is not repaid by its guest-side cost, so the balloon is kept instead.IDLE_BALLOON_FLOORis removed: the 1 GiB reserve strictly dominates the 384 MB floor, so it was unreachable code. Dropping it also removes a latentclamp(min, max)panic for any configured size below 384 MB.For the 16 GB idle guest above, the target moves from 384 MB to ~8 GB — the host still gets ~8 GB back, without the cliff.
Notes
BalloonDeps::reclaim_capableisfalseon every macOS backend (fix(core): disable the idle balloon — no macOS backend reclaims ballooned memory #504). This is the precondition for re-enabling shrinking anywhere (CORE-44 flips HV back only after this lands).final_target()callsidle_targetand the expected step sequence is derived fromnext_step, so a future policy change cannot silently desync the test.idle_guest_is_never_walked_to_the_floor,planned_reclaim_always_leaves_the_reserve(sweeps availability 0–16 GiB),entry_keeps_when_the_reclaim_is_not_worth_it,idle_target_never_exceeds_a_tiny_configured_size.Validation
cargo test -p arcbox-core --lib— 172 passed.cargo clippy -p arcbox-core --all-targets— zero warnings.cargo fmt --check— clean. No e2e run: the shrink path this governs cannot execute whilereclaim_capableis false, soidle_balloon(which pins the never-shrinks contract) is unaffected.Closes CORE-45.