fix(balloon): stop idle-shrink oscillation (fail-open backoff + HV free-page-reporting gate)

[AprilNEA]

Balloon-subsystem fixes. Commits 1–2 fix the idle-shrink oscillation; commits 3–4 are cross-repo-audit findings in the same subsystem.

1. Back off idle shrink after repeated fail-opens (VZ + HV)

On a memory-pressured host, balloon inflation stalls and the controller oscillates indefinitely: VM idles → shrink → inflation churns guest pages through exhausted host swap and stalls → guest hits available_bytes=0 / floods Out of puff! → pressure watch times out → fail-open (restore full + note activity → idle→running) → 5-min idle timeout → repeat. macOS reports normal pressure throughout (kern.memorystatus_vm_pressure_level == 1), so there is no host signal to gate on.

Fix: track consecutive fail-opens; suppress the idle-shrink entry (no probe) within fail_open_backoff() — 10 min, doubling to a 60 min cap. A shrink that holds clears the streak, so reclaim self-heals when the host recovers. Corrected the over-claimed "Oscillation is structurally impossible" doc.

Observed: a genuinely idle 16 GiB System VM (~700 MiB of containers, 14.6 GiB free) on a host at 113/128 GiB + swap full oscillated every 5 minutes for over an hour — 5477 Out of puff lines in one daemon log.

2. Skip host-driven idle shrink on the HV backend

HV's arcbox-virtio-balloon advertises free page reporting + deflate-on-oom and the guest kernel enables reporting (CONFIG_VIRTIO_BALLOON select PAGE_REPORTING, verified in arcboxlabs/kernel), so the HV guest self-reclaims idle memory. BalloonDeps::guest_self_reclaims() (true on HV) gates the inflate/pressure-watch descent off; it runs only on VZ.

3. MADV_FREE on macOS (audit finding)

madvise(MADV_DONTNEED) is a no-op reclaim on Darwin — XNU maps VM_BEHAVIOR_DONTNEED to a deactivation hint that never releases physical backing, so every HV balloon inflate / free-page-reporting range freed zero bytes while returning success. Switch to MADV_FREE on macOS (matching libkrun's macOS balloon path). This is what makes commit 2's "HV self-reclaims via FPR" real rather than a silent no-op — without it, HV had no working reclaim at all.

4. Guard idle_target() against sub-384-MiB machines (audit finding)

idle_target() did clamp(IDLE_BALLOON_FLOOR, full); for a VM configured below the 384 MiB floor that is min > max, which std::clamp asserts on — crashing the daemon on the idle transition of any tiny machine. Cap the floor at full.

Tests

vm_lifecycle::balloon — 4 new controller tests + idle_target_below_floor_machine_does_not_panic; arcbox-virtio-balloon unchanged-green. 160 + 16 pass; clippy + fmt clean.

Notes / deferred

• End-to-end HV verification still awaits the HV blk-capacity bug fix (HV can't run Docker today). Commits 2–3 are grounded in the device feature bits + verified kernel config, and are backend-gated so VZ is unaffected.
• A host-memory-pressure gate was rejected: macOS's own pressure signal reads "normal" under this load. Backoff is signal-agnostic.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a5af2d529

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Wait before declaring the final shrink successful

When a retry reaches its final target, SETTLED only means the guest detector observed three healthy samples and armed (guest/arcbox-agent/src/memory_pressure.rs:25-36,133-152); it does not prove inflation has caught up, as the adjacent descent branch itself notes. Clearing the streak here means that if the remaining inflation subsequently stalls and this same watch reports pressure or silence, fail_open increments the streak from zero again, so identical failures after every final SETTLED frame remain permanently at the 10-minute base interval instead of growing to the promised one-hour backoff. Clear the streak only after the final target has survived a meaningful hold period rather than immediately upon detector arming.

Useful? React with 👍 / 👎.

[pullfrog]

✅ No new issues found.

Reviewed changes — adds exponential backoff to the idle-balloon controller so a shrink that fail-opens the same way every idle cycle (host itself OOM, inflation stalls, macOS reports "normal" pressure so there is no host signal to gate on) stops restaging its 5-minute reclaim storm.

• Track a fail-open streak — new consecutive_fail_opens + last_fail_open_at on BalloonController; fail_open() increments (saturating_add) and stamps the instant.
• Suppress idle entry within the backoff window — enter_idle() returns IdleUnshrunk without even probing while inside fail_open_backoff(streak).
• fail_open_backoff(n) — BASE (600s) << min(n-1, 3), capped at MAX (3600s); the shift clamp also guards against 1u32 << shift overflow for large streaks (tested at n=99).
• Self-heal via note_shrink_held() — clears the streak at final-target Settled, healthy poll at final, and on ExitIdle from any shrunk mode.
• Corrected the over-claiming doc — the "Oscillation is structurally impossible" comment now describes the periodic-reattempt failure mode and the backoff that fixes it.

ℹ️ Streak reset on any idle-exit is slightly broader than the doc claims

note_shrink_held runs on ExitIdle from a shrunk mode (Watching/Dwelling/Polling) and resets the streak unconditionally, while the method doc frames it as "settled at the final target, or was still healthy when activity arrived." In practice this is benign — reaching a shrunk mode without having fail-opened already implies the shrink was holding, and activity restores full memory regardless.

The only theoretical gap is a tokio::select! race where both a Pressure frame and an ExitIdle command are ready in the same poll and ExitIdle wins: the streak resets despite pending pressure, dropping an escalated backoff back to base on the next idle cycle. Surfaced for awareness only — no change needed; the design choice is documented and the worst case costs one extra reclaim attempt.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes — this run covers the second commit (a947f25), which gates the whole host-driven idle-shrink descent off on the HV backend, on the grounds that HV's guest self-reclaims idle memory via virtio-balloon free page reporting.

• Add BalloonDeps::guest_self_reclaims() — RealBalloonDeps returns true iff the System VM backend is HV (self.shared.backend(), an atomic load, so a runtime VZ↔HV switch is picked up on the next idle entry); VZ's traditional balloon has no such mechanism and keeps the descent.
• Short-circuit enter_idle() on HV — returns Mode::Active (controller goes inert with no retry timer) when guest_self_reclaims(), checked after full_memory_bytes() but before the fail-open backoff/probe path, so HV never shrinks, opens no pressure watch, and fires no activity callback.
• Document the rationale — module doc + the new trait-method doc explain that HV advertises free page reporting + deflate-on-oom so the guest kernel drains idle pages itself, making the host-driven target policy both unnecessary and the sole source of the inflate-stall pathology.
• Add self_reclaiming_backend_skips_idle_shrink — asserts a self-reclaiming backend produces no target, no watch, and activity_count == 0.

The central premise checks out end-to-end: arcbox-virtio-balloon advertises VIRTIO_BALLOON_F_REPORTING + VIRTIO_BALLOON_F_DEFLATE_ON_OOM (virt/arcbox-virtio-balloon/src/lib.rs:132-138), and the reporting-queue handler actually releases the guest-reported free ranges via madvise(MADV_DONTNEED) (handle_reported_range, lib.rs:364-397) — so HV genuinely returns idle memory to the host rather than merely acking descriptors. Returning Mode::Active rather than IdleUnshrunk is the right choice for a backend that will never shrink (no pointless retry timer; Mode is private with no external consumer). The author's ⚠️ note that end-to-end HV verification is deferred until the HV blk-capacity bug is fixed stands; the change is fully backend-gated, so VZ is unaffected.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[greptile-apps]

Greptile Summary

This PR updates idle balloon reclaim behavior across the VZ and HV backends. The main changes are:

• Adds bounded backoff after repeated fail-open shrink attempts.
• Skips host-driven idle shrink for HV guests.
• Uses MADV_FREE to release ballooned pages on macOS.
• Handles VM memory sizes below the balloon floor.

Confidence Score: 5/5

This looks safe to merge after resolving the existing capability-negotiation finding.

No additional blocking issue was found in the updated code.

The remaining HV capability concern is already covered by the existing finding.

T-Rex Logs

What T-Rex did

• A build/run was logged that captured the exact command, working directory, complete compiler output, and exit status 101, and it identified an unresolved import KvmVcpu at virt/arcbox-hypervisor/src/linux/vm/virtual_machine.rs:10 as the blocker.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
app/arcbox-core/src/vm_lifecycle/actor.rs	Adds the HV self-reclaim decision used by the balloon controller.
app/arcbox-core/src/vm_lifecycle/balloon/controller.rs	Adds fail-open backoff, streak reset behavior, and the HV idle-shrink gate.
app/arcbox-core/src/vm_lifecycle/balloon/mod.rs	Caps the balloon floor by configured memory for small VMs.
virt/arcbox-virtio-balloon/src/lib.rs	Selects platform-specific memory release advice for ballooned and reported pages.

_{Reviews (2): Last reviewed commit: "fix(balloon): use MADV_FREE on macOS and..." | Re-trigger Greptile}

[pullfrog]

✅ No new issues found.

Reviewed changes — this run covers the third commit (8a9269a), which switches the HV balloon device to MADV_FREE on macOS and hardens idle_target against VMs configured below the reclaim floor.

• MADV_FREE on macOS via RELEASE_ADVICE — new backend-conditional const (MADV_FREE on macOS, MADV_DONTNEED elsewhere) swapped into both madvise sites (handle_pfn_list inflate, handle_reported_range reporting) in virt/arcbox-virtio-balloon/src/lib.rs, with module/field doc updates. XNU maps MADV_DONTNEED to a mere deactivation hint so the VM footprint never shrinks; MADV_FREE is the advice that actually lets the host reclaim under pressure (matching libkrun's macOS balloon path).
• Guard idle_target against tiny VMs — .clamp(IDLE_BALLOON_FLOOR.min(full), full) in balloon/mod.rs avoids the min > max clamp panic when a VM is configured below the 384 MiB floor; a below-floor machine has nothing to reclaim, so the target is full and entry decides Keep. Covered by the new idle_target_below_floor_machine_does_not_panic regression test.

The MADV_FREE swap is sound for both descriptor paths: the guest keeps inflated pages off-limits until deflate, and free-page-reporting pages are re-initialized by the guest on realloc, so MADV_FREE's stale-content-until-reclaim behavior stays within the balloon / free-page-reporting contract. The one behavioral consequence — RSS now drops only when the host is under memory pressure rather than immediately — is inherent to what XNU can actually do and is called out in the constant's doc comment. The idle_target change matches the corrected file (the range-diff artifact happens to display both the old and new clamp lines together; the committed source has the single correct clamp). Balloon device code is HV/macOS-only and untestable in Linux CI, consistent with the repo's HV-verification-deferred posture.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

CI failure is unrelated to this PR — no changes were pushed.

The failing check is a timing-sensitive flake in common/splicetcp, a crate this PR does not touch. CI reported tcp_bridge::tests::poll_fast_path_respects_guest_window ("second tranche must respect the re-advertised window, got 68930"). This PR only modifies balloon files (app/arcbox-core/src/vm_lifecycle/balloon/*, vm_lifecycle/actor.rs, virt/arcbox-virtio-balloon/src/lib.rs); git diff --merge-base origin/master -- common/splicetcp is empty.

Evidence it's a pre-existing load flake, not a regression:

• The test passes 8/8 in isolation but fails ~100% under full-crate parallel cargo test -p splicetcp --lib.
• The specific failing test varies by run — locally upload_ack_never_exceeds_host_writable failed instead of the one CI hit. Both rely on wall-clock deadlines + tokio::time::sleep drain loops, so they break under CPU contention.

I did not modify splicetcp — fixing an unrelated flaky network test from a balloon PR would be inappropriate scope. Recommendation: re-run the check suite (the flake is intermittent on the CI runner, which showed only 1 failure), or address the splicetcp timing tests in a dedicated PR.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}