Feature RFC: Opt-In CPU Sharing and Runtime CPU Reallocation for Bao on RISC-V (QEMU Virt + UltraRISC UR-DP1000)

[showmeth3code]

• Status: Draft
• Authors: Zhixiang Wei, Ziyang Zhang, Minglang Huang, Yun Wang, Tianmai Deng, Jing Zhang, Mingyuan Xia and Zhengwei Qi from Shanghai Jiao Tong University & UltraRISC (fusionvisor, we have a paper version in the submission process)

• Target Project: bao-project/bao-hypervisor
• Phase 1 Arch: RISC-V
• Phase 1 Platforms: qemu-riscv64-virt, UR-DP1000
• Scope: Experimental, opt-in, default-off

---

Summary

An opt-in CPU sharing model for Bao on RISC-V to enable policy-bounded runtime CPU reallocation while keeping Bao’s static-partition guarantees for critical workloads.

Includes:

• Scheduler domains with explicit priority and CPU bounds
• Runtime CPU reallocation between domains
• Controlled online CPU expansion for static-partition VMs
• Fail-safe rollback; feature is disabled by default

---

Motivation

Static partitioning gives determinism and strong isolation, but can underutilize CPUs when workload intensity varies.

Goal is mixed-criticality support:

• Critical VMs: deterministic CPU reserve
• Non-critical workloads: optional elasticity via bounded reallocation

---

Goals / Non-goals

Goals

• Opt-in CPU sharing mode (RISC-V)
• Runtime CPU reallocation between scheduler domains
• Online CPU expansion for static-partition VMs (within configured limits)
• No behavior change when disabled
• Phase 1 support for qemu-riscv64-virt and UR-DP1000

Non-goals

• Memory hotplug (Continue in the next few RFCs)
• VirtIO-GPIO / generic virtio backend work (Continue in the next few RFCs)
• Non–RISC-V support in Phase 1
• Replacing Bao static partitioning as the default model

---

Design

Model

Two types of domains:

• Static critical domain(s): deterministic execution, reserved CPUs
• Dynamic shared domain(s): CPU sharing with bounded reallocation

CPU movement is allowed only under explicit policy + safety constraints.

Scheduler domains

Each domain includes:

• priority
• algo (initially FIFO)
• boot_cpu_num, min_cpu_num, max_cpu_num
• enable_rebalance
• runtime counters + decision window state

Arbitration: FIFO + priority-based domain selection (initial implementation).

CPU ownership invariants (must hold)

1. A physical CPU has exactly one owner at a time.
2. Domain CPU count stays within [min_cpu_num, max_cpu_num].
3. Critical domain reserve must not be violated.
4. Reallocation is atomic w.r.t. visible ownership.
5. On failure, rollback to the last valid ownership mapping.

Runtime signals and policy

Decision inputs (initially IPC-related window signals):

• upper/lower thresholds
• sliding window size
• upset/good ratio thresholds

Intent:

• borrow CPU on high-priority pressure
• return CPU when pressure drops
• avoid oscillation via windowing + gating

Reallocation protocol

1. Check policy eligibility + bounds
2. Pick donor CPU
3. Set both domains to REALLOCATING
4. Transfer CPU ownership + scheduler context
5. Update vCPU mapping + interrupt/timer state
6. ACK, return to NORMAL

Failure handling:

• restore original ownership + scheduler state
• continue running without panic escalation

Static VM online CPU expansion

Allowed only if:

• VM policy enables expansion
• donor domain stays above its minimum reserve
• target VM stays below its configured maximum

Uses existing runtime control paths (SBI/HSM/IPI); no reboot required.

SBI / timer / interrupt integration

• periodic timer sampling as decision points
• cross-core coordination via CPU messaging / IPI
• integrate with HSM start/stop semantics for vCPU transitions

Extension IDs / call semantics are implementation details, but must be clearly namespaced + documented.

Configuration (illustrative)

• Global: scheduler_num, schedulers[]
• Per-scheduler: priority, algo, boot_cpu_num, min_cpu_num, max_cpu_num, enable_rebalance, window/threshold params
• Per-VM: static/dynamic CPU policy marker, scheduler association, per-VM rebalance enable

Default behavior

Feature is disabled by default. When disabled:

• Bao behavior remains unchanged (static partitioning)
• no CPU reallocation path is active

---

Platform scope (Phase 1)

• qemu-riscv64-virt
• UR-DP1000

Requires correct platform CPU-ID mapping, including non-contiguous hart ID handling where applicable.

---

Compatibility

• Backward compatible via default-off gating
• Existing static configs work unchanged
• New config fields are optional unless CPU sharing is enabled
• No ABI/API break for non-participating VMs

---

Security / isolation / safety

Risks

• incorrect ownership transitions
• migration races
• interrupt routing inconsistencies
• policy oscillation → jitter spikes

Required controls

• strict invariants
• lock discipline in reallocation path
• explicit transition state machine
• bounded policy + anti-thrashing windows
• trace hooks for every ownership transfer

---

Reference-only performance notes (not merge gates)

• Cyclictest latency in static mode: ~5us / 7us / 12us (min/avg/max), with >99.999% samples below 10us
• Static VM online CPU expansion overhead: millisecond range
• Workload acceleration (selected expansion scenarios): ~16.5%–21.3%
• Scheduler-to-scheduler reallocation: high-priority improvement ~20.9%, low-priority recovery gain ~24.2%

Must be re-measured in upstream CI/lab.

---

Testing plan

Functional

• ownership transition correctness
• domain bounds enforcement
• ACK + rollback
• static VM online CPU expansion

Concurrency / fault injection

• simulated transfer failure
• timer/interrupt races
• repeated high-frequency attempts

Performance

• cyclictest jitter vs baseline Bao
• mixed workload throughput
• reallocation overhead + frequency

Stability

• 12h+ soak tests
• no ownership leaks, deadlocks, unbounded oscillation

---

Rollout plan

• Phase A: infra + config gating (no active reallocation)
• Phase B: scheduler-domain reallocation (experimental)
• Phase C: static VM online CPU expansion (experimental)
• Phase D: docs + platform matrix hardening

Each phase should be independently mergeable and testable.

---

Alternatives / drawbacks / open questions

Alternatives

• pure static partitioning (simplest, least flexible)
• fully dynamic fair scheduling (better avg utilization, weaker determinism)
• external privileged management VM (more flexible, larger TCB/complexity)

Drawbacks

• higher implementation complexity
• policy tuning burden
• larger test matrix (platform × policy × workload)

Open questions

• add schedulers beyond FIFO in Phase 1?
• minimal upstream-visible policy surface for first merge?
• standard telemetry format for reallocation traces?
• keep UR-DP1000 name or generalize later?

---

Conclusion

Adds a conservative, default-off CPU elasticity option to Bao on RISC-V: keep static determinism by default, allow bounded runtime CPU reallocation only when explicitly enabled.