Context switching overhead for RVV (Vector) in static RISC-V Type-1 Hypervisors under Edge AI workloads

[alex-kolovsky]

Hi! I'm currently designing a static Type-1 hypervisor for RISC-V with lazy vector context switching tailored for Edge AI workloads. I wanted to ask if you've encountered significant memory/latency overhead when switching between multiple neural inference models on a single core, and how you approach this constraint in your architecture. Would love to hear your thoughts if you have a moment!

[josecm]

Hey @alex-kolovsky,

You may have noticed that Bao is a static-partitioning hypervisor, meaning that there is no CPU sharing and, consequently, no context switching.

That said, we are indeed working on a version of Bao that supports CPU sharing. Our focus remains on determinism, so we do not rely on lazy state switching. At configuration time, it must be explicitly defined which VMs may access each coprocessor or architectural extension that carries substantial architectural state.

Regarding RISC-V and the V extension specifically, the architecture provides controls that can be used to avoid unnecessary state save and restore operations. And, of course, several implementation and memory-layout techniques can be applied to reduce the overhead further.

Could you tell us more about the project you are developing, its target use cases, and what you mean specifically by a hypervisor tailored for Edge AI workloads?

[alex-kolovsky]

Hey @josecm, thanks for your detailed answer! I asked about this because I want to get a sense of how much this topic has been discussed and if I’m moving in the right direction.

My hypervisor is meant for industrial and consumer sectors related to AI (like self-driving cars, drones, industrial robots, etc.); it helps run multiple local neural networks in limited conditions without overloading the CPU (as would happen without lazy switching).

As for the predictability of lazy switching, sure, it’s probably worse than without it, but still, if you compare the potential timing fluctuations of lazy switching with those of multiple AIs, it’s likely to be negligible.