task: assign a PCID to each task page-table root by tanish111 · Pull Request #1157 · coconut-svsm/svsm

tanish111 · 2026-07-04T09:58:10Z

Process Context IDs let the CPU tag TLB entries with the active page-table root. Without that, every context switch that reloads CR3 must invalidate the whole TLB, even when the outgoing and incoming tasks are unrelated and their mappings could safely coexist in the cache. Giving each live task root its own PCID (1..4095; 0 reserved for per-CPU transitional tables) is the foundation for cheaper switches later and keeps stale translations from one dead root from matching a reused tag.

Each task root gets a TaskPcid from a global pool of 4096 IDs (PCID 0 stays reserved for per-CPU transitional page tables). The allocator keeps a simple in_use bitmap: allocation scans from 1 upward for the first free slot and marks it taken; no flush happens at allocation time because a freshly handed-out ID either was never used or was fully cleaned when the previous owner died. Freeing is tied to task lifetime via RAII — when the Task is destroyed, TaskPcid’s Drop runs release(), which first broadcasts a PCID-targeted TLB shootdown on every CPU (flush_tlb_pcid_sync, INVPCID on native / INVLPGB+TLBSYNC on SNP) so no stale translations remain tagged with that ID, then clears the in_use bit so the number can be reused. That guarantees the invariant that any PCID loaded into CR3 for a new root has no valid entries under its tag anywhere in the system.

tanish111 · 2026-07-04T10:10:11Z

I used Linux as a reference for both instruction encodings. For INVPCID, I looked at arch/x86/include/asm/invpcid.h, which uses a 16-byte descriptor ({pcid, addr}) passed as a memory operand with the invalidation type in any GPR — generic across all 4 INVPCID types since it has several call sites. I followed the same descriptor layout and operand convention in flush_pcid_percpu, but narrowed it to type 1 (single-context) only, since my PCID allocator only ever needs to flush everything tagged with one PCID, never a single address.
Linux Code:-

struct { u64 d[2]; } desc = { { pcid, addr } };
asm volatile("invpcid %[desc], %[type]":: [desc] "m" (desc), [type] "r" (type) : "memory");

For INVLPGB and TLBSYNC, I referenced arch/x86/include/asm/tlb.h. Linux's __invlpgb() always sets INVLPGB_FLAG_ASID in rax and packs edx as pcid << 16 | asid. I mirrored both of these in my flush_tlb_scope PCID branch.
Linux Code:-

u64 rax = addr | flags | INVLPGB_FLAG_ASID;
u32 ecx = (stride << 31) | (nr_pages - 1);
u32 edx = (pcid << 16) | asid;
asm volatile(".byte 0x0f, 0x01, 0xfe" :: "a" (rax), "c" (ecx), "d" (edx));

tanish111 · 2026-07-04T10:10:32Z

@luigix25 can you take first pass before I mark it ready for review

luigix25

In the commit message you mention GSoC, this is not necessary.

You are doing changes that are not trivial at all, PLEASE explain them.

luigix25 · 2026-07-06T08:42:37Z

+
+#[derive(Debug)]
+struct PcidAllocator {
+    in_use: [bool; PCID_COUNT],


not sure it's worth fixing, but we are using a whole page just for pcid tracking.

Need to discuss this further. Since we're only using a very small subset of PCIDs (around 8–10), we should also look for a way to avoid reserving all 4096 slots upfront. Perhaps we could use a data structure that allocates entries lazily, only when they're needed.

luigix25 · 2026-07-06T09:12:11Z

+        let cr3 = {
+            let mut cr3 = (*next).page_table.lock().cr3_value().bits();
+            if let Some(pcid) = (*next).pcid() {
+                cr3 |= usize::from(pcid);
+            }
+            cr3
+        };


Claude pointed out this to me, and then I checked in the intel manual (https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html) Section 5.10.4.1

Without bit 63 set, all writes to CR3 will flush all the entries from the old PCID.

Open question for experts: Should we also set bit 63 to preserve the entries when a context switch happens?

luigix25 · 2026-07-06T09:36:36Z

+        if let Some(pcid) = self.pcid {
+            flush_pcid_percpu(pcid);
+            return;
+        }


why this change?

@luigix25 this is new branch which will handle TLB flush requests with pcid. If TlbFlushScope has pcid set in it I would run flush_pcid_percpu and not global __flush_tlb_global_percpu. I wanted to keep the main caller unchanged (TaskPcid::drop → flush_tlb_pcid_sync → flush_percpu_all).

msft-jlange · 2026-07-07T00:08:43Z

+    }
+
+    fn alloc(&mut self) -> Result<u16, SvsmError> {
+        for pcid in 1..PCID_COUNT as u16 {


We can do better than a linear search across an array of bool. A BitmapAllocatorTree<BitmapAllocator64> would be much faster, not to mention significantly more compact.

Yes it's a good idea I would update it with BitmapAllocatorTree<BitmapAllocator64>

One thing to note is that BitmapAllocatorTree<BitmapAllocator64> can allocate up to 1024 PCIDs, which is well beyond our current needs but still below the architectural maximum of 4096. Using BitmapAllocatorTree<BitmapAllocatorTree<BitmapAllocator64>> increases the capacity to 16,384 entries, which is far more than the 4096 PCIDs we actually need. Any suggestion on what should we go ahead with?

msft-jlange · 2026-07-07T00:09:40Z

+                return Ok(pcid);
+            }
+        }
+        Err(SvsmError::Alloc(AllocError::OutOfMemory))


So does this mean that we are limiting the kernel to a maximum number of processes that cannot exceed the number of PCIDs? That seems overly restrictive. We need a better story for the situation in which we try to create a new process and all PCIDs are in use.

@msft-jlange yes we are moving ahead with this limitation considering svsm has a max of about 4-8 tasks it wouldn't hurt current use cases in any way but yes I agree it's not a good story.

Why can we not say that if a task cannot allocate a PCID, then it can simply run with PCID disabled? We are already using an Option to hold the PCID in a task, so surely we can support a mix of Some and None on the same system.

To support that, we’d need to toggle CR4.PCIDE at runtime every-time we fallback. Right now, CR4 is configured only once during initialization. (without resetting cr4 if we don't set self.pcid during context switch it falls back to 0 and cpu cannot distinguish their TLB entries by PCID leading to accessing corrupt memory)

I discussed a few alternatives with @joergroedel and @luigix25 during my mentorship meeting. One option is to associate the PCID with TaskMM rather than individual Task, so all threads belonging to the same task share a single PCID.

This also aligns well with the current SVSM design, which supports at most 1024 TaskMM instances—well below the architectural limit of 4096 PCIDs—making the allocation story much simpler.

To support that, we’d need to toggle CR4.PCIDE at runtime every-time we fallback.

That's not required. If you reserve PCID zero to mean "overflow PCID", then you can simply ensure that you set CR3[63]=0 when you load CR3 for a task that does not have a unique PCID, and all of the non-global entries will be flushed. The result is that every task that has PCID=0 would get a full TLB flush on every CR3 load, exactly as if there were no separate PCIDs. The fact is that the PCID architecture is designed so that you can set CR4.PCIDE=1 and get no impact whatsoever in TLB management until you start opting in by setting CR3[PCID] to non-zero and by setting CR3[63]=1. So it's trivial to let PCID and non-PCID address spaces coexist.

This also aligns well with the current SVSM design, which supports at most 1024 TaskMM instances—well below the architectural limit of 4096 PCIDs—making the allocation story much simpler.

I don't know how long the limit of 1024 instances will persist. If we are able to expand that limit to 8192 or higher, then we would end up being stuck because of the PCID assignment limitation. In general, the code is most robust when we don't design in any implicit relationships between resource limits in one area and resource limits in another area.

Process Context IDs let the CPU tag TLB entries with the active page-table root. Without that, every context switch that reloads CR3 must invalidate the whole TLB, even when the outgoing and incoming tasks are unrelated and their mappings could safely coexist in the cache. Giving each live task root its own PCID (1..4095; 0 reserved for per-CPU transitional tables) is the foundation for cheaper switches later and keeps stale translations from one dead root from matching a reused tag. Each task root gets a TaskPcid from a global pool of 4096 IDs (PCID 0 stays reserved for per-CPU transitional page tables). The allocator keeps a simple in_use bitmap: allocation scans from 1 upward for the first free slot and marks it taken. Freeing is tied to task lifetime when the Task is destroyed, TaskPcid’s Drop runs release(), which first broadcasts a PCID-targeted TLB shootdown on every CPU (flush_tlb_pcid_sync, INVPCID on native / INVLPGB+TLBSYNC on SNP) so no stale translations remain tagged with that ID, then clears the in_use bit so the number can be reused. That guarantees the invariant that any PCID loaded into CR3 for a new root has no valid entries under its tag anywhere in the system. Signed-off-by: tanish111 <tanishdesai37@gmail.com>

tanish111 requested review from 00xc, joergroedel, msft-jlange, peterfang and stefano-garzarella as code owners July 4, 2026 09:58

tanish111 marked this pull request as draft July 4, 2026 09:58

luigix25 suggested changes Jul 6, 2026

View reviewed changes

msft-jlange reviewed Jul 7, 2026

View reviewed changes

tanish111 force-pushed the pcid_allocation branch from 6089728 to 7df7993 Compare July 7, 2026 17:46

Uh oh!

Conversation

tanish111 commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanish111 commented Jul 4, 2026

Uh oh!

tanish111 commented Jul 4, 2026

Uh oh!

luigix25 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msft-jlange Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tanish111 commented Jul 4, 2026 •

edited

Loading

msft-jlange Jul 7, 2026 •

edited

Loading