[CP] Add support for multiprocess sharing + Fix a QTIP xdp bug (#5798) (#5747) (#5395)#6050
Merged
Conversation
Since the XDP datapath is initialized on a "best-effort" basis, it will fall back to normal OS sockets whenever there is an issue with creating the XDP datapath. One check that it does is whether or not we previously have already created the same datapath on the same IP/port for local/remote. The assumption is, we already hooked into XDP and it's already snooping on the same IP/port, no need to do it again. It will fall back to using normal OS sockets. This is misleading because when using QTIP, it is possible that a client machine has connections bound to the same ephemeral port, except the difference is it's UDP/TCP. Previously, this was not possible as MsQuic is using either ALL QTIP, or None at all. Fixes #5718 <img width="3997" height="2507" alt="image" src="https://github.com/user-attachments/assets/8497efd7-5cd3-4502-9282-8d60b35fc352" /> NOTE: On any machine, either ALL listeners have QTIP enabled or all of them have them disabled. This is controlled via the GLOBAL MsQuic settings. It's only client connections that can vary this on a per-connection basis with local settings. Logs from CI. QTIP.md --------- Co-authored-by: Guillaume Hetier <guhetier@microsoft.com> Co-authored-by: Michael Friesen <3517159+mtfriesen@users.noreply.github.com>
Fixes #5795 The XDP datapath can be configured to intercept packets based on QUIC Connection ID instead of local port. This behavior existed in MsQuic but was not heavily exercised until recently. One issue was that MsQuic always attempted to reserve UDP / TCP sockets for each application server process. But for multiple server processes that may want to share a single port, we would run into port collision errors. This PR adds support for CIBIR across multiple processes on the same port and document the behavior - **Option 1:** MsQuic delegates port protection to applications and provides best practice recommendations. > **Analysis:** Deferring the responsibility of port protection and isolation to the application has the upside of enabling the most potential scenarios but could also be a footgun. - **Option 2:** MsQuic makes sure *some* persistent reservation exists at a port. > **Analysis:** Note that LookUpPersistentReservation does not require admin privileges, but CreatePersistentReservation does require admin. This is useful in that if any reservation exists on a port, we can reasonably trust that an admin knew what they were doing when they created it. safety, and ensure the consumers of CIBIR must know what they are doing. - **Option 3:** MsQuic creates per-proc sockets with SIO_CPU_AFFINITY, but does not reserve the port. > **Analysis:** If another unrelated app creates a socket with SIO_CPU_AFFINITY, then they can bind to the CIBIR shared port. But for all other apps, trying to bind a socket to a CIBIR port will result in a collision. MsQuic's stance is that the application takes responsibility for book-keeping and protecting sharing shared local ports when using XDP + CIBIR. - Multiple MsQuic processes in Cibir+XDP mode can share a local port for **server sockets only.** - Applications should also not assume the shared port is safe from other non-Msquic processes binding to it. >MsQuic will NOT make an OS port reservation for server sockets when CIBIR+XDP is enabled. Clients on the other hand, MsQuic will always make OS port reservations. - Applications using server sockets + CIBIR/XDP must specify a well-known local port. - Server sockets with XDP+CIBIR both enabled/available will skip OS port reservation and OS socket creation to rely on XDP. > any failures plumbing xdp rules will bubble up as a socket creation error to the app. Can't fall back to OS sockets. - Client sockets with XDP+CIBIR both enabled/available will still do OS port reservation and socket creation but rely on XDP. > any failures plumbing xdp rules will silently fall back to using OS sockets. CIBIR transport negotiation can still work without XDP. - Server sockets with CIBIR enabled but XDP not available/enabled will do OS port reservation and fall back to OS sockets - Client sockets with CIBIR enabled but XDP not available/enabled will do OS port reservation and fall back to OS sockets - Windows has the https://learn.microsoft.com/en-us/windows/win32/api/iphlpapi/nf-iphlpapi-createpersistentudpportreservation API, to allow sysadmins to pre-allocate a block of ports and disallow other applications from binding to it. - A well known CIBIR registry key can be used to detail shared ports, and sysadmins can coordinate their system such that other apps will not bind to those ports. - ALE policies; applications can configure WFP to block certain ports from being binded to by other apps. A new DataPathTest was added. Settings.md
guhetier
previously approved these changes
May 29, 2026
Collaborator
guhetier
left a comment
guhetier
left a comment
There was a problem hiding this comment.
nit: Consider cherry-picking PRs one at a time, it makes it easier to see changes in the release notes.
guhetier
reviewed
May 29, 2026
Remove stale ListenerPartitionIndexSet CLOG entries that were cherry-picked from main but don't exist in the release/2.5 source. Regenerate clog.sidecar to match current branch state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (72.72%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## release/2.5 #6050 +/- ##
===============================================
- Coverage 87.17% 86.74% -0.43%
===============================================
Files 59 59
Lines 18212 18229 +17
===============================================
- Hits 15876 15813 -63
- Misses 2336 2416 +80 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
guhetier
reviewed
Jun 3, 2026
| Rules[RuleSize].Action = XDP_PROGRAM_ACTION_REDIRECT; | ||
| Rules[RuleSize].Redirect.TargetType = XDP_REDIRECT_TARGET_TYPE_XSK; | ||
| Rules[RuleSize].Redirect.Target = NULL; | ||
| Rules[RulesSize].Match = XDP_MATCH_UDP_DST; |
Collaborator
There was a problem hiding this comment.
You had to also cherry-pick !5395, basically, is that correct? Let's list in in the PR description / title so it is easy to map find it.
guhetier
approved these changes
Jun 3, 2026
Collaborator
guhetier
left a comment
guhetier
left a comment
There was a problem hiding this comment.
Looks good assuming the ids of all 3 cherry-picked PRs are added to the title.
ProjectsByJackHe
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cherry picking:
#5798
and
#5747
Testing
CI
Documentation
No