Add opt-in QUIC_OPENSSL_SYMBOL_PREFIX to namespace bundled OpenSSL symbols (Linux)

[leikong]

Description

This PR adds an opt-in CMake cache variable, QUIC_OPENSSL_SYMBOL_PREFIX, that namespace-prefixes every globally-visible symbol in the bundled OpenSSL static archives (and rewrites MsQuic's own undefined references to match). When QUIC_OPENSSL_SYMBOL_PREFIX is left empty (the default), no new code runs and the build is byte-for-byte identical to today.

Motivation

When MsQuic is statically linked into a process that also pulls in another copy of OpenSSL — for example, a system libcrypto.so.3 brought in transitively by an unrelated dependency (a logging library, a database client, any C++ library that itself uses OpenSSL) — the two OpenSSL copies share the same global C symbols (SSL_CTX_new, EVP_*, BN_*, ERR_*, the per-module init constructors, …). The dynamic linker resolves every reference to the first definition loaded, so all callers — including MsQuic — silently end up sharing one OpenSSL's state machine while their headers and ABI assumptions came from the other. Typical symptoms:

• Crashes in OPENSSL_init_crypto / RAND_load_file when one OpenSSL's per-module init runs against the other's global registries.
• Spurious SSL handshake failures when callbacks installed against one SSL_CTX see the other's vtable layout.
• ABI mismatches when one OpenSSL is 3.0.x (system libcrypto.so.3 on Ubuntu 22.04 / RHEL 9) and MsQuic's bundled OpenSSL is 3.5.x (required for SSL_set_quic_tls_cbs).

Note that MsQuic's bundled-OpenSSL path always links libssl.a / libcrypto.a statically into MsQuic, regardless of QUIC_BUILD_SHARED — SHARED just controls whether the result is libmsquic.so or libmsquic.a. Building SHARED with --exclude-libs=ALL therefore strips the (statically linked) OpenSSL symbols from libmsquic.so's dynamic export table, which is sufficient when the consumer dlopens/links against libmsquic.so itself. It does not help:

• Static consumers that link libmsquic.a into a final binary — symbols flow straight into the executable's global table.
• Consumers that bundle libmsquic.a into their own .so — same exposure unless they also --exclude-libs it.
• Either case at runtime — even with exports hidden, both libcryptos coexist in one address space and OpenSSL's process-global init/registries (OPENSSL_init_crypto, error/atexit tables, RAND state) can still collide.

How it works

When QUIC_OPENSSL_SYMBOL_PREFIX=<prefix> is passed at CMake configure time, the build:

1. Builds the bundled OpenSSL submodule normally to produce libssl.a and libcrypto.a.
2. Extracts every globally-defined external symbol from those archives via nm --defined-only --extern-only and writes a redefine-syms file mapping each <sym> to <prefix><sym>.
3. Produces prefixed copies via objcopy --redefine-syms=<file> (touches both definitions and undefined references inside each member object).
4. Applies the same --redefine-syms step as a POST_BUILD action on libmsquic_platform.a so MsQuic's own undefined references to OpenSSL (from tls_openssl.c, tls_quictls.c, crypt_openssl.c, selfsign_openssl.c) get rewritten to match.
5. Routes the existing OpenSSL interface target at the prefixed archives, so the rest of the build is unchanged.

The result is a libmsquic.{a,so} whose only externally-visible OpenSSL symbols are the prefixed ones. The dynamic linker has no reason to resolve them against any other OpenSSL copy present in the same process.

Constraints

Constraint	Why
Linux only (CX_PLATFORM=linux)	Uses GNU binutils objcopy --redefine-syms. macOS would need llvm-objcopy >= 13 (untested); PE/COFF lacks a flat-namespace symbol table. Rejected with FATAL_ERROR on other platforms.
Bundled OpenSSL only	External/system OpenSSL is owned by the caller and cannot be renamed. Rejected with FATAL_ERROR if combined with QUIC_USE_EXTERNAL_OPENSSL, QUIC_OPENSSL_INCLUDE_DIR, QUIC_OPENSSL_LIB_DIR, QUIC_OPENSSL_ROOT_DIR, or QUIC_USE_SYSTEM_LIBCRYPTO.
Cross-compile aware	${CMAKE_NM} / ${CMAKE_OBJCOPY} are forwarded to the helper script, so aarch64-linux-gnu-objcopy etc. are used when configured.

Files

• cmake/openssl-prefix-rename.sh — helper script (gen-syms / apply modes; honors NM / OBJCOPY env vars).
• cmake/PrefixOpenSSLArchives.cmake — helper function prefix_openssl_archives(PREFIX … INPUT_TARGET … OUTPUT_TARGET …).
• CMakeLists.txt — new option + validation + plumbing in the bundled-OpenSSL branch.
• src/platform/CMakeLists.txt — POST_BUILD rename on libmsquic_platform.a.
• docs/OpenSSLSymbolPrefix.md — motivation, usage, caveats, verification recipe.

Future direction

The cleanest long-term solution is for OpenSSL itself to expose a configure-time --symbol-prefix= option that compiles every public symbol with the prefix baked in (an analog of BoringSSL's BSSL_NAMESPACE or LibreSSL's recurring discussion). I plan to file that issue upstream with this PR linked as concrete prior art demonstrating consumer demand. Until that lands, this CMake helper provides an equivalent at link time without requiring an OpenSSL fork.

Testing

Manual verification (local, this PR's branch)

Built on Linux x86_64 with -G Ninja -DQUIC_TLS_LIB=quictls -DQUIC_BUILD_SHARED=OFF:

With -DQUIC_OPENSSL_SYMBOL_PREFIX=msqtest_:

# 0 unprefixed defined globals in the renamed archives:
nm --defined-only --extern-only build/openssl-prefixed/msqtest_/libssl.a \
  | awk '$2 ~ /^[TDRBWVC]$/ {print $3}' | grep -vc '^msqtest_'    # → 0
nm --defined-only --extern-only build/openssl-prefixed/msqtest_/libcrypto.a \
  | awk '$2 ~ /^[TDRBWVC]$/ {print $3}' | grep -vc '^msqtest_'    # → 0

# 0 unprefixed OpenSSL undefs in msquic_platform.a, 161 prefixed undefs:
nm -u build/obj/Release/libmsquic_platform.a | awk '$1=="U"{print $2}' \
  | grep -E '^(SSL_|EVP_|BN_|ERR_|X509_|OPENSSL_|RAND_|RSA_|EC_|BIO_|ASN1_|PEM_|CRYPTO_)' \
  | wc -l                                                         # → 0
nm -u build/obj/Release/libmsquic_platform.a | awk '$1=="U"{print $2}' \
  | grep -c '^msqtest_'                                           # → 161

Without the option (default build):

• 138 normal unprefixed OpenSSL undefs in libmsquic_platform.a.
• build/openssl-prefixed/ directory not created.
• Build time identical to baseline (no new compilation; the rename rules simply don't fire).

Production validation

The same prefix-rename technique has been deployed in Microsoft's meru codebase (a consumer of MsQuic). It is currently validated across this matrix:

• Release x86_64
• Release_ASAN, Debug_ASAN, Debug_UBSAN, Debug_TSAN x86_64
• arm_debug_crosscompile (aarch64)
• clang_tidy

See microsoft/meru-common#4011 for the consumer-side integration that ports the same helper script + CMake module pattern that this PR upstreams.

MsQuic CI coverage

The default (option-empty) path is unchanged, so existing CI should be unaffected. The option-set path is Linux-only and opt-in, so it does not need to enter the default matrix. Happy to add a single Linux CI leg that exercises -DQUIC_OPENSSL_SYMBOL_PREFIX=msquic_ if maintainers think that's worthwhile.

Documentation

New file: docs/OpenSSLSymbolPrefix.md documents motivation, usage, constraints, and a verification recipe.

[Copilot]

Pull request overview

This PR introduces an opt-in build mechanism to namespace-prefix all bundled OpenSSL symbols (Linux, bundled OpenSSL only) to avoid global symbol collisions when MsQuic is statically linked into processes that also load another OpenSSL copy.

Changes:

• Adds a new CMake cache variable QUIC_OPENSSL_SYMBOL_PREFIX with validation and Linux-only enforcement in the bundled-OpenSSL build path.
• Implements CMake + shell helpers to generate an objcopy --redefine-syms map, produce prefixed libssl.a/libcrypto.a, and rewrite MsQuic’s OpenSSL references in libmsquic_platform.a.
• Adds documentation describing motivation, constraints, usage, and verification steps.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
CMakeLists.txt	Adds QUIC_OPENSSL_SYMBOL_PREFIX, validates constraints, and wires in the prefixing flow for bundled OpenSSL.
cmake/PrefixOpenSSLArchives.cmake	New CMake helper to generate syms map and produce renamed OpenSSL static archives + exported paths.
cmake/openssl-prefix-rename.sh	New helper script that generates the rename map with nm and applies it with objcopy.
src/platform/CMakeLists.txt	Adds a POST_BUILD step to rewrite OpenSSL references inside libmsquic_platform.a when prefixing is enabled.
docs/OpenSSLSymbolPrefix.md	New documentation for the feature, constraints, and verification.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.44%. Comparing base (23711d2) to head (069692d).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6031      +/-   ##
==========================================
+ Coverage   85.03%   85.44%   +0.40%     
==========================================
  Files          60       60              
  Lines       18792    18798       +6     
==========================================
+ Hits        15980    16062      +82     
+ Misses       2812     2736      -76     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[guhetier]

@leikong We are currently working toward making OpenSSL 3.5 the default option for distributions where it is available, with dynamic linking instead of static linking (which is an artifact of what was needed for QuicTLS).
Would dynamic linking OpenSSL 3.5 address your issue?

[leikong]

@leikong We are currently working toward making OpenSSL 3.5 the default option for distributions where it is available, with dynamic linking instead of static linking (which is an artifact of what was needed for QuicTLS). Would dynamic linking OpenSSL 3.5 address your issue?

OpenSSL 3.5 is not available on Ubuntu versions older than Ubuntu 26.04, and my team will not be able to move to 26.04 or newer any time soon. We are trying to enable static builds to avoid the need to copy MsQuic so files, in our environment, loading MsQuic so files from non-standard folders will require root, which we are trying to avoid.

[guhetier]

EDIT: Delete the previous comment, I was not getting the right scenario.

This is to run with OpenSSL 3.5 on a Linux distro without OpenSSL 3.5, and with the intention to statically link MsQuic + another dependency that dynamically link the system OpenSSL.

Unless it is possible for the project to dynamically link MsQuic (which solves the issue), this symbol renaming is needed.

[guhetier]

The approach seems ok.
It would be good to find a way to factor the logic around OpenSSL in its own cmake file, as a dependency of the platform, since it has grown in complexity so much, but that isn't in scope for this PR.

[guhetier]

Seems good to me.
As a disclaimer, please treat this as an unstable feature we will feel free to do breaking changes between versions as we improve the way we ingest OpenSSL.

[leikong]

@guhetier
Thanks for the review, would you please merge the PR for me? I don't have permission to do so:

[guhetier]

Done, thanks for the PR!