Add opt-in QUIC_OPENSSL_SYMBOL_PREFIX to namespace bundled OpenSSL symbols (Linux)

[leikong]

Description

This PR adds an opt-in CMake cache variable, QUIC_OPENSSL_SYMBOL_PREFIX, that namespace-prefixes every globally-visible symbol in the bundled OpenSSL static archives (and rewrites MsQuic's own undefined references to match). When QUIC_OPENSSL_SYMBOL_PREFIX is left empty (the default), no new code runs and the build is byte-for-byte identical to today.

Motivation

When MsQuic is statically linked into a process that also pulls in another copy of OpenSSL — for example, a system libcrypto.so.3 brought in transitively by an unrelated dependency (a logging library, a database client, any C++ library that itself uses OpenSSL) — the two OpenSSL copies share the same global C symbols (SSL_CTX_new, EVP_*, BN_*, ERR_*, the per-module init constructors, …). The dynamic linker resolves every reference to the first definition loaded, so all callers — including MsQuic — silently end up sharing one OpenSSL's state machine while their headers and ABI assumptions came from the other. Typical symptoms:

• Crashes in OPENSSL_init_crypto / RAND_load_file when one OpenSSL's per-module init runs against the other's global registries.
• Spurious SSL handshake failures when callbacks installed against one SSL_CTX see the other's vtable layout.
• ABI mismatches when one OpenSSL is 3.0.x (system libcrypto.so.3 on Ubuntu 22.04 / RHEL 9) and MsQuic's bundled OpenSSL is 3.5.x (required for SSL_set_quic_tls_cbs).

Note that MsQuic's bundled-OpenSSL path always links libssl.a / libcrypto.a statically into MsQuic, regardless of QUIC_BUILD_SHARED — SHARED just controls whether the result is libmsquic.so or libmsquic.a. Building SHARED with --exclude-libs=ALL therefore strips the (statically linked) OpenSSL symbols from libmsquic.so's dynamic export table, which is sufficient when the consumer dlopens/links against libmsquic.so itself. It does not help:

• Static consumers that link libmsquic.a into a final binary — symbols flow straight into the executable's global table.
• Consumers that bundle libmsquic.a into their own .so — same exposure unless they also --exclude-libs it.
• Either case at runtime — even with exports hidden, both libcryptos coexist in one address space and OpenSSL's process-global init/registries (OPENSSL_init_crypto, error/atexit tables, RAND state) can still collide.

How it works

When QUIC_OPENSSL_SYMBOL_PREFIX=<prefix> is passed at CMake configure time, the build:

1. Builds the bundled OpenSSL submodule normally to produce libssl.a and libcrypto.a.
2. Extracts every globally-defined external symbol from those archives via nm --defined-only --extern-only and writes a redefine-syms file mapping each <sym> to <prefix><sym>.
3. Produces prefixed copies via objcopy --redefine-syms=<file> (touches both definitions and undefined references inside each member object).
4. Applies the same --redefine-syms step as a POST_BUILD action on libmsquic_platform.a so MsQuic's own undefined references to OpenSSL (from tls_openssl.c, tls_quictls.c, crypt_openssl.c, selfsign_openssl.c) get rewritten to match.
5. Routes the existing OpenSSL interface target at the prefixed archives, so the rest of the build is unchanged.

The result is a libmsquic.{a,so} whose only externally-visible OpenSSL symbols are the prefixed ones. The dynamic linker has no reason to resolve them against any other OpenSSL copy present in the same process.

Constraints

Constraint	Why
Linux only (CX_PLATFORM=linux)	Uses GNU binutils objcopy --redefine-syms. macOS would need llvm-objcopy >= 13 (untested); PE/COFF lacks a flat-namespace symbol table. Rejected with FATAL_ERROR on other platforms.
Bundled OpenSSL only	External/system OpenSSL is owned by the caller and cannot be renamed. Rejected with FATAL_ERROR if combined with QUIC_USE_EXTERNAL_OPENSSL, QUIC_OPENSSL_INCLUDE_DIR, QUIC_OPENSSL_LIB_DIR, QUIC_OPENSSL_ROOT_DIR, or QUIC_USE_SYSTEM_LIBCRYPTO.
Cross-compile aware	${CMAKE_NM} / ${CMAKE_OBJCOPY} are forwarded to the helper script, so aarch64-linux-gnu-objcopy etc. are used when configured.

Files

• cmake/openssl-prefix-rename.sh — helper script (gen-syms / apply modes; honors NM / OBJCOPY env vars).
• cmake/PrefixOpenSSLArchives.cmake — helper function prefix_openssl_archives(PREFIX … INPUT_TARGET … OUTPUT_TARGET …).
• CMakeLists.txt — new option + validation + plumbing in the bundled-OpenSSL branch.
• src/platform/CMakeLists.txt — POST_BUILD rename on libmsquic_platform.a.
• docs/OpenSSLSymbolPrefix.md — motivation, usage, caveats, verification recipe.

Future direction

The cleanest long-term solution is for OpenSSL itself to expose a configure-time --symbol-prefix= option that compiles every public symbol with the prefix baked in (an analog of BoringSSL's BSSL_NAMESPACE or LibreSSL's recurring discussion). I plan to file that issue upstream with this PR linked as concrete prior art demonstrating consumer demand. Until that lands, this CMake helper provides an equivalent at link time without requiring an OpenSSL fork.

Testing

Manual verification (local, this PR's branch)

Built on Linux x86_64 with -G Ninja -DQUIC_TLS_LIB=quictls -DQUIC_BUILD_SHARED=OFF:

With -DQUIC_OPENSSL_SYMBOL_PREFIX=msqtest_:

# 0 unprefixed defined globals in the renamed archives:
nm --defined-only --extern-only build/openssl-prefixed/msqtest_/libssl.a \
  | awk '$2 ~ /^[TDRBWVC]$/ {print $3}' | grep -vc '^msqtest_'    # → 0
nm --defined-only --extern-only build/openssl-prefixed/msqtest_/libcrypto.a \
  | awk '$2 ~ /^[TDRBWVC]$/ {print $3}' | grep -vc '^msqtest_'    # → 0

# 0 unprefixed OpenSSL undefs in msquic_platform.a, 161 prefixed undefs:
nm -u build/obj/Release/libmsquic_platform.a | awk '$1=="U"{print $2}' \
  | grep -E '^(SSL_|EVP_|BN_|ERR_|X509_|OPENSSL_|RAND_|RSA_|EC_|BIO_|ASN1_|PEM_|CRYPTO_)' \
  | wc -l                                                         # → 0
nm -u build/obj/Release/libmsquic_platform.a | awk '$1=="U"{print $2}' \
  | grep -c '^msqtest_'                                           # → 161

Without the option (default build):

• 138 normal unprefixed OpenSSL undefs in libmsquic_platform.a.
• build/openssl-prefixed/ directory not created.
• Build time identical to baseline (no new compilation; the rename rules simply don't fire).

Production validation

The same prefix-rename technique has been deployed in Microsoft's meru codebase (a consumer of MsQuic). It is currently validated across this matrix:

• Release x86_64
• Release_ASAN, Debug_ASAN, Debug_UBSAN, Debug_TSAN x86_64
• arm_debug_crosscompile (aarch64)
• clang_tidy

See microsoft/meru-common#4011 for the consumer-side integration that ports the same helper script + CMake module pattern that this PR upstreams.

MsQuic CI coverage

The default (option-empty) path is unchanged, so existing CI should be unaffected. The option-set path is Linux-only and opt-in, so it does not need to enter the default matrix. Happy to add a single Linux CI leg that exercises -DQUIC_OPENSSL_SYMBOL_PREFIX=msquic_ if maintainers think that's worthwhile.

Documentation

New file: docs/OpenSSLSymbolPrefix.md documents motivation, usage, constraints, and a verification recipe.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.55%. Comparing base (23711d2) to head (1af0e6e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6029      +/-   ##
==========================================
+ Coverage   85.03%   85.55%   +0.51%     
==========================================
  Files          60       60              
  Lines       18792    18792              
==========================================
+ Hits        15980    16077      +97     
+ Misses       2812     2715      -97     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds an opt-in CMake cache variable (QUIC_OPENSSL_SYMBOL_PREFIX) to namespace-prefix all globally visible symbols in MsQuic’s bundled OpenSSL static archives on Linux, and rewrites MsQuic’s OpenSSL undefined references to match. This mitigates symbol collisions when a process contains another OpenSSL copy.

Changes:

• Introduces QUIC_OPENSSL_SYMBOL_PREFIX option with Linux-only / bundled-OpenSSL-only validation and wiring.
• Adds CMake + shell helpers to generate an nm-derived symbol map and apply objcopy --redefine-syms to produce prefixed archives.
• Applies a POST_BUILD rename step to libmsquic_platform.a so MsQuic’s OpenSSL references resolve against the prefixed archives; adds new documentation.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
CMakeLists.txt	Adds the cache option, validation, module include, and routes OpenSSL interface target to prefixed archives when enabled.
src/platform/CMakeLists.txt	Adds a POST_BUILD step to rewrite OpenSSL references inside libmsquic_platform.a when the option is enabled.
cmake/PrefixOpenSSLArchives.cmake	New CMake helper that generates a redefine-syms map and creates prefixed copies of libssl.a/libcrypto.a.
cmake/openssl-prefix-rename.sh	New helper script implementing gen-syms and apply modes via nm and objcopy.
docs/OpenSSLSymbolPrefix.md	New documentation describing motivation, usage, constraints, and verification steps.

[leikong]

Deep-mode code review (Claude Opus 4.7 extra-high + GPT-5.5 blind parallel Pass 1s, merged by Claude Opus 4.7 high reasoning). 2 major + 7 minor + 1 grouped nit. Highest-impact summary:

• major Configure-time zero-byte placeholders for the renamed .a files can shadow real outputs when source archives are restored from a CI cache with older mtimes — link silently consumes empty archives. cmake/PrefixOpenSSLArchives.cmake lines 156-167.
• major BUILD_SHARED_LIBS=ON + QUIC_OPENSSL_SYMBOL_PREFIX=<set> breaks install(EXPORT msquic) because OpenSSL's link interface gains an unexported OpenSSLQuicPrefixed (declared INTERFACE IMPORTED GLOBAL). Either reject the combination, drop IMPORTED and add OpenSSLQuicPrefixed to the export set, or skip the OpenSSL-targets export when prefix is set.
• minor (both passes) The nm filter [TDRBWVC] omits IFUNC types i/I; not exercised by the current bundled OpenSSL config but trivially-safe defensive coverage for future submodule bumps.
• minor (both passes) QUIC_OPENSSL_SYMBOL_PREFIX is unvalidated and is interpolated both as a symbol prefix and as a build-tree path component — needs a ^[A-Za-z_][A-Za-z0-9_]*$ guard.
• minor Several other items: nm stderr swallowed, dlsym-based OpenSSL plug-ins escape rename (docs), POST_BUILD lacks OBJECT_DEPENDS on the syms file, INTERFACE_LINK_LIBRARIES regex is fragile, add_dependencies on an INTERFACE IMPORTED target is a no-op, docs verification commands have grep -vc exit-code traps.
• nit (grouped) Thin-archive guard, LTO incompatibility callout, readlink -f portability, warn when prefix is set with QUIC_TLS_LIB=schannel, CMAKE_MODULE_PATH pollution in add_subdirectory, two documentation accuracy nits.

No blockers. Default-disabled path is byte-identical, agreed.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

[leikong]

Code review — deep mode (Opus-extra-high + GPT-5.5 blind parallel Pass 1s; Opus high-reasoning merge & audit)

Five major findings, mostly around the POST_BUILD step's fragility in CMake's dependency graph and missing validation. The opt-in design and Linux-only gate are sound; concerns are about edge cases that silently produce broken/un-prefixed binaries — exactly the failure modes this PR is meant to prevent.

Highlights (by severity)

• major — POST_BUILD rename is outside the dependency graph. The rename on libmsquic_platform.a won't re-fire when only QUIC_OPENSSL_SYMBOL_PREFIX, redefine.syms, or openssl-prefix-rename.sh changes — and won't re-fire after an interruption between ar and objcopy. Reconfiguring with a new prefix on an existing build tree produces a corrupted incremental build that silently leaks unprefixed OpenSSL symbols (the exact failure mode this PR exists to prevent). Restructure as a tracked add_custom_command(OUTPUT ...) that produces a separate libmsquic_platform.prefixed.a and route consumers at that artifact.
• major — Prefix charset/length validation missing. The value is interpolated straight into objcopy --redefine-syms lines and into the openssl-prefixed/<PREFIX>/ path. Whitespace, #, leading digit, /, .., shell metachars all silently corrupt the rename map or build tree. Reject prefixes not matching ^[A-Za-z_][A-Za-z0-9_]*$ with FATAL_ERROR.
• major — Validation block silently ignored with QUIC_TLS_LIB=schannel. The whole prefix block lives inside if(QUIC_TLS_LIB STREQUAL quictls OR ...openssl). Setting QUIC_OPENSSL_SYMBOL_PREFIX with schannel (or unset on Windows) is silently no-op'd — no FATAL_ERROR, no warning. Lift the prefix-set guard out so it errors loudly with an incompatible TLS lib.
• major — install(EXPORT msquic) broken when prefixing is enabled. With BUILD_SHARED_LIBS=ON, OpenSSL's INTERFACE_LINK_LIBRARIES now references OpenSSLQuicPrefixed (INTERFACE IMPORTED GLOBAL), which can't be exported. The generated share/msquic/msquic-targets.cmake will fail at install or reference build-tree-only paths.
• major — awk regex ^[TDRBWVC]$ misses I (IFUNC) and u (unique-global). OpenSSL 3.x on x86_64 Linux does ship IFUNCs (e.g. AES-NI/SHA-NI dispatch resolvers). Those defined externs won't be in the rename map → MsQuic's undefs to those names don't get renamed at POST_BUILD → unprefixed OpenSSL symbols leak through, again defeating the PR's purpose.

Minor findings

• gen-syms swallows nm stderr with 2>/dev/null and treats a 0-line syms file as success. Real errors (bad cross-compile nm, malformed archive) are hidden, and a silently-empty syms file produces a no-op rename — build looks green, symbols are not prefixed.
• apply mode does cp → objcopy in place on the destination with no trap. Ctrl-C between the two leaves a fresh, valid copy of the un-renamed input with a newer mtime than the source → next build skips the rename.
• file(WRITE "") placeholder pattern in PrefixOpenSSLArchives.cmake is unnecessary and masks a failure mode: if the custom_command silently fails to run, consumers see a 0-byte archive, which ar/ld accept as valid-but-empty — final link succeeds with NO OpenSSL implementation, deferring failure to runtime.
• cmake/PrefixOpenSSLArchives.cmake _out_dir parses OpenSSLQuic's INTERFACE_LINK_LIBRARIES via a hardcoded libssl\.a$ / libcrypto\.a$ regex. Fragile against any future genexpr wrapping of the Linux paths — the FATAL_ERROR doesn't include the genexpr content, making diagnosis hard.
• Inconsistent env-var forwarding: PrefixOpenSSLArchives.cmake conditionally forwards NM=/OBJCOPY= only when set; src/platform/CMakeLists.txt always forwards them, so CMAKE_NM="" (empty) silently re-defaults to host nm even on cross-builds.
• Documentation under-states the reconfigure caveat: changing QUIC_OPENSSL_SYMBOL_PREFIX in an existing build tree requires a clean / --fresh configure.

Nits

• src/platform/CMakeLists.txt comment references submodules/CMakeLists.txt for the rename wiring; the rename actually lives in cmake/PrefixOpenSSLArchives.cmake invoked from the top-level CMakeLists.txt.
• docs/OpenSSLSymbolPrefix.md verification recipe uses nm -gC (demangling) but the rename script doesn't — false-positive risk if downstream consumers contain any C++ TUs.
• Doc example uses mymsquic_ literally while the same doc tells users to pick a globally-unique prefix; copy-paste produces collisions across projects.
• awk 'NF==3' strict-equality filter is fragile vs. nm-wrappers that emit <member>: <addr> <type> <name> (4 fields).
• No command -v existence check on $NM/$OBJCOPY before invocation.
• The directory openssl-prefixed/<OLD_PREFIX>/ is orphaned in the build tree when the prefix changes — disk leak.

(Multi-model deep review; merge by Claude Opus 4.7 high-reasoning. Inline comments below mark the exact lines.)

[leikong]

Superseded by #6031 — same head SHA 1af0e6e4f, identical content. Migrating off personal fork (leikong/meru-staging-msquic) so the head branch lives on microsoft/msquic directly. Re-direct review activity to #6031.