Security-harden macOS and Linux binaries

[martincostello]

Why

Harden native C/C++ DLLs for macOS and Linux similar to work for Windows in #4768.

See #4768 (review).

Fixes #4769.

What

Apply compiler flags/options for macOS and Linux builds to harden the security of the native libraries.

Tests

Existing tests should pass.

Checklist

• CHANGELOG.md is updated.
• Documentation is updated.
• New features are covered by tests.

[martincostello]

BinSkim output from 0c261f5:

> .\binskim.exe analyze *.so --recurse true --kind "Fail"
Analyzing...
THREADS: 20
C:\Users\marti\Downloads\zero-code\bin-ubuntu-22.04\linux-x64\OpenTelemetry.AutoInstrumentation.Native.so: error BA3031: 'OpenTelemetry.AutoInstrumentation.Native.so' was compiled using Clang but without the SafeStack instrumentation pass, which should be used to mitigate the risk of stack-based buffer overflows. To enable SafeStack, pass '-fsanitize=safe-stack' flag to both compile and link command lines.
C:\Users\marti\Downloads\zero-code\bin-ubuntu-22.04-arm\linux-arm64\OpenTelemetry.AutoInstrumentation.Native.so: error BA3031: 'OpenTelemetry.AutoInstrumentation.Native.so' was compiled using Clang but without the SafeStack instrumentation pass, which should be used to mitigate the risk of stack-based buffer overflows. To enable SafeStack, pass '-fsanitize=safe-stack' flag to both compile and link command lines.
C:\Users\marti\Downloads\zero-code\bin-ubuntu1604-native\linux-x64\OpenTelemetry.AutoInstrumentation.Native.so: error BA3031: 'OpenTelemetry.AutoInstrumentation.Native.so' was compiled using Clang but without the SafeStack instrumentation pass, which should be used to mitigate the risk of stack-based buffer overflows. To enable SafeStack, pass '-fsanitize=safe-stack' flag to both compile and link command lines.

Done. 3 files scanned.
Analysis completed successfully.

One or more rules was disabled for an analysis target, as it was determined not to be applicable to it (this is a common condition). Include 'NotApplicable' on an explicit '--kind' command-line argument (e.g., '--kind "Fail;NotApplicable"') for more information.

Need to work out why -fsanitize=safe-stack doesn't appear to be having an effect.

BA3011 warnings have gone away though.

[martincostello]

Need to work out why -fsanitize=safe-stack doesn't appear to be having an effect.

I can't get this one to work for unknown reasons, even locally in Ubuntu 24.04 under WSL. I'm no expert in this area so I don't think it's worth sinking any further time into.

BinSkim has picked up at least one improvement here, but I don't think it checks all the flags that Copilot suggested were enabled.

Thoughts on next steps?

[Kielek]

@eftiquar, could you please advice here?

[eftiquar]

@eftiquar, could you please advice here?
Looking at the error, the issue is that SafeStack (-fsanitize=safe-stack) is being conditionally added based on compiler support, but it appears the check is passing yet the flag isn't being applied correctly, or there's a conflict with other options.
The error BA3031 occurs because the SafeStack flag must be used where following conditions are met

1. Both the compiler and linker should be passed that flag. This is correctly implemented.
2. it cannot coexist with static linking of the C++ runtime. >> we fail this condition. - SafeStack is incompatible with -static-libgcc and -static-libstdc++.

[martincostello]

What would you like me to do about SafeStack?

1. Leave the flags there, but with a comment stating it won't work in practice due to the constraint above
2. Remove the flags
3. Something else?

C/C++ is outside my area of expertise, so I won't be going down the path of trying to do anything about the static linking.

[eftiquar]

What would you like me to do about SafeStack?

1. Leave the flags there, but with a comment stating it won't work in practice due to the constraint above
2. Remove the flags
3. Something else?

C/C++ is outside my area of expertise, so I won't be going down the path of trying to do anything about the static linking.

Messing with linking will need thorough testing, I suggest we omit the flags, but leave a comment saying SafeStack is incompatible with static linking and this will be undertaken in future when we upgrade to dynamic linking.

[Copilot]

Pull request overview

This PR applies security hardening compiler flags to macOS and Linux native library builds, following similar work done for Windows in PR #4768. The changes enhance the security posture of the native profiler libraries through various compiler and linker protections.

Changes:

• Added stack protection flags for buffer overflow detection
• Enabled Control Flow Integrity protections (CFI, BTI for ARM64)
• Applied Linux-specific hardening (_FORTIFY_SOURCE, RELRO, non-executable stack)
• Applied macOS-specific hardening (stack checking)
• Enabled position-independent code for static and shared libraries

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/OpenTelemetry.AutoInstrumentation.Native/CMakeLists.txt	Added compiler flags for stack protection, control flow integrity, and platform-specific security hardening; added linker options for ASLR and RELRO on Linux; enabled POSITION_INDEPENDENT_CODE properties
CHANGELOG.md	Added entry documenting the security hardening for macOS and Linux native libraries

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[eftiquar]

What would you like me to do about SafeStack?

1. Leave the flags there, but with a comment stating it won't work in practice due to the constraint above
2. Remove the flags
3. Something else?

C/C++ is outside my area of expertise, so I won't be going down the path of trying to do anything about the static linking.

Messing with linking will need thorough testing, I suggest we omit the flags, but leave a comment saying SafeStack is incompatible with static linking and this will be undertaken in future when we upgrade to dynamic linking.

We can conclude this PR with above mentioned note, this is about as much has we can fortify the binaries.

[martincostello]

We can conclude this PR with above mentioned note

It's already been added - see lines 118-119 in the CMakeLists.txt file in the diff.

[Kielek]

Approving per @eftiquar comment. Lack of one switch is well documented in CMakeLists.