Add support for Intel CET and -fcf-protection

[omajid]

I am trying to harden .NET Core 3.1 on Linux. One recent technique a number of other packages are using for security hardening is to use Intel's CET (Control-Flow Enforcement Technology):

https://www.linuxplumbersconf.org/event/2/contributions/147/attachments/72/83/CET-LPC-2018.pdf

https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf

Basics

Intel CET uses two techniques to improve the security of applications:

1. Indirect Branch Tracking (IBT): All targets of (x86_64 assembly-level) indirect branches are annotated as being valid jump targets. On every jump, the runtime/processor verifies that the target is an expected jump location. This makes it possible to catch issues where an exploit tries to jump into an unexpected piece of code to do an action that was not anticipated by the programmer.

2. Shadow Stack (SHSTK): A call stack is maintained on the side that's mostly invisible to the process and that's used to verify that the normal stack has not been exploited. A buffer overflow, for example, can be used to overwrite the normal stack. But when that happens the contents of the shadow stack and the normal stack will no longer match and the attack can be stopped.

Together, these help stop attacks where an exploit takes control of the application stack and then abuses that to perform malicious actions.

Implementation Issues

Fully supporting CET varies by language (C, assembler, etc) and the component (runtime, JIT, unwinder).

For a normal C/C++ application, most of the work is done by the C/C++ compiler, and all we really need to do is to add -fcf-protection (which is supported by both GCC and Clang). I can inject that by overriding CFLAGS and CXXFLAGS when building runtime (via source-build). And that seems to work.

For assembly code, some special actions are needed.

1. All locations that are targets of indirect branches must add additional endbr32/endbr64 instructions. These instructions are a no-op on older processors that do not understand endbr32/endbr64 instructions. These instructions don't break compatiblity, but may introduce delay in parsing and skipping over these no-op instructions by the processor.

2. The assembly code needs to be annotated with a property to indicate that IBT and SHSTK are enabled. This is a GNU_PROPERTY_NOTE to indicate that CET (more specifically, ibt and shstk) is enabled. The note can be a lie, and it's possible to use this note to convince the run-time linker that everything is okay even if endbr instructions are missing.

In addition, some compoents need extra fixes.

The code emitted by the x86_64 jit needs to emit endbr32/endbr64 for valid targets of all indirect jumps.

The unwinder (libunwind) needs to be aware of the shadow stack. It should unwind (or other handle) the shadow stack in parallel with the normal stack. The "system" unwinder on Linux (gcc_s) can already handle the shadow stack, but libunwind doesn't know about it (https://bugs.llvm.org/show_bug.cgi?id=45946 ?).

Verifying

There are two parts of verifying:

1. Making sure all binaries are annotated correctly

2. Making sure the application runs correctly on systems where CET is enabled. A missing endbr instruction or incorrect use of the stack will cause the application to blow up and fail. From what I have heard from @fweimer, CET needs hardware as well as kernel+userspace support, which is not yet fully (or commonly) implemented across the stack on Linux.

For number 1, we can use annocheck to verify that ELF objects (shared libraries (*.so), and executables) are all compiled with CET support:

$ annocheck /usr/lib64/dotnet/ | grep FAIL
Hardened: libdbgshim.so: FAIL: Control flow protection has been enabled for only some parts of the binary.  Other parts (probably assembler sources) are missing the protection, and without it global control flow protection cannot be enabled.
Hardened: libclrjit.so: FAIL: Control flow protection has been enabled for only some parts of the binary.  Other parts (probably assembler sources) are missing the protection, and without it global control flow protection cannot be enabled.
Hardened: libmscordbi.so: FAIL: Control flow protection has been enabled for only some parts of the binary.  Other parts (probably assembler sources) are missing the protection, and without it global control flow protection cannot be enabled.

Or we can use readelf directly on ELF objects looking for IBT/SHSTK:

readelf --notes "$object" | grep -E 'IBT|SHSTK'

Looking at a source-build build directory using find-missing-ibt, a number of files are missing IBT flags:

$ ~/find-missing-ibt
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/dlls/mscordbi/CMakeFiles/mscordbi.dir/__/mscordac/palredefines.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/dlls/mscordac/CMakeFiles/mscordaccore.dir/libredefines.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/libunwind/src/CMakeFiles/libunwind.dir/x86_64/getcontext.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/libunwind/src/CMakeFiles/libunwind.dir/x86_64/setcontext.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/CMakeFiles/coreclrpal.dir/arch/amd64/context2.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/CMakeFiles/coreclrpal.dir/arch/amd64/exceptionhelper.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/CMakeFiles/coreclrpal.dir/arch/amd64/debugbreak.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/pal/src/CMakeFiles/coreclrpal.dir/arch/amd64/callsignalhandlerwrapper.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/debug/ee/wks/CMakeFiles/cordbee_wks.dir/__/amd64/dbghelpers.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/debug/di/CMakeFiles/cordbdi.dir/amd64/floatconversion.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/debug/createdump/CMakeFiles/createdump.dir/__/__/dlls/mscordac/palredefines.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/asmhelpers.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/jithelpers_slow.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/calldescrworkeramd64.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/virtualcallstubamd64.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/theprestubamd64.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/crthelpers.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/pinvokestubs.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/jithelpers_singleappdomain.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/externalmethodfixupthunk.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/unixasmhelpers.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/getstate.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/jithelpers_fastwritebarriers.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/jithelpers_fast.S.o is missing IBT
error: file src/coreclr.018cfd06dceb19b6eb1e9217a500fb1071946fcd/bin/obj/Linux.x64.Release/src/vm/wks/CMakeFiles/cee_wks.dir/__/amd64/umthunkstub.S.o is missing IBT

All these are generated form assembly files, so it sounds like we need to update the assembly files and the assembly file generators to add the CET instructions.

Configuration

This is a self-built 3.1 SDK using source-build:

 /usr/lib64/dotnet/dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.106
 Commit:    0f94483edc

Runtime Environment:
 OS Name:     rhel
 OS Version:  8
 OS Platform: Linux
 RID:         rhel.8-x64
 Base Path:   /usr/lib64/dotnet/sdk/3.1.106/

Host (useful for support):
  Version: 3.1.6
  Commit:  3acd9b0cd1

.NET Core SDKs installed:
  3.1.106 [/usr/lib64/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.6 [/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.6 [/usr/lib64/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

I built it using source-build with this configuration:

export 'CFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables  -fcf-protection'
export 'CXXFLAGS=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches   -m64 -mtune=generic -fasynchronous-unwind-tables  -fcf-protection'
export 'LDFLAGS=-Wl,-z,relro -Wl,-z,now '
export ASMFLAGS=-fcf-protection

Regression?

No

Open Questions

• Is this worth doing? The runtime basically executes bytecode. Do we really need to worry about code injection?
• What is the performance impact (if any) of adding endbr32/endbr64 instructions?

[omajid]

cc @tmds

[jkotas]

cc @janvorli

[janvorli]

@omajid thank you for the initial investigations, we definitely want to enable Intel CET support. Do you want to try to modify one of the assembly files by adding the endbr64 instructions to see if it is sufficient to fix the problem or would you prefer me to do that? I would use the getstate.S as the one to experiment with as it contains only three simple asm functions.

[omajid]

@omajid thank you for the initial investigations, we definitely want to enable Intel CET support. Do you want to try to modify one of the assembly files by adding the endbr64 instructions to see if it is sufficient to fix the problem or would you prefer me to do that?

I am a bit swamped for the next two weeks or so, but I am happy to give this a shot right after.

I would use the getstate.S as the one to experiment with as it contains only three simple asm functions.

Thanks for the pointers!

[fweimer]

Keep in mind that you need to tweak the JIT first to generate the necessary ENDBR instructions. Until then, the lack of CET markup is what keeps .NET Core running on CET-enabled systems (because CET is automatically disabled for such processes).

[omajid]

I have also realized that full CET support means teaching our unwinder (libunwind) about the shadow stack.

[omajid]

For the assembly files, this seems to work:

diff --git a/src/coreclr/src/vm/amd64/getstate.S b/src/coreclr/src/vm/amd64/getstate.S
index 04d9ab707d5..31dcf16a253 100644
--- a/src/coreclr/src/vm/amd64/getstate.S
+++ b/src/coreclr/src/vm/amd64/getstate.S
@@ -5,8 +5,23 @@
 #include "unixasmmacros.inc"
 #include "asmconstants.h"
 
+.section .note.gnu.property, "a", @note
+        .balign 8
+        .long 4                 /* n_namsz: Length of n_name */
+        .long 16                /* n_descz: note descriptor size */
+        .long 5                 /* n_type: NT_GNU_PROPERTY_TYPE_0 */
+        .asciz "GNU"            /* n_name */
+        .balign 8
+        .long 3221225474        /* pr_type: GNU_PROPERTY_X86_FEATURE_1_AND */
+        .long 4                 /* pr_datasz: Length of pr_data = 4 */
+        .long 0x3               /* pr_data: property descriptor */
+        .balign 8
+
+.section text
+
 LEAF_ENTRY GetCurrentSP, _TEXT
 
+        endbr64
         mov rax, rsp
         add rax, 8
         ret
@@ -16,6 +31,7 @@ LEAF_END GetCurrentSP, _TEXT
 
 LEAF_ENTRY GetCurrentIP, _TEXT
 
+        endbr64
         mov rax, [rsp]
         ret
 
@@ -25,6 +41,7 @@ LEAF_END GetCurrentIP, _TEXT
 // EXTERN_C void LazyMachStateCaptureState(struct LazyMachState *pState)
 LEAF_ENTRY LazyMachStateCaptureState, _TEXT
 
+        endbr64
         mov     rdx, [rsp]      // get the return address
 
         mov     [rdi + OFFSETOF__MachState__m_Capture + 0*8], r12

This seems to be enough to our tools to confirm that the assembly is marked as supporting CET (IBT/SHSTK).

$ readelf --notes ./artifacts/obj/coreclr/Linux.x64.Debug/src/vm/wks/CMakeFiles/cee_wks_core.dir/__/amd64/getstate.S.o

Displaying notes found in: .note.gnu.property
  Owner                Data size        Description
  GNU                  0x00000010       NT_GNU_PROPERTY_TYPE_0
      Properties: x86 feature: IBT, SHSTK

I am not sure if all these targets need endbr64... It's only needed for targets of indirect branches.

Also, verifing that CET is completely supported would require testing on hardware/software stack that checks for CET (my local machine doesn't).