Ian/sleigh support rebased (#607)

* cmake: Bring in SLEIGH as a dependency

* sleigh: Boilerplate for adding a new arch

* sleigh: Begin passing instruction sequences into SLEIGH

* cmake: Rename target to be X86 specific

* sleigh: Copy over more X86 runtime code and get things running

* sleigh: Begin populating operands in the returned instruction

* sleigh: Set instruction category for a few opcodes

* sleigh: Initial attempt at generating LLVM IR for P-Code

* sleigh: Implement enough opcodes to run the `sleigh-lift` example

* sleigh: Get things building with an up-to-date Remill tree

* sleigh: Use the new SLA helpers

* sleigh: Clear operands between invalid lifts

* added thumb2

* stub out thumb

* need to refactor to relift the instruction

* factored lifter out of handler:

* allow lifters to decide wether to use sleigh

* relift bytes

* fixed both lifter contexts

* hey a copy lifted :)

* need to refactor to use parameterptrs to unify interface to memory

* refactored to use parameter abstraction

* lift at correct location

* add control flow to pc and next

* add categories mostly

* fix direct semantics add cbranch

* track cbranch metavar also lift returns

* cmake: Fix SleighArch.h path

* sleigh: Support more binary ops

* sleigh: Implement ZEXT

* sleigh: Support SEXT op

* sleigh: Implement 2COMP and INT_NEGATE

* mutex around parsing sleigh specs

* fixed typos in custom image

* added handling for exceptions, need to fix issue with xml parsing

* sleigh: Support boolean binops

* sleigh: Support float binary ops

* sleigh: Support a few more float pcode ops that require LLVM intrinsics

* sleigh: Add entries for ops that require overflow intrinsics

* sleigh: Implement more float unary ops

* sleigh: Reduce duplication in ops that require float intrinsics

* expose mutex

* fix fallthough and format of control flow resolution

* sleigh: Support CARRY, SCARRY and SBORROW

* Remove duplicate THUMB2 arch in enum

* Remove duplicate THUMB2 entry in archnames array

* Implement STORE op

* Fix incorrect index

* Support PIECE and SUBPIECE ops

* Support remaining set of PCode ops in the base set. Now to do the
pseudo ops.

* Implement additional ops

* Partially implement a few pseudo ops + a few missing from the docs

* Fix the variadic op condition

* Include <mutex> to build on Linux

* stop lying about insn categories/functions

* set function insns with mnemonic

* fix fallthrough for ind calls

* More virtualization, fewer pimples

* switch back to x86 normal

* Fix off-by-eight issue with ADR in AArch32

* Get rid of mutex in sleigh, make a globally-available remill locker for handling sleigh issues

* CHange allow 32 bit shift to true, not sure if valid but oh well

* Use the node size when creating constants

* fix memory

* store memory state back

* fix calling intrinsic table

* fix comp sizes

* load pspec data

* add pspec names

* add differential test

* fix tests start pretty printer

* add runner

* added really slow memory

* add comparison of memory state

* do reset

* fix uninitialized module

* fixed lift crash

* add amd64 runtime

* unfix fs, gs, and pc

* added whitelist file

* fix memory to update state with uninitialized reads

* better pc handling

* add main.py

* x86 compiles

* fixup patch generation

* add replacements for mem locs and constant varnodes, now need to handle special branches

* added replacement for direct branches

* add context clears

* add handler for claim_eq

* comment out clears for now

* added ambiguity check

* handle duplicate names in same constructor

* mantain sleigh invariant of 1 or 0 in flag

* added uint8t memory intrinsics

* extend shift value when needed

* handle cmovs

* update patch generation

* removed deprecated load

* fix for returned type

* builds

* add logging

* add comment on why not compute GEPAccessors in Arch.cpp

* enable opaque pointers

* fix memory state update

* add mem_16 impls

* not preserve 1 bit width

* fix CR changes

* fix L and R for shift operands bigger than target

* revert use after move

* more CR fixes

* remove lookahead... lets try this again

* rework to create internal function that is inlined to allow for early return control flow

* terminate conditionally

* avoid large iteration counts for .REP insns

* fix patch for rep insns

* handle inst_start in patches

* handle inserting insn_size constructor outside of macros

* hint size of insn_size

* refactor bool functions to make more sense

* just use ;

* remove delete of operands when fail to lift

* remove copy and paste

* remove unused deps

* add patch file to sleigh list

* prep git user

* refer to checkout

* maybe shell type?

* fix layout

* fix format again

* fix

* update dockerfile

* fix gitwatcher to point to current project:

* Revert "fix gitwatcher to point to current project:"

This reverts commit 711da11.

* fix ninja builds

* disable sleigh tests

* remove duplicates

* refactor

* fix non reset context in sleigh lifter

* remove debugging prints

* wrote CI runner

* add handwritten test infra

* refactor to allow sharing JIT test running between differential tester and hand written tests

* allow for internal ownership of semantics module

* handwritten test framework

* fix hand written tests

* pc rel testruns

* add pc rel regression

* fix script for pc rel

* fix names for priors to ignore whitespace

* fix action ops

* add xor to repl ops

* add xors to patch

* add arm patch to build

* fix test for semantics

* allow cross platform tests to run

* add running diff tester

* 2 underscores?

* run tests on macos too

* fix command

* fix whitelist path

* remove unused test data

* install python deps for differential tests

* update script to git patches

* update patches to git format

* attempt to fixup patches

* update ref for sleigh

* install test depends docker

* point docker to correct dir

* install pydeps in macos too

* initialize address field

* exports, also tag master on sleigh

* add comments and whitelist undefined value OF in shifts

* add of accessor to whitelist impl

* change variables to non alloca and remove dead code

* remove dead code:

* replace dump with print

* code quality

* make non null into ref

* type alias

* remove allocas from unit tests

* Address copypasta comment.

* Address PR comment.

* Address PR comment.

* fix build

* unify diff modules into single structure

* remove extra newline

* lock sleigh to specific commit

* remove new lines

* remove useless fenv headers

* caps

* copyright notices

* update more copyright

* fix reviews

* early returns out of accessors

* early return

* auto

* return getarchbyname

* insert register rather than modify reference

* just return

* only log in the assembly logger

* prefer functional style

* remove commented code

* move defaults to header

* that's not how arch switches work in pcode

* informative names

* simplify control flow in fill

* early return

* fix early exit condition

* refactor register default into function

* make one liner

* early return

* unary instead of unop

* construct pair with {}

* move cbranch into binops instead of integer binops

* refactor float ops into getter

* factor out float type

* early return

* separate out callother handling

* do pointer extensions cleaner

* braces

* add private headers to lifters

* refactor redirecting control flow out of instruction

* use constant check function

* remove has_value

* structure for preconditions

* return success when applying eq claim

* expose arch base and move shared functionality into x86 base

* dedup x86 code

* dedup aarch32

* remove needless assignment, also remove else after an if that returns

Co-authored-by: Alex Cameron <[email protected]>
Co-authored-by: Artem Dinaburg <[email protected]>
Co-authored-by: Peter Goodman <[email protected]>

Author: 4 people

.github/actions/prepare_git_user/action.yml

@@ -0,0 +1,10 @@
+name: 'Setup Git User for Applying Patches'
+description: 'Setup bot user so git doesnt complain when patching'
+  # See this thread for more details https://github.community/t/github-actions-bot-email-address/17204/5
+runs:
+  using: "composite"
+  steps:
+    - shell: bash
+      run: |
+        git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
+        git config --global user.name "github-actions[bot]"

.github/workflows/ci.yml

@@ -36,6 +36,7 @@ jobs:
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0
+      - uses: ./.github/actions/prepare_git_user
       - name: Build with build script
         shell: bash
         run: |
@@ -48,6 +49,11 @@ jobs:
           export VCPKG_ROOT=$(pwd)/../lifting-bits-downloads/vcpkg_${{ matrix.image.name }}-${{ matrix.image.tag }}_llvm-${{ matrix.llvm }}_amd64
           export INSTALL_DIR=$(pwd)/remill-preset-install
           ./scripts/build-preset.sh release
+
+      - name: Install Python Test Deps
+        shell: bash
+        run: |
+          pip3 install --user ./scripts/diff_tester_export_insns
       - name: Run tests
         shell: bash
         working-directory: remill-build
@@ -102,6 +108,7 @@ jobs:
       - uses: actions/checkout@v2
         with:
           fetch-depth: 0
+      - uses: ./.github/actions/prepare_git_user
       - name: Build with build script
         shell: bash
         run: |
@@ -114,6 +121,10 @@ jobs:
           export VCPKG_ROOT=$(pwd)/../lifting-bits-downloads/vcpkg_${{ matrix.os}}_llvm-${{ matrix.llvm }}_xcode-13.0_amd64
           export INSTALL_DIR=$(pwd)/remill-preset-install
           ./scripts/build-preset.sh release
+      - name: Install Python Test Deps
+        shell: bash
+        run: |
+          pip3 install --user ./scripts/diff_tester_export_insns
       - name: Run tests
         shell: bash
         working-directory: remill-build

.gitignore

@@ -26,7 +26,6 @@ cmake-build-debug
 cmake-build-release
 compile_commands.json
 
-bin/*
 
 third_party/*
 build/*

CMakeLists.txt

@@ -4,7 +4,7 @@
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#     http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
@@ -20,7 +20,7 @@ include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/ccache.cmake")
 
 project(remill C CXX ASM)
 include(GNUInstallDirs)
-
+include(FetchContent)
 include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/settings.cmake")
 include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/utils.cmake")
 include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/options.cmake")
@@ -72,6 +72,7 @@ include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/BCCompiler.cmake")
 # Go find only the static libraries of LLVM, and link against those.
 foreach(LLVM_LIB IN LISTS LLVM_AVAILABLE_LIBS)
   get_target_property(LLVM_LIB_TYPE ${LLVM_LIB} TYPE)
+
   if(LLVM_LIB_TYPE STREQUAL "STATIC_LIBRARY")
     list(APPEND LLVM_LIBRARIES "${LLVM_LIB}")
   endif()
@@ -88,7 +89,7 @@ target_link_libraries(thirdparty_llvm INTERFACE
 # Microsoft Z3 with LLVM. Not exactly used in remill, but LLVM doesn't link
 # against it correctly
 # NOTE: If changing this, also replicate in remillConfig file
-if (LLVM_WITH_Z3)
+if(LLVM_WITH_Z3)
   find_package(Z3 CONFIG REQUIRED 4.7.1)
   get_target_property(LLVMSupport_LIBS LLVMSupport INTERFACE_LINK_LIBRARIES)
   list(REMOVE_ITEM LLVMSupport_LIBS Z3)
@@ -120,15 +121,32 @@ target_link_libraries(thirdparty_gflags INTERFACE
   gflags
 )
 
+set(sleigh_ENABLE_TESTS OFF)
+set(sleigh_ADDITIONAL_PATCHES "${CMAKE_CURRENT_SOURCE_DIR}/patches/sleigh/x86-ia.patch;${CMAKE_CURRENT_SOURCE_DIR}/patches/sleigh/arm-thumb.patch")
+
+# GHIDRA SLEIGH
+FetchContent_Declare(sleigh
+  GIT_REPOSITORY https://github.com/lifting-bits/sleigh.git
+  GIT_TAG 5ee2f2c16250a6529108e6a6fff89e0f147502d2
+)
+
+FetchContent_MakeAvailable(sleigh)
+add_library(thirdparty_sleigh INTERFACE)
+target_link_libraries(thirdparty_sleigh INTERFACE
+  sleigh::sla
+  sleigh::decomp
+  sleigh::support
+)
+
 # Windows SDK
 add_library(thirdparty_win32 INTERFACE)
+
 if(DEFINED WIN32)
   target_link_libraries(thirdparty_win32 INTERFACE
-     "Kernel32.lib"
+    "Kernel32.lib"
   )
 endif()
 
-
 # For Linux builds, group LLVM libraries into a single group
 # that avoids frustrating library ordering issues.
 if(UNIX AND NOT APPLE)
@@ -150,7 +168,6 @@ option(REMILL_BUILD_SPARC32_RUNTIME "Build the Runtime for SPARC32. Turn this of
 #
 
 # add everything as public.
-
 add_library(remill_settings INTERFACE)
 
 target_include_directories(remill_settings INTERFACE
@@ -235,7 +252,7 @@ target_compile_definitions(remill_settings INTERFACE
   "REMILL_BUILD_SEMANTICS_DIR_SPARC64=\"${REMILL_BUILD_SEMANTICS_DIR_SPARC64}\""
 )
 
-set(THIRDPARTY_LIBRARY_LIST thirdparty_llvm thirdparty_xed thirdparty_glog thirdparty_gflags)
+set(THIRDPARTY_LIBRARY_LIST thirdparty_llvm thirdparty_xed thirdparty_glog thirdparty_gflags thirdparty_sleigh)
 target_link_libraries(remill_settings INTERFACE
   ${THIRDPARTY_LIBRARY_LIST}
 )
@@ -259,7 +276,6 @@ target_link_libraries(remill INTERFACE
 #
 # Also install clang, libllvm and llvm-link
 #
-
 set(INSTALLED_CLANG_NAME "remill-clang-${REMILL_LLVM_VERSION}${CMAKE_EXECUTABLE_SUFFIX}")
 set(INSTALLED_LLVMLINK_NAME "remill-llvm-link-${REMILL_LLVM_VERSION}${CMAKE_EXECUTABLE_SUFFIX}")
 
@@ -269,15 +285,18 @@ InstallExternalTarget("ext_llvmlink" "${LLVMLINK_PATH}" "BIN" "${INSTALLED_LLVML
 
 GetTargetTree(THIRDPARTY_LIBRARIES ${THIRDPARTY_LIBRARY_LIST})
 GetPublicIncludeFolders(THIRDPARTY_INCLUDE_DIRECTORIES ${THIRDPARTY_LIBRARIES})
+
 foreach(THIRDPARTY_LIB IN LISTS THIRDPARTY_LIBRARIES)
   string(SUBSTRING "${THIRDPARTY_LIB}" 0 1 THIRDPARTY_LIB_PREFIX)
+
   if(TARGET ${THIRDPARTY_LIB})
     get_target_property(THIRDPARTY_LIB_TYPE ${THIRDPARTY_LIB} TYPE)
+
     if(THIRDPARTY_LIB_TYPE STREQUAL "STATIC_LIBRARY" OR THIRDPARTY_LIB_TYPE STREQUAL "SHARED_LIBRARY")
       list(APPEND THIRDPARTY_LIBRARY_FILES "$${}<TARGET_FILE:${THIRDPARTY_LIB}>")
     endif()
   elseif("${THIRDPARTY_LIB_PREFIX}" STREQUAL "$${}")
-    # E.g. $<LINK_ONLY:...>
+  # E.g. $<LINK_ONLY:...>
   else()
     list(APPEND THIRDPARTY_LIBRARY_FILES "${THIRDPARTY_LIB}")
   endif()
@@ -288,31 +307,33 @@ list(REMOVE_DUPLICATES THIRDPARTY_LIBRARY_FILES)
 #
 # additional targets
 #
-
 add_custom_target(semantics)
 
+# shared JIT tools
+add_subdirectory(test_runner_lib)
+
 # tools
 add_subdirectory(bin)
 
 if(REMILL_ENABLE_INSTALL_TARGET)
   install(TARGETS remill EXPORT remillTargets)
-  
+
   install(TARGETS remill_settings ${THIRDPARTY_LIBRARY_LIST}
     EXPORT remillTargets
   )
-  
+
   # First do the basic substitutions.
   configure_file(
     "${CMAKE_CURRENT_SOURCE_DIR}/cmake/remillConfig.cmake.in"
     "${CMAKE_CURRENT_BINARY_DIR}/remillConfig.cmake"
     @ONLY
   )
-  
+
   install(FILES "${CMAKE_CURRENT_BINARY_DIR}/remillConfig.cmake"
-                "${CMAKE_CURRENT_LIST_DIR}/cmake/vcpkg_helper.cmake"
+    "${CMAKE_CURRENT_LIST_DIR}/cmake/vcpkg_helper.cmake"
     DESTINATION "${REMILL_INSTALL_LIB_DIR}/cmake/remill"
   )
-  
+
   install(DIRECTORY "${REMILL_INCLUDE_DIR}/remill/"
     DESTINATION "${REMILL_INSTALL_INCLUDE_DIR}/remill"
   )
@@ -322,14 +343,19 @@ if(REMILL_ENABLE_INSTALL_TARGET)
 endif()
 
 # tests
-if (REMILL_ENABLE_TESTING)
+if(REMILL_ENABLE_TESTING)
   # Tests require enabling exports on binaries
   # https://cmake.org/cmake/help/latest/variable/CMAKE_ENABLE_EXPORTS.html#variable:CMAKE_ENABLE_EXPORTS
   set(CMAKE_ENABLE_EXPORTS ON)
 
   find_package(Threads REQUIRED)
   add_custom_target(test_dependencies)
 
+  if(REMILL_ENABLE_TESTING_SLEIGH_THUMB)
+    message(STATUS "thumb tests enabled")
+    add_subdirectory(tests/Thumb)
+  endif()
+
   if(REMILL_ENABLE_TESTING_X86)
     message(STATUS "X86 tests enabled")
     add_subdirectory(tests/X86)

Dockerfile

@@ -30,10 +30,15 @@ ARG LLVM_VERSION
 
 WORKDIR /remill
 COPY ./ ./
+
+RUN git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" && git config --global user.name "github-actions[bot]"
+
 RUN ./scripts/build.sh \
-  --llvm-version ${LLVM_VERSION} \
-  --prefix /opt/trailofbits \
-  --extra-cmake-args "-DCMAKE_BUILD_TYPE=Release"
+    --llvm-version ${LLVM_VERSION} \
+    --prefix /opt/trailofbits \
+    --extra-cmake-args "-DCMAKE_BUILD_TYPE=Release"
+
+RUN pip3 install ./scripts/diff_tester_export_insns
 
 RUN cd remill-build && \
     cmake --build . --target test_dependencies -- -j $(nproc) && \

bin/CMakeLists.txt

@@ -1,10 +1,10 @@
-# Copyright (c) 2017 Trail of Bits, Inc.
+# Copyright (c) 2022 Trail of Bits, Inc.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#     http://www.apache.org/licenses/LICENSE-2.0
+# http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
@@ -14,3 +14,6 @@
 
 add_subdirectory(lift)
 
+if(REMILL_ENABLE_DIFFERENTIAL_TESTING)
+    add_subdirectory(differential_tester_x86)
+endif()

bin/differential_tester_x86/CMakeLists.txt

@@ -0,0 +1,35 @@
+# Copyright (c) 2022 Trail of Bits, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+find_package(Python COMPONENTS Interpreter)
+add_executable(
+  lift-and-compare
+  LiftAndCompare.cpp
+  Whitelist.cpp
+  Whitelist.h
+)
+
+target_link_libraries(
+  lift-and-compare
+  PRIVATE
+  remill
+  thirdparty_glog
+  test-runner
+)
+
+set_property(TARGET lift-and-compare PROPERTY ENABLE_EXPORTS ON)
+set_property(TARGET lift-and-compare PROPERTY POSITION_INDEPENDENT_CODE ON)
+enable_testing()
+
+add_test(NAME "small_diff_test" COMMAND "${Python_EXECUTABLE}" ${REMILL_SOURCE_DIR}/scripts/diff_tester_export_insns/diff_tester_export_insns/ci_runner.py --required_success_rate 1.0 --difftester_bin ${CMAKE_BINARY_DIR}/bin/differential_tester_x86/lift-and-compare --workdir ${CMAKE_BINARY_DIR} ${REMILL_SOURCE_DIR}/bin/differential_tester_x86/data/small_test/ --whitelist_file ${REMILL_SOURCE_DIR}/bin/differential_tester_x86/whitelist.json)