Skip to content

Commit

Permalink
Ian/sleigh support rebased (#607)
Browse files Browse the repository at this point in the history
* cmake: Bring in SLEIGH as a dependency

* sleigh: Boilerplate for adding a new arch

* sleigh: Begin passing instruction sequences into SLEIGH

* cmake: Rename target to be X86 specific

* sleigh: Copy over more X86 runtime code and get things running

* sleigh: Begin populating operands in the returned instruction

* sleigh: Set instruction category for a few opcodes

* sleigh: Initial attempt at generating LLVM IR for P-Code

* sleigh: Implement enough opcodes to run the `sleigh-lift` example

* sleigh: Get things building with an up-to-date Remill tree

* sleigh: Use the new SLA helpers

* sleigh: Clear operands between invalid lifts

* added thumb2

* stub out thumb

* need to refactor to relift the instruction

* factored lifter out of handler:

* allow lifters to decide wether to use sleigh

* relift bytes

* fixed both lifter contexts

* hey a copy lifted :)

* need to refactor to use parameterptrs to unify interface to memory

* refactored to use parameter abstraction

* lift at correct location

* add control flow to pc and next

* add categories mostly

* fix direct semantics add cbranch

* track cbranch metavar also lift returns

* cmake: Fix SleighArch.h path

* sleigh: Support more binary ops

* sleigh: Implement ZEXT

* sleigh: Support SEXT op

* sleigh: Implement 2COMP and INT_NEGATE

* mutex around parsing sleigh specs

* fixed typos in custom image

* added handling for exceptions, need to fix issue with xml parsing

* sleigh: Support boolean binops

* sleigh: Support float binary ops

* sleigh: Support a few more float pcode ops that require LLVM intrinsics

* sleigh: Add entries for ops that require overflow intrinsics

* sleigh: Implement more float unary ops

* sleigh: Reduce duplication in ops that require float intrinsics

* expose mutex

* fix fallthough and format of control flow resolution

* sleigh: Support CARRY, SCARRY and SBORROW

* Remove duplicate THUMB2 arch in enum

* Remove duplicate THUMB2 entry in archnames array

* Implement STORE op

* Fix incorrect index

* Support PIECE and SUBPIECE ops

* Support remaining set of PCode ops in the base set. Now to do the
pseudo ops.

* Implement additional ops

* Partially implement a few pseudo ops + a few missing from the docs

* Fix the variadic op condition

* Include <mutex> to build on Linux

* stop lying about insn categories/functions

* set function insns with mnemonic

* fix fallthrough for ind calls

* More virtualization, fewer pimples

* switch back to x86 normal

* Fix off-by-eight issue with ADR in AArch32

* Get rid of mutex in sleigh, make a globally-available remill locker for handling sleigh issues

* CHange allow 32 bit shift to true, not sure if valid but oh well

* Use the node size when creating constants

* fix memory

* store memory state back

* fix calling intrinsic table

* fix comp sizes

* load pspec data

* add pspec names

* add differential test

* fix tests start pretty printer

* add runner

* added really slow memory

* add comparison of memory state

* do reset

* fix uninitialized module

* fixed lift crash

* add amd64 runtime

* unfix fs, gs, and pc

* added whitelist file

* fix memory to update state with uninitialized reads

* better pc handling

* add main.py

* x86 compiles

* fixup patch generation

* add replacements for mem locs and constant varnodes, now need to handle special branches

* added replacement for direct branches

* add context clears

* add handler for claim_eq

* comment out clears for now

* added ambiguity check

* handle duplicate names in same constructor

* mantain sleigh invariant of 1 or 0 in flag

* added uint8t memory intrinsics

* extend shift value when needed

* handle cmovs

* update patch generation

* removed deprecated load

* fix for returned type

* builds

* add logging

* add comment on why not compute GEPAccessors in Arch.cpp

* enable opaque pointers

* fix memory state update

* add mem_16 impls

* not preserve 1 bit width

* fix CR changes

* fix L and R for shift operands bigger than target

* revert use after move

* more CR fixes

* remove lookahead... lets try this again

* rework to create internal function that is inlined to allow for early return control flow

* terminate conditionally

* avoid large iteration counts for .REP insns

* fix patch for rep insns

* handle inst_start in patches

* handle inserting insn_size constructor outside of macros

* hint size of insn_size

* refactor bool functions to make more sense

* just use ;

* remove delete of operands when fail to lift

* remove copy and paste

* remove unused deps

* add patch file to sleigh list

* prep git user

* refer to checkout

* maybe shell type?

* fix layout

* fix format again

* fix

* update dockerfile

* fix gitwatcher to point to current project:

* Revert "fix gitwatcher to point to current project:"

This reverts commit 711da11.

* fix ninja builds

* disable sleigh tests

* remove duplicates

* refactor

* fix non reset context in sleigh lifter

* remove debugging prints

* wrote CI runner

* add handwritten test infra

* refactor to allow sharing JIT test running between differential tester and hand written tests

* allow for internal ownership of semantics module

* handwritten test framework

* fix hand written tests

* pc rel testruns

* add pc rel regression

* fix script for pc rel

* fix names for priors to ignore whitespace

* fix action ops

* add xor to repl ops

* add xors to patch

* add arm patch to build

* fix test for semantics

* allow cross platform tests to run

* add running diff tester

* 2 underscores?

* run tests on macos too

* fix command

* fix whitelist path

* remove unused test data

* install python deps for differential tests

* update script to git patches

* update patches to git format

* attempt to fixup patches

* update ref for sleigh

* install test depends docker

* point docker to correct dir

* install pydeps in macos too

* initialize address field

* exports, also tag master on sleigh

* add comments and whitelist undefined value OF in shifts

* add of accessor to whitelist impl

* change variables to non alloca and remove dead code

* remove dead code:

* replace dump with print

* code quality

* make non null into ref

* type alias

* remove allocas from unit tests

* Address copypasta comment.

* Address PR comment.

* Address PR comment.

* fix build

* unify diff modules into single structure

* remove extra newline

* lock sleigh to specific commit

* remove new lines

* remove useless fenv headers

* caps

* copyright notices

* update more copyright

* fix reviews

* early returns out of accessors

* early return

* auto

* return getarchbyname

* insert register rather than modify reference

* just return

* only log in the assembly logger

* prefer functional style

* remove commented code

* move defaults to header

* that's not how arch switches work in pcode

* informative names

* simplify control flow in fill

* early return

* fix early exit condition

* refactor register default into function

* make one liner

* early return

* unary instead of unop

* construct pair with {}

* move cbranch into binops instead of integer binops

* refactor float ops into getter

* factor out float type

* early return

* separate out callother handling

* do pointer extensions cleaner

* braces

* add private headers to lifters

* refactor redirecting control flow out of instruction

* use constant check function

* remove has_value

* structure for preconditions

* return success when applying eq claim

* expose arch base and move shared functionality into x86 base

* dedup x86 code

* dedup aarch32

* remove needless assignment, also remove else after an if that returns

Co-authored-by: Alex Cameron <[email protected]>
Co-authored-by: Artem Dinaburg <[email protected]>
Co-authored-by: Peter Goodman <[email protected]>
  • Loading branch information
4 people authored Jul 28, 2022
1 parent 66ef46b commit eef338d
Show file tree
Hide file tree
Showing 81 changed files with 8,661 additions and 1,226 deletions.
10 changes: 10 additions & 0 deletions .github/actions/prepare_git_user/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: 'Setup Git User for Applying Patches'
description: 'Setup bot user so git doesnt complain when patching'
# See this thread for more details https://github.community/t/github-actions-bot-email-address/17204/5
runs:
using: "composite"
steps:
- shell: bash
run: |
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
11 changes: 11 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: ./.github/actions/prepare_git_user
- name: Build with build script
shell: bash
run: |
Expand All @@ -48,6 +49,11 @@ jobs:
export VCPKG_ROOT=$(pwd)/../lifting-bits-downloads/vcpkg_${{ matrix.image.name }}-${{ matrix.image.tag }}_llvm-${{ matrix.llvm }}_amd64
export INSTALL_DIR=$(pwd)/remill-preset-install
./scripts/build-preset.sh release
- name: Install Python Test Deps
shell: bash
run: |
pip3 install --user ./scripts/diff_tester_export_insns
- name: Run tests
shell: bash
working-directory: remill-build
Expand Down Expand Up @@ -102,6 +108,7 @@ jobs:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: ./.github/actions/prepare_git_user
- name: Build with build script
shell: bash
run: |
Expand All @@ -114,6 +121,10 @@ jobs:
export VCPKG_ROOT=$(pwd)/../lifting-bits-downloads/vcpkg_${{ matrix.os}}_llvm-${{ matrix.llvm }}_xcode-13.0_amd64
export INSTALL_DIR=$(pwd)/remill-preset-install
./scripts/build-preset.sh release
- name: Install Python Test Deps
shell: bash
run: |
pip3 install --user ./scripts/diff_tester_export_insns
- name: Run tests
shell: bash
working-directory: remill-build
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ cmake-build-debug
cmake-build-release
compile_commands.json

bin/*

third_party/*
build/*
Expand Down
58 changes: 42 additions & 16 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
Expand All @@ -20,7 +20,7 @@ include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/ccache.cmake")

project(remill C CXX ASM)
include(GNUInstallDirs)

include(FetchContent)
include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/settings.cmake")
include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/utils.cmake")
include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/options.cmake")
Expand Down Expand Up @@ -72,6 +72,7 @@ include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/BCCompiler.cmake")
# Go find only the static libraries of LLVM, and link against those.
foreach(LLVM_LIB IN LISTS LLVM_AVAILABLE_LIBS)
get_target_property(LLVM_LIB_TYPE ${LLVM_LIB} TYPE)

if(LLVM_LIB_TYPE STREQUAL "STATIC_LIBRARY")
list(APPEND LLVM_LIBRARIES "${LLVM_LIB}")
endif()
Expand All @@ -88,7 +89,7 @@ target_link_libraries(thirdparty_llvm INTERFACE
# Microsoft Z3 with LLVM. Not exactly used in remill, but LLVM doesn't link
# against it correctly
# NOTE: If changing this, also replicate in remillConfig file
if (LLVM_WITH_Z3)
if(LLVM_WITH_Z3)
find_package(Z3 CONFIG REQUIRED 4.7.1)
get_target_property(LLVMSupport_LIBS LLVMSupport INTERFACE_LINK_LIBRARIES)
list(REMOVE_ITEM LLVMSupport_LIBS Z3)
Expand Down Expand Up @@ -120,15 +121,32 @@ target_link_libraries(thirdparty_gflags INTERFACE
gflags
)

set(sleigh_ENABLE_TESTS OFF)
set(sleigh_ADDITIONAL_PATCHES "${CMAKE_CURRENT_SOURCE_DIR}/patches/sleigh/x86-ia.patch;${CMAKE_CURRENT_SOURCE_DIR}/patches/sleigh/arm-thumb.patch")

# GHIDRA SLEIGH
FetchContent_Declare(sleigh
GIT_REPOSITORY https://github.com/lifting-bits/sleigh.git
GIT_TAG 5ee2f2c16250a6529108e6a6fff89e0f147502d2
)

FetchContent_MakeAvailable(sleigh)
add_library(thirdparty_sleigh INTERFACE)
target_link_libraries(thirdparty_sleigh INTERFACE
sleigh::sla
sleigh::decomp
sleigh::support
)

# Windows SDK
add_library(thirdparty_win32 INTERFACE)

if(DEFINED WIN32)
target_link_libraries(thirdparty_win32 INTERFACE
"Kernel32.lib"
"Kernel32.lib"
)
endif()


# For Linux builds, group LLVM libraries into a single group
# that avoids frustrating library ordering issues.
if(UNIX AND NOT APPLE)
Expand All @@ -150,7 +168,6 @@ option(REMILL_BUILD_SPARC32_RUNTIME "Build the Runtime for SPARC32. Turn this of
#

# add everything as public.

add_library(remill_settings INTERFACE)

target_include_directories(remill_settings INTERFACE
Expand Down Expand Up @@ -235,7 +252,7 @@ target_compile_definitions(remill_settings INTERFACE
"REMILL_BUILD_SEMANTICS_DIR_SPARC64=\"${REMILL_BUILD_SEMANTICS_DIR_SPARC64}\""
)

set(THIRDPARTY_LIBRARY_LIST thirdparty_llvm thirdparty_xed thirdparty_glog thirdparty_gflags)
set(THIRDPARTY_LIBRARY_LIST thirdparty_llvm thirdparty_xed thirdparty_glog thirdparty_gflags thirdparty_sleigh)
target_link_libraries(remill_settings INTERFACE
${THIRDPARTY_LIBRARY_LIST}
)
Expand All @@ -259,7 +276,6 @@ target_link_libraries(remill INTERFACE
#
# Also install clang, libllvm and llvm-link
#

set(INSTALLED_CLANG_NAME "remill-clang-${REMILL_LLVM_VERSION}${CMAKE_EXECUTABLE_SUFFIX}")
set(INSTALLED_LLVMLINK_NAME "remill-llvm-link-${REMILL_LLVM_VERSION}${CMAKE_EXECUTABLE_SUFFIX}")

Expand All @@ -269,15 +285,18 @@ InstallExternalTarget("ext_llvmlink" "${LLVMLINK_PATH}" "BIN" "${INSTALLED_LLVML

GetTargetTree(THIRDPARTY_LIBRARIES ${THIRDPARTY_LIBRARY_LIST})
GetPublicIncludeFolders(THIRDPARTY_INCLUDE_DIRECTORIES ${THIRDPARTY_LIBRARIES})

foreach(THIRDPARTY_LIB IN LISTS THIRDPARTY_LIBRARIES)
string(SUBSTRING "${THIRDPARTY_LIB}" 0 1 THIRDPARTY_LIB_PREFIX)

if(TARGET ${THIRDPARTY_LIB})
get_target_property(THIRDPARTY_LIB_TYPE ${THIRDPARTY_LIB} TYPE)

if(THIRDPARTY_LIB_TYPE STREQUAL "STATIC_LIBRARY" OR THIRDPARTY_LIB_TYPE STREQUAL "SHARED_LIBRARY")
list(APPEND THIRDPARTY_LIBRARY_FILES "$${}<TARGET_FILE:${THIRDPARTY_LIB}>")
endif()
elseif("${THIRDPARTY_LIB_PREFIX}" STREQUAL "$${}")
# E.g. $<LINK_ONLY:...>
# E.g. $<LINK_ONLY:...>
else()
list(APPEND THIRDPARTY_LIBRARY_FILES "${THIRDPARTY_LIB}")
endif()
Expand All @@ -288,31 +307,33 @@ list(REMOVE_DUPLICATES THIRDPARTY_LIBRARY_FILES)
#
# additional targets
#

add_custom_target(semantics)

# shared JIT tools
add_subdirectory(test_runner_lib)

# tools
add_subdirectory(bin)

if(REMILL_ENABLE_INSTALL_TARGET)
install(TARGETS remill EXPORT remillTargets)

install(TARGETS remill_settings ${THIRDPARTY_LIBRARY_LIST}
EXPORT remillTargets
)

# First do the basic substitutions.
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/remillConfig.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/remillConfig.cmake"
@ONLY
)

install(FILES "${CMAKE_CURRENT_BINARY_DIR}/remillConfig.cmake"
"${CMAKE_CURRENT_LIST_DIR}/cmake/vcpkg_helper.cmake"
"${CMAKE_CURRENT_LIST_DIR}/cmake/vcpkg_helper.cmake"
DESTINATION "${REMILL_INSTALL_LIB_DIR}/cmake/remill"
)

install(DIRECTORY "${REMILL_INCLUDE_DIR}/remill/"
DESTINATION "${REMILL_INSTALL_INCLUDE_DIR}/remill"
)
Expand All @@ -322,14 +343,19 @@ if(REMILL_ENABLE_INSTALL_TARGET)
endif()

# tests
if (REMILL_ENABLE_TESTING)
if(REMILL_ENABLE_TESTING)
# Tests require enabling exports on binaries
# https://cmake.org/cmake/help/latest/variable/CMAKE_ENABLE_EXPORTS.html#variable:CMAKE_ENABLE_EXPORTS
set(CMAKE_ENABLE_EXPORTS ON)

find_package(Threads REQUIRED)
add_custom_target(test_dependencies)

if(REMILL_ENABLE_TESTING_SLEIGH_THUMB)
message(STATUS "thumb tests enabled")
add_subdirectory(tests/Thumb)
endif()

if(REMILL_ENABLE_TESTING_X86)
message(STATUS "X86 tests enabled")
add_subdirectory(tests/X86)
Expand Down
11 changes: 8 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,15 @@ ARG LLVM_VERSION

WORKDIR /remill
COPY ./ ./

RUN git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" && git config --global user.name "github-actions[bot]"

RUN ./scripts/build.sh \
--llvm-version ${LLVM_VERSION} \
--prefix /opt/trailofbits \
--extra-cmake-args "-DCMAKE_BUILD_TYPE=Release"
--llvm-version ${LLVM_VERSION} \
--prefix /opt/trailofbits \
--extra-cmake-args "-DCMAKE_BUILD_TYPE=Release"

RUN pip3 install ./scripts/diff_tester_export_insns

RUN cd remill-build && \
cmake --build . --target test_dependencies -- -j $(nproc) && \
Expand Down
7 changes: 5 additions & 2 deletions bin/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Copyright (c) 2017 Trail of Bits, Inc.
# Copyright (c) 2022 Trail of Bits, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
Expand All @@ -14,3 +14,6 @@

add_subdirectory(lift)

if(REMILL_ENABLE_DIFFERENTIAL_TESTING)
add_subdirectory(differential_tester_x86)
endif()
35 changes: 35 additions & 0 deletions bin/differential_tester_x86/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Copyright (c) 2022 Trail of Bits, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

find_package(Python COMPONENTS Interpreter)
add_executable(
lift-and-compare
LiftAndCompare.cpp
Whitelist.cpp
Whitelist.h
)

target_link_libraries(
lift-and-compare
PRIVATE
remill
thirdparty_glog
test-runner
)

set_property(TARGET lift-and-compare PROPERTY ENABLE_EXPORTS ON)
set_property(TARGET lift-and-compare PROPERTY POSITION_INDEPENDENT_CODE ON)
enable_testing()

add_test(NAME "small_diff_test" COMMAND "${Python_EXECUTABLE}" ${REMILL_SOURCE_DIR}/scripts/diff_tester_export_insns/diff_tester_export_insns/ci_runner.py --required_success_rate 1.0 --difftester_bin ${CMAKE_BINARY_DIR}/bin/differential_tester_x86/lift-and-compare --workdir ${CMAKE_BINARY_DIR} ${REMILL_SOURCE_DIR}/bin/differential_tester_x86/data/small_test/ --whitelist_file ${REMILL_SOURCE_DIR}/bin/differential_tester_x86/whitelist.json)
Loading

0 comments on commit eef338d

Please sign in to comment.