VERIFOXX-CHERI-WAMR

This is a private "fork" of the WebAssembly Micro Runtime Repository which is modified to be CHERI-capability-aware and support features to compartmentalise executed WASM. The version in "main" will build the iwasm program (VM Core) for both CHERI Hybrid and pure-cap on Linux, but note the pure-cap version is very cut down and is likely to only run the simplest WASM code!

The version in "develop" is latest completed changes representing the port to CHERI Morello Linux. Interpreter mode is supported, both Fast and Classic, and all build-time flags apart from WAMR_BUILD_DEBUG_INTERP. AOT mode is supported, JIT mode is not supported.

An experimental compartmentalised WAMR is additionally supported - this is implemented as a separate "mini-product". This utilises iwasm as a capability manager executable which manages a compartment that contains all WAMR code. All WAMR API calls therefore take place via a switch to the compartment. During a WAMR function processing, the function can "call back" to a service in the compartment to perform system functions such as allocating memory. Note this work is very much in progress and therefore the compartment product does not have as much functional support as the main WAMR port to CHERI. The following are not currently supported in the compartmentalised version:

• AOT mode
• Calling native functions in a loaded library ("native-lib" option) (including building a native test library)
• Most system functions are not moved to the capability manager
• The full WAMR API is not supported (only a subset to implement iwasm)
• Building with the LLVM toolchain is not supported yet, only GNU/GCC is supported

*Note: The compartmentalised version is only available in pure-cap, hybrid-cap cannot be supported

Building CHERI-WAMR with CMake

You can build from the root folder directly with CMake, but this requires a number of flags to configure WAMR and the build system. It is therefore recommended to consult ./build_iwasm.sh which allows you to configure the following:

• CHERI_PURECAP=[0|1] (set to 1 for purecap mode, 0 for hybrid)
• CONFIG_TYPE=Debug|Release
• NATIVE_TEST_LIB=[0|1] (set to 1 to additionally build a native test shared object for iwasm, 0 to skip this)
• AOT_CHERI_PTR=[0|8|16] (set to 0 when building CHERI pure-cap, set to pure-cap native pointer size when building other architectures. Must match setting used to build WAMRc.)
• WAMR_EXTERNREF_APP=[0|1] (set to 1 to build a native executable for externref testing INSTEAD of building WAMR. Defaults to build WAMR.)
• INSTALL_PREFIX= (where you want output to be written from cmake --install)
• CHERI_STATIC_BUILD=[0|1] (set to 0 to build dynamic iwasm which requires e.g libc available at runtime. Default to static.)

This bash script assumes you will be using a toolchain file, see below for more on tihs.

Supported WAMR Mini-Products

The CMake flag WAMR_BUILD_PLATFORM determines which product platform to build for, these map to a folder in the WAMR source tree. For CHERI-WAMR we derive products from linux which is a native Linux build (e.g on x86_64 platforms).

The following are supported:

• WAMR_BUILD_PLATFORM=linux-cheri-purecap : The standard CHERI port of WAMR, enables you to build either pure-cap or hybrid-cap WAMR
• WAMR_BUILD_PLATFORM=linux-cheri-purcap-capmgr : The experimantal compartmentalised version which builds iwasm as a capability manager executable, and WAMR into a dynamic shared object library (libiwasm.so)

The Toolchain File on CHERI platforms

The Cmake build can use the toolchain file toolchain.cmake to build for CHERI platforms using the GNU toolchain, or llvm_toolchain.cmake for the LLVM toolchain. You should edit the desired file accordingly to specify the path to the GCC and GCC compiler binaries.

GNU Toolchain

By default the toolchain file assumes that gcc and g++ are found on your path. You must also set en evnironment variable CHERI_GNU_TOOLCHAIN_DIR to the folder containing the GNU toolchain, this is used to dervive sysroot.

LLVM Toolchain

If using the LLVM toolchain, set environment variable CHERI_LLVM_TOOLCHAIN_DIR to the LLVM root folder (such that clang would be found as CHERI_LLVM_TOOLCHAIN_DIR/bin/clang.
As LLVM toolchain for Morello depends on MUSL_LIBC, which is separately installed, you also need to set environment variables CHERI_MUSL_TOOLCHAIN_DIR to the purecap build of MUSL and CHERI_MUSL_TOOLCHAIN_DIR_HYBRIDCAP to the hybrid build of MUSL. Doing this allows the sysroot to be set correctly.

Building without a Toolchain

Use of the toolchain file is optional. If you do not use the toolchain file, the CMakelists.txt will attempt to resolve your compilers based on the architecture you are running on. If this is aarch64 then it assumes you are building on the morello board. Otherwise, it assumes you are cross-compiling.

You will though need to provide the path to the GNU or LLVM toolchain root if they are not on your path. You can provide this directly by passing -DCHERI_GNU_TOOLCHAIN_DIR=/path/to/gnu/root or -DCHERI_LLVM_TOOLCHAIN_DIR=/path/to/llvm/root as a Cmake argument. For LLVM you will also need to specify the MUSL root folder with -DCHERI_MUSL_TOOLCHAIN_DIR (purecap) or -DCHERI_MUSL_TOOLCHAIN_DIR_HYBRIDCAP (hybrid) as described above for building with an LLVM toolchain file.

The Toolchain File on Linux x86_64 native platforms

A basic toolchain file is also provided for GNU and LLVM to build for Linux on non-CHERI (assumed to be x86_64) platforms. This can be used to build a baseline test-harness for Linux that can run natively.

Native Test Harness Library

Note: Native Test Library not currently supported on the compartmentalised / capability manager WAMR.

To call from WASM -> native or native -> WASM requires an implementation of suitable native functions which need to be known by WAMR. iwasm handles this by loading a DLL (linux shared object library) at runtime and discovering native functions.

The CMake build system supports building a test native library as part of the root-level CMake build. This can build on Linux x86_64, Linux CHERI Morello Pure-cap and hybrid-cap. The code can be found in tests/wamr-linux-cheri-purecap-tests/native_libs and there is a CMakeLists.txt for each supported target.

The root CMakeLists.txt will build the native libs shared object after building WAMR if CMake flag WAMR_BUILD_NATIVE_TEST_LIB flag is set to 1.

Please refer to the Native Test Library ReadMe for further information.

Building without the bash script

mkdir build && cd build
cmake .. [--toolchain ../toolchain.cmake|--toolchain ../llvm_toolchain.cmake|-DCHERI_GNU_TOOLCHAIN_DIR=<path>|-DCHERI_LLVM_TOOLCHAIN_DIR=<path> -DCHERI_MUSL_TOOLCHAIN_DIR=<path>] \
     -DCMAKE_BUILD_TYPE=Debug|Release --install-prefix=<path> \
	-DWAMR_BUILD_PLATFORM=[linux-cheri-purecap|linux-cheri-purecap-capmgr] [-DCHERI_PURECAP=0|1] [-DCHERI_STATIC_BUILD=1|0] [-DWAMR_BUILD_NATIVE_TEST_LIB=0|1] \
	-DWAMR_BUILD_AOT_CHERI_PTR=[0|8|16] -DWAMR_BUILD_AOT_EXCEPTION_WORKAROUND=[0|1] [-DWAMR_EXTERNREF_APP=0|1][<wamr-build-flags>]

cmake --build .
cmake --install .

Where:

• WAMR_BUILD_PLATFORM is required, for CHERI this must be "linux-cheri-purecap" for the main port, or "linux-cheri-purecap-capmgr" for the experimantal compartmentalisation CHERI-WAMR.
• CHERI_GNU_TOOLCHAIN_DIR is path to the morello gnu toolchain root on the build machine, not required if folder containing binaries is on your bash path OR you are using a toolchain file
• CHERI_PURECAP is 1 for purecap builds and 0 for hybrid capability builds (default 1)
• CHERI_STATIC_BUILD is 1 for making a static executable, 0 for requiring .so libs (default 1)
• WAMR_BUILD_NATIVE_TEST_LIB is 0 for not additionally building the native test lib, 1 for building it (default 0)
• WAMR_BUILD_AOT_CHERI_PTR is 0 on pure-cap platforms, pure-cap pointer size for hybrid or non-CHERI platforms (refer to WAMRc build for more details, and below)
• WAMR_BUILD_AOT_EXCEPTION_WORKAROUND is 1 to include, 0 to exclude, a workaround for _wasi_proc_exit not terminating the AOT script when DWAMR_DISABLE_HW_BOUND_CHECK=1 (ByteCodeAlliance WAMR deficit)
• WAMR_EXTERNREF_APP is 1 to build a standalone native application for externref testing INSTEAD of WAMR. Using this flag will NOT build WAMR at all (default 0).
• are any flags to configure WAMR (refer to WAMR build readme for more info).

Note: For CHERI Morello, WAMR build options contain some restrictions:

• JIT mode is not supported so must be 0
• WAMR_BUILD_DEBUG_INTERP is not yet supported
• WAMR_BUILD_SIMD must be 0
• WAMR_DISABLE_HW_BOUND_CHECK must be 1

Note: For linux-cheri-purecap-capmgr the following must at this time be set otherwise WAMR will not build or run correctly:

• WAMR_BUILD_AOT=0
• WAMR_EXTERNREF_APP=0
• WAMR_BUILD_NATIVE_TEST_LIB=0
• CHERI_PURECAP=1

Bulding on the Morello Target

This document assumes you will be cross-compiling, however you can build on the Morello target itself. To do this either update the relveant toolchain cmake file or supply the CHERI_[GNU|LLVM]_TOOLCHAIN_DIR flag if the GNU toolchain is not on your path (on the Morello board). The cmake script will then resolve the correct toolchain binaries, as if the architecure where it is being run is aarch64 then it is assumed to be building on the morello target. Otherwise, it will cross-compile.

Building CHERI-WAMR using Visual Studio and WSL2

The file CMakePresets.json is provided to support visual studio C++ CMake remote builds on a Linux Ubuntu machine running under WSL2. This provides separate configurations for building locally and cross-compliling for Morello:

• Hybrid Debug
• PureCap Debug (Standard CHERI port)
• PureCap CapMgr Debug (Experimental compartmentalised CHERI port)
• Hybrid Release
• PureCap Release (Standard CHERI port)
• PureCap CapMgr Release (Experimental compartmentalised CHERI port)
• Linux x86_64 Debug (a non-CHERI baseline for santity testing)
• Linux x86_64 Release (a non-CHERI baseline for santity testing)
• Externref_App_Purecap (build a separate, standalone native exe for externref testing instead of WAMR (on Morello purecap))
• Externref_App_x86_64 (build a separate, standalone native exe for externref testing instead of WAMR (on x86_64))
• WAMRc Linux Debug x86_64 (build the WAMR Compiler to run on a non-Morello machine)
• Benchmarks Debug Arm64 Hybrid (build the native benchmark tests for Morello hybrid)
• Benchmarks Debug Arm64C+ Purecap (build the native benchmark tests for Morello purecap)

To use VS with CMake and this project "out of the box" you will need to have first carried out the following pre-requisite steps:

1. Enable WSL2 in Windows (10 or 11) and installed a suitable Ubuntu distro
2. Install CMake on the Ubuntu machine, you will need version 3.19 or newer (note: not 100% guaranteed to work with earlier than 3.24!)
3. Set up the Arm Morello GNU Toolchain on your WSL2 distribution for cross-compilation to Morello (i.e linux-x86_64 -> aarch64(+c64))
4. Add the path to the GNU toolchain binaries (gcc, g++ etc.) to your sign-on bash shell path (e.g in .bashrc)
5. Add export CHERI_GNU_TOOLCHAIN_DIR=/path/to/toolchain/root, or export CHERI_LLVM_TOOLCHAIN_DIR=/path/to/toolchain/root and export CHERI_MUSL_TOOLCHAIN_DIR=/path/to/musl/root, to your sign-on bash shell path (e.g in .bashrc)
6. Install Visual Studio 2019 or newer with the C++ for Linux / CMake component

Launching the project

In Visual Studio, select to open and choose CMake and then locate the CMakeLists.txt in the WAMR root folder. Visual Studio will automatically detect the CMakePresets.json.

You must then select the your Ubuntu machine as the build target. You can then choose your configuration, the following are available:

• Debug armC64 Hybrid (internal name "ARMc64-hybrid-debug")
• Debug armC64+ PureCap (internal name "ARMc64-PureCap-debug")
• Debug armC64+ CapMgr PureCap (internal name "ARMc64-PureCap-CapMgr-debug")
• Release armC64 Hybrid (internal name "ARMc64-hybrid-release")
• Release armC64+ PureCap (internal name "ARMc64-PureCap-release")
• Release armC64+ CapMgr PureCap(internal name "ARMc64-PureCap-CapMgr-release")
• Linux Debug x86_64 (internal name "Linux_x86_64_Debug")
• Linux Release x86_64 (internal name "Linux_x86_64_Release")
• Externref_App_Purecap (internal name "Externref_App_Purecap")
• Externref_App_x86_64 (internal name "Externref_App_x86_64")

NOTE: CapMgr PureCap versions are for the experimental compartmentalisation CHERI-WAMR, other PureCap versions are for the standard CHERI-WAMR port.

All options are then set up correctly. Visual Studio will automatically build makefiles via CMake and you can build the codebase. NOTE: Please modify the following flags as necessary to configure your build:

• WAMR_BUILD_AOT (default to 1 to include AOT support, set to 0 to exclude AOT)
• WAMR_BUILD_FAST_INTERP (default 1, set to 0 for classic interpreter mode)
• WAMR_BUILD_DEBUG_PREPROCESSOR (default 0, set to 1 for debug info)
• WAMR_BUILD_MEMORY_TRACING (default 0, set to 1 for extra trace info)
• WAMR_BUILD_NATIVE_TEST_LIB (default 0, set to 1 to additionally build the native test shared object file)

Building the Externref Test Application

Note that if building Externref_App then WAMR is Not additionally built, instead a native application is built which embeds WAMR library. Also note that all WAMR build options are hardcoded for this native application and cannot be overriden from CMakePresets.

Troubleshooting

1. If your project is located on a virtual drive under windows (i.e a subst drive) then you will have to make this available to WSL2. Although most files are copied to WSL2 via rsync, the Ubuntu installation will still need your virtual drive mounted (i.e available as /mnt/drive_letter).

There are a number of ways to do this, but by far the easiest is to create a symbolic link. For example, assuming your windows substituted drive is N: which maps to C:\Verifoxx then proceed as follows on Ubunutu:

sudo ln -s /mnt/c/Verifoxx /mnt/n

2. You should ensure your .bashrc is able to be executed in "headless" mode i.e when there is no terminal. The easiest way to achieve this is to add the following line near the top of your .bashrc:

[[ $- != *i* ]] && return

3. You are recommended to add the path to the toolchain binaries to your .bashrc file. Ensure this is BEFORE the line shown in the no terminal mode, above. However it can be supplied by modifying CMakePresets.json. The problem though with this is it will reduce the portability of the project.

**Note: The CMakePresets.json is set up with the correct WAMR options to build a version that works in Fast Interpreter mode without a native .so included **

Debugging on Morello Board with Visual Studio

You can remotely debug from Visual Studio with the target being the Morello board. However, this requires some setup because the debug target is not the same as the build target.

You can either run gdb on the morello board, or run gdbserver on the morello board and gdb locally. The former is recommended, not least as there were some issues with gdbserver on morello-aarch64+c64 at the time of writing. This is the method described below.

Prerequisites

1. The Morello board must have a known IP address on the internal network (e.g configure it with a static IP)
2. The Morello board should be running sshd and accept remote connections
3. You may need to enable connections in your firewall

Visual Studio Setup

1. In Tools -> Options, add a remote connection. You will need to supply the IP address of the morello board along with the login credentials (user="root", pass="morello").
2. Once a build has succeeded, under Debug configure a launch configuration. This will open a file launch.vs.json which you must then edit to resemble the below:

{
  "version": "0.2.1",
  "defaults": {},
  "configurations": [
    {
      "args": [
        "-v=5",
        "--interp",
        "--native-lib=/root/libwamr-native-test.so",
        "/root/wamr_natives_and_call_wasm.wasm"
      ],
      "cwd": "/root",
      "debuggerConfiguration": "gdb",
      "env": {},
      "gdbPath": "/root/morello_gnu/bin/aarch64-none-linux-gnu-gdb",
      "name": "Morello Target",
      "project": "CMakeLists.txt",
      "projectTarget": "iwasm",
      "remoteMachineName": "192.168.0.39",
      "type": "cppgdb",
      "targetArchitecture": "arm64",
      "externalConsole": true,
      "deployDirectory": "/root/tmp",
      "disableDeploy": false,
    }
  ]
}

Where:

• args are command-line arguments to pass to iwasm, run iwasm --help for details
• cwd is the working folder for debug session
• debuggerConfiguration should be gdb
• gdbPath is the filepath for the gdb binary. Version shown above is built for the morello board (i.e runs on morello).
• name and projectTarget set to appropriate for your project configuration
• remoteMachineName is the IP address of the Arm morello board
• targetArchitecture must be an architecture known to visual studio; the nearest we can get is "arm64"
• name is the debug target name to display in the debug target selection box.
• externalConsole should be true to correctly view terminal output in VS Debugger window
• deployDirectory is the folder to copy the target to on debugger launch
• disableDeploy should be false

You can now proceed to select debug target as per name, above, and then debug. The process will launch remotely on the Morello board connecting across your internal network.

Enhanced Visual Studio Debug Configuration

You can define additional debug configurations. For example, you may have a separate configuration for the experiemental compartmentalisation version of iwasm. As this requires a shared object (.so) at runtime which contains the WAMR compartment code, it is beneficial to have the debug step automatically copy files to the remote target.

The configuration shown below will achieve this:

{
  "version": "0.2.1",
  "defaults": {},
  "configurations": [
    {
	  "name": "Morello CAP-MGR",
      "args": [
        "-v=5",
        "/root/helloworld.wasm"
      ],
      "cwd": "/purecap-lib",
      "debuggerConfiguration": "gdb",
      "env": {},
      "gdbPath": "/root/morello_gnu/bin/aarch64-none-linux-gnu-gdb",
      "project": "CMakeLists.txt",
      "projectTarget": "iwasm",
      "remoteMachineName": "192.168.0.39",
      "type": "cppgdb",
      "targetArchitecture": "arm64",
      "externalConsole": true,
      "deployDirectory": "/purecap-lib",
      "deployDebugRuntimeLibraries": "true",
      "disableDeploy": false,
      "program": "/purecap-lib/iwasm",
      "MIMode": "gdb",
      "miDebuggerServerAddress": "192.168.0.39",
      "deploy": [
        {
          "targetMachine": "192.168.0.39",
          "sourcePath": "${cmake.remoteBuildRoot}/linux-cheri-purecap-capmgr/libiwasm.so",
          "targetPath": "/purecap-lib/libiwasm.so",
          "deploymentType": "RemoteRemote",
          "executable": false
        },
        {
          "targetMachine": "192.168.0.39",
          "sourcePath": "${cmake.remoteBuildRoot}/linux-cheri-purecap-capmgr/iwasm",
          "targetPath": "/purecap-lib/iwasm",
          "deploymentType": "RemoteRemote",
          "executable": true
        }
      ]
    }
  ]
}

This version will deploy both iwasm (capability manager iwasm executable) and libiwasm.so (WAMR compartment library) to a folder /purecap-lib. This folder is what was set as the library RPATH so that WAMR can find all other dynamic shared objects at runtime.

The deploy settings tell Visual Studio the files to be copied to the remote machine - these are sourced from the default build folder for linux-cheri-purecap-capmgr.

This configruation is named "Morello CAP-MGR". Therefore if you have a "Morello Target" configuration this can be used to debug standard WAMR, and "Morello CAP-MGR" can be used for the experimental capability manager / commpartments version.

Debugging and Intellisense Operation

There are limitations to how well intellisense and debugging can work with Visual Studio, because it has to be told the architecture is Arm64 and it doesn't know about Morello.

However, the CMakePresets.json has been specifically set up - and this is why it also needs to use a toolchain file - to make the process as effective as possible:

• C and C++ headers will be picked up from the GNU toolchain on the Linux machine, not the VS local flavour
• C/C++ flags and built-in macro definitions should work in Intellisense, for example CHERI_PURE_CAPABILITY is defined for morello pure-cap.
• Debugging should work fairly seamlessly although you are unlikely to be able to debug into library functions and e.g STL headers (this feature can be enabled/disabled in VS Tools -> Options)

The only real problem is that compiler builtins which are pure-cap specific will likely not work.

Experimental Compartmentalised WAMR

The experimental version of CHERI-WAMR is added as the separate product linux-cheri-purcap-capmgr to avoid corrupting the main CHERI-WAMR port. This product builds an executable, iwasm, and a dynamic shared object library, libiwasm.so. The executable is a capability manager which will dynamically load (via dlopen()) the WAMR library at runtime.

The iwasm executable manages a single CHERI compartment within which all WAMR API functions are run. Each call to a WAMR function involves a switch from the capability manager into the compartment, then the function is executed, and then execution returns to the capability manager. At the moment only a single compartment is implemented.

The capability manager runs in the Morello Executive PE state and all compartment functions run in the Restricted PE state. To support this, all global code and data symbols implemented in the compartment library need to be patched after the library is first loaded, as by default the dynamic loader will resolve all the symbol addresses with a capability that has the executive permissions and bounds set to the executable. This "symbol relocation patching" is performed by the capability manager executable after the library is first loaded.

Additionally, during processing a WAMR function in the compartment it may be necessary to call a "service" within the capability manager. This needs to be done whenever there is a native or system call that can only be performed from the executive PE state. This is supported via a framework which allows for a call back into the capability manager to run a particulr service, before then returning to the compartment. Currently, memory allocation are the only services which make use of this mechanism.

NOTE: System calls should not be possible from within the compartment, but currently we still make use of this because the Arm Linux port does not prevent said calls from the PE restricted state.

Runtime Changes to iwasm Program

This experiemental version of iwasm introduces a modified command-line option and a new command-line option. With reference to iwasm usage information, the following are changed:

iwasm [--wamr-lib=/path/to/libiwasm.so] [-v=0|1|2|3|4|5|6]

Where:

• --wamr-lib is a new optional argument which specifies the name of the WAMR compartment shared object library to load at runtime. The default value is ./libiwasm.so i.e libiwasm.so in the current directory.
• -v=6 is a new debug level which is identical to the WAMR verbose level of 5, but additionally causes the capability manager to dump all symbol relocation tables parsed from the libiwasm.so ELF

Toolchain Support

At the current time LLVM is not supported, therefore the GNU toolchain is required. LLVM support will need to deal with the differences in how symbol relocation is performed when a shared object is loaded at runtime.

Install Location

Performing the install step (e.g "install iwasm" from Visual Studio, or cmake --install from command-line) will generate:

• /bin/iwasm
• /lib/libiwasm.so

By default, will be ${HOME}/install/ARMc64-PureCap-CapMgr-debug.

Static or Dynamic Capability Manager

The capability manager can either be built to be a statically linked executable or a dynamically linked executable. This is controlled with the CHERI_STATIC_BUILD CMake flag. The default is statically linked, and at the time this is recommended. Dynamically linked executable is likely to be needed in the future, when libraries for which there is not a static version will be needed.

Note that the WAMR compartment library (libiwasm.so) has to always be built dynamically.

Implications of Dynamic Capability Manager Executable

If the capability manager is build to a dyanmic executable then all dependent libraries must be loaded at runtime, as is the case for the libiwasm.so WAMR library. However the same copy of depedent libraries cannot be used by the executable and the libiwasm.so, because they run in different PE states. Therefore if the capability manager executable is built dynamic, the libiwasm.so library is loaded to a different namespace (via Linux system call dlmopen()) which will cause multiple versions of dependent libraries (such as libc.so) to be loaded. This is all handled by the capability manager.

Library Search Path

The WAMR library requires other shared object libraries (e.g libC) at runtime. As these need to be a PureCap flavour, the default search path cannot be used therefore the search path must be specified. By default, this is /purecap-lib on the Morello box. It is recommended that all libraries and capability manager iwasm executable are placed in this same folder, to make things easier during development.

If you want to change the search path, modify the following flag in CMakePresets.json:

• MORELLO_PURECAP_LIBS_FOLDER=your/path/here

NOTE: If the capability manager executable is dynamically linked, it will search for needed libraries in this folder as well.

Capability Manager Logging

The iwasm Capability Manager executable uses a separate logger to the WAMR library, but the logging levels are analagous to WAMR. The level of both is controlled by the -v=n iwasm command-line argument.

Note that at VERBOSE levels (> 4) a significant amount of logging is produced by the capability manager because all symbol reloctaion tables and patch-ups are logged. Note also that the log is not flushed each time and happens through separate mechanisms, therefore is is possible for a WAMR log point to be written before an earlier capability manager log point.

Source Code Changes for Compartmentalisation

A new folder is created specifically for supporting the capability manager and compartment framework, this can be found in: ./verifoxx-cheri-cap-mgr

Subfolders within this are:

• compartment/ : Compartment entry point and wrapper to call into a WAMR function
• common/ : Code shared between compartment and capability manager
• utils/ : Logger and utility classes for managing capabilities
• capgmr/ : Capability Manager files including main(), within main.cpp
• capmgr/cap_relocs/ : Responsible for patching relocation tables of loaded libraries
• capmgr/cap_relocs/link_map_internal/ : Access to internal GNU libC structures that are not normally available through the Std C API

Additionally, to support service callbacks for memory allocation and the now two CHERI related products, a shared folder is created for product platform specifics: ./core/shared/platform/common/cheri-purecap

With the specialisations in:

• ./core/shared/platform/common/cheri-purecap
• ./core/shared/platform/linux-cheri-purecap (standard CHERI-WAMR port)
• ./core/shared/platform/linux-cheri-purecap-capmgr (experiemental compartmentalised CHERI-WAMR)

Incomplete Functionality

At the time of writing there are a number of features not supported by the experimantal capability manager and compartments port:

• AOT mode is not supported yet (requires code changes to the internal PLT table in WAMR AOT handling)
• User native library functions are not supported (requires code changes to be able to call back to capability manager service for native calls)
• Most system calls / native calls (e.g WASI low-level calls) remain in compartment (only memory allocation calls back to capability manager; requires extensive changes to split WAMR into multiple parts)
• Full WAMR API is not yet supported, so only iwasm calls can be used (means calling WAMR from a native application may be limited; would require additional WAMR functions to be compartmentalised)

WAMR-App Front-end

IMPORTANT: The separate front-end is now deprecated and should no longer be used

For development purposes a Limited WAMR front-end is provided which can be used instead of iwasm. This is found in /front-end folder.

You can build for the front-end instead of iwasm by passing an additional flag to CMake or add to CMakePresets.json as follows: WAMR_APP=1

(default is 0). This will cause the root CMakeLists.txt to include the CMakeLists.txt in frontend/ which will build wamr-app instead of iwasm. Note that wamr-app does not include the libvm instead all source code is built into the application.

Basic usage of wamr-app:

wamr-app <wasm_file_to_execute>

For full usage run wamr-app with no arguments.

wamr-app is functional in classic or fast interpreter mode, verbose logging is enabled (and cannot be changed) and stack and app heap size are fixed. Test native functions are included by building them into the wamr-app program.

Externref Test Application

The externref test application is a native application that embeds WAMR functionality. It is needed for testing externrefs when it is not possible to call WASM -> native -> WASM, because the called native cannot create a full execution environment. Refer to Externref Test Application for details.

NOTE: The Extenref Test Application does not suppor the experiemental compartmentalised WAMR at this time.

Using AOT Mode and Compatibility with WAMRc

Please also refer to Building the AOT Compiler for more details on WAMRc support for CHERI.

WAMRc is used to compile WASM and generate an AOT file. This works by compiling WASM to LLVM IR and then generating target-specific object code from this.

Unlike WAMR, WAMRc is not intended to be built for a single target. Instead, the WAMRc target is resolved from WAMRc command line arguments.

In the case of CHERI Morello targets, WAMRc will run on Linux x86_64 and cross-compile for Morello hybrid-cap or pure-cap.

However this causes a problem because some WAMR structures must be laid out identically on both WAMRc build machine and WAMR run machine. In the case of WAMR on Morello pure-cap, all pointers are full capability and twice the size of a non-pure-cap platform. To solve this issue, the build flag WAMR_BUILD_AOT_CHERI_PTR is introduced.

Using WAMR_BUILD_AOT_CHERI_PTR build flag

This macro is used on both WAMRc and WAMR. When non-zero, it specifies the maximum size of a pointer on any target which WAMRc can generate an AOT file for. This causes WAMRc to always pad and align structure pointers to the value given in WAMR_BUILD_AOT_CHERI_PTR.

In order for this to then work, the WAMR which will execute the AOT file must also be sure to align and pad pointers in the same structures to the same size. On CHERI pure-cap platforms, when building WAMR then WAMR_BUILD_AOT_CHERI_PTR should be set to 0. This is because the pointers are naturally the specified size anyway. On all other platforms, when building WAMR then WAMR_BUILD_AOT_CHERI_PTR should be set the same as on WAMRc.

Runtime checks when WAMR_BUILD_AOT_CHERI_PTR used

When WAMR_BUILD_AOT_CHERI_PTR is != 0 then the following checks will take place:

• When running WAMRc, it will warn you need to build WAMR with the flag set if you build for a target that is not CHERI Pure-cap
• When running WAMR, it will check affected structures are laid out correctly based on the value of WAMR_BUILD_AOT_CHERI_PTR

Example of using WAMR_BUILD_AOT_CHERI_PTR

When CHERI Morello pure-cap is a potential WAMRc Target: - Build WAMRc with WAMR_BUILD_AOT_CHERI_PTR=16 - Build Linux x86_64 with WAMR_BUILD_AOT_CHERI_PTR=16 - Build Morello Hybrid-cap with WAMR_BUILD_AOT_CHERI_PTR=16 - Build Morello pure-cap with WAMR_BUILD_AOT_CHERI_PTR=0 (or not defined)

Benchmark Tests

The original WAMR implementation supports a number of benchmark tests. See Benchmarks for more information.
These are still relevant, and have been adapted to support CHERI as follows:

• Coremark, Dhrystone, Polybench and Sightglass Benchmark folders now each contain a cheri_build.sh script along with build.sh which allows building a hybrid aarch64 native as well as a purecap flavour application.
• Native applications support CMake building, and CMakePresets.json_ contains a "Benchmark Hybrid" and "Benchmark Purecap" configuration to allow hybrid and purecap native debug applications to be built for testing purposes, e.g from Visual Studio.

Note that the native code has not been ported to CHERI; for example, the coremark native test application if built "as is" will cause a segmentation fault on Morello purecap.

Benchmark Autorun Script

To obtain a measure of CHERI purecap performance vs hybrid performance we can run benchmark tests through WAMR on all relevant platforms. For this we have WASM on both WAMR purecap and WAMR hybrid, AOT for both and optionally a native application as well.
This is a significant loading when considering the large number of benchmark tests available. To help in the execution of all of these a script has been developed which will automate the process of spinning up WAMR on the Morello box, run each test, and optionally additionally compile WASM to AOT suitable for the platform in question. The script will run through all benchmarks, gather results in a CSV file and allow for offline processing to obtain metrics.

Please refer to Autorun Benchmark Script for more details.

ORIGINAL DOCUMENT -> WebAssembly Micro Runtime

A Bytecode Alliance project

Guide  Website  Chat

Build WAMR | Build AOT Compiler | Embed WAMR | Export Native API | Build Wasm Apps | Samples

WebAssembly Micro Runtime (WAMR) is a lightweight standalone WebAssembly (Wasm) runtime with small footprint, high performance and highly configurable features for applications cross from embedded, IoT, edge to Trusted Execution Environment (TEE), smart contract, cloud native and so on. It includes a few parts as below:

• VMcore: A set of runtime libraries for loading and running Wasm modules. It supports several execution modes including interpreter, Ahead-of-Time compilation(AoT) and Just-in-Time compilation (JIT). The WAMR supports two JIT tiers - Fast JIT, LLVM JIT, and dynamic tier-up from Fast JIT to LLVM JIT.
• iwasm: The executable binary built with WAMR VMcore supports WASI and command line interface.
• wamrc: The AOT compiler to compile Wasm file into AOT file
• Useful components and tools for building real solutions with WAMR vmcore:
  • App-framework: A framework for supporting APIs for the Wasm applications
  • App-manager: a framework for dynamical loading the Wasm module remotely
  • WAMR-IDE: An experimental VSCode extension for developping WebAssembly applications with C/C++

Key features

• Full compliant to the W3C Wasm MVP
• Small runtime binary size (~85K for interpreter and ~50K for AOT) and low memory usage
• Near to native speed by AOT and JIT
• Self-implemented AOT module loader to enable AOT working on Linux, Windows, MacOS, Android, SGX and MCU systems
• Choices of Wasm application libc support: the built-in libc subset for the embedded environment or WASI for the standard libc
• The simple C APIs to embed WAMR into host environment, see how to integrate WAMR and the API list
• The mechanism to export native APIs to Wasm applications, see how to register native APIs
• Multiple modules as dependencies, ref to document and sample
• Multi-thread, pthread APIs and thread management, ref to document and sample
• wasi-threads, ref to document and sample
• Linux SGX (Intel Software Guard Extension) support, ref to document
• Source debugging support, ref to document
• XIP (Execution In Place) support, ref to document
• Berkeley/Posix Socket support, ref to document and sample
• Multi-tier JIT and Running mode control
• Language bindings: Go, Python

Wasm post-MVP features

• wasm-c-api, ref to document and sample
• 128-bit SIMD, ref to samples/workload
• Reference Types, ref to document and sample
• Non-trapping float-to-int conversions
• Sign-extension operators, Bulk memory operations
• Multi-value, Tail-call, Shared memory

Supported architectures and platforms

The WAMR VMcore supports the following architectures:

• X86-64, X86-32
• ARM, THUMB (ARMV7 Cortex-M7 and Cortex-A15 are tested)
• AArch64 (Cortex-A57 and Cortex-A53 are tested)
• RISCV64, RISCV32 (RISC-V LP64 and RISC-V LP64D are tested)
• XTENSA, MIPS, ARC

The following platforms are supported, click each link below for how to build iwasm on that platform. Refer to WAMR porting guide for how to port WAMR to a new platform.

• Linux, Linux SGX (Intel Software Guard Extension), MacOS, Android, Windows, Windows (MinGW)
• Zephyr, AliOS-Things, VxWorks, NuttX, RT-Thread, ESP-IDF

Getting started

• Build VM core and Build wamrc AOT compiler
• Build iwasm (mini product): Linux, SGX, MacOS and Windows
• Embed into C/C++, Embed into Python, Embed into Go
• Register native APIs for Wasm applications
• Build wamrc AOT compiler
• Build Wasm applications
• Port WAMR to a new platform
• VS Code development container
• Samples and Benchmarks

Performance and memory

• Blog: The WAMR memory model
• Blog: Understand WAMR heaps and stacks
• Blog: Introduction to WAMR running modes
• Memory usage tunning: the memory model and how to tune the memory usage
• Memory usage profiling: how to profile the memory usage
• Benchmarks: checkout these links for how to run the benchmarks: PolyBench, CoreMark, Sightglass, JetStream2
• Performance and footprint data: the performance and footprint data

Project Technical Steering Committee

The WAMR PTSC Charter governs the operations of the project TSC. The current TSC members:

• dongsheng28849455 - Dongsheng Yan, dongsheng.yan@sony.com
• loganek - Marcin Kolny, mkolny@amazon.co.uk
• lum1n0us - Liang He， liang.he@intel.com
• no1wudi Qi Huang, huangqi3@xiaomi.com
• qinxk-inter - Xiaokang Qin， xiaokang.qxk@antgroup.com
• wei-tang - Wei Tang， tangwei.tang@antgroup.com
• wenyongh - Wenyong Huang， wenyong.huang@intel.com
• xujuntwt95329 - Jun Xu， Jun1.Xu@intel.com
• xwang98 - Xin Wang， xin.wang@intel.com (chair)
• yamt - Takashi Yamamoto, yamamoto@midokura.com

License

WAMR uses the same license as LLVM: the Apache 2.0 license with the LLVM exception. See the LICENSE file for details. This license allows you to freely use, modify, distribute and sell your own products based on WAMR. Any contributions you make will be under the same license.

More resources

• Who use WAMR?
• WAMR Blogs
• Community news and events
• WAMR TSC meetings