docs: add syzos.md

ramosian-glider · ramosian-glider · commit 31c0c6ae6b3d · 2026-01-13T10:01:10.000Z
Initial commit of the SYZOS technical documentation
diff --git a/docs/syzos.md b/docs/syzos.md
@@ -0,0 +1,214 @@
+# SYZOS Technical Documentation
+
+## 1. System Overview
+
+### Concept
+SYZOS is not a traditional operating system but an **immutable C library** designed to run as a Guest (L1) within a KVM virtual machine. Its primary purpose is to expose an easy-to-fuzz API to the Host (L0) fuzzer (`syzkaller`), allowing for state-aware interactions that are difficult to achieve with raw instruction fuzzing.
+
+In this architecture, the **Host (syz-executor)** acts as the orchestrator, while the **Guest (SYZOS)** acts as the execution engine for a pre-defined sequence of commands.
+
+### Execution Flow
+SYZOS leverages syzkaller's standard execution model, where the fuzzer generates a sequence of syscalls (a `syzlang` program) to be executed by the host executor. For SYZOS, this program constructs the VM and defines the guest's internal logic via pseudo-syscalls.
+
+1.  **VM Creation:** The fuzzer calls standard KVM ioctls (e.g., `openat`, `KVM_CREATE_VM`) to create the VM container.
+2.  **Environment Setup (`syz_kvm_setup_syzos_vm`):** This pseudo-syscall automates the complex setup of Guest memory, ensuring the VM has valid code and stack regions.
+3.  **VCPU & Program Loading (`syz_kvm_add_vcpu`, see 3.2):**
+    * Instead of a bare `KVM_CREATE_VCPU`, the fuzzer calls `syz_kvm_add_vcpu` that creates a new VCPU in the VM and initializes its state.
+        * This call takes the **entire sequence of SYZOS commands** as an argument. This sequence effectively becomes the "program" the guest will execute.
+    * **Concurrency:** SYZOS supports up to 4 separate VCPUs sharing the same address space, allowing the fuzzer to schedule concurrent guest operations.
+4.  **Execution (`KVM_RUN`):** The fuzzer triggers execution via standard `KVM_RUN` calls. The Guest executes its pre-loaded commands step-by-step.
+    * **Yielding:** When the Guest needs to perform an action that requires Host intervention (e.g., a transition during Nested Virtualization), it yields to L0 via `UEXIT`.
+    * **Resumption:** If the program contains multiple `KVM_RUN` calls, they are used to resume the Guest until the pre-loaded program completes.
+
+### Design Philosophy
+* **Logical Mutation:** Instead of fuzzing raw assembly bytes, SYZOS exposes high-level primitives to the fuzzer. The fuzzer mutates the arguments of the SYZOS commands.
+* **State Validity:** By implementing setup sequences in C, SYZOS ensures that complex structures like IRQ tables or Page Tables are valid enough to reach deep kernel code paths.
+
+---
+
+## 2. Memory Layout & ABI (Communication Interface)
+
+The Host and Guest communicate via a shared memory protocol. The Host writes commands and arguments into specific physical memory addresses, which the Guest maps and reads.
+
+### Communication Interface
+* **Command Channel:** A dedicated memory region where the Host writes the commands and their arguments.
+* **Result Channel:** Mechanism for the Guest to report status back to the Host, piggybacked on the `UEXIT` mechanism.
+* **Scratch Space:** Mutable memory used by the Guest to generate dynamic code blobs or store temporary data needed for operations like `MSR` writes.
+
+### ARM64 Memory Map
+The ARM64 implementation relies on a static physical memory layout to ensure the Host knows exactly where to place data.
+
+| Physical Address | Description | Usage |
+| :--- | :--- | :--- |
+| `0x08000000` | GIC v3 Distributor | Interaction with Generic Interrupt Controller |
+| `0x080a0000` | GIC v3 Redistributor | Per-CPU Interrupt Controller interface |
+| `0xdddd0000` - `0xeeee0000` | Read-only / Command Page | Host writes SYZOS commands here. Also used to trigger page faults for `UEXIT` |
+| `0xeeee8000` - `0xeeef0000` | Code / Scratch Space | Where SYZOS resides. Also used for generated code (e.g., MSR trampolines) |
+| `0xffff1000` | EL1 Stack | Stack space for the SYZOS Guest execution |
+
+### x86 Memory Map
+While the exact addresses may vary by implementation, the x86 layout follows similar principles:
+* **Guest Code:** Allocated via `KVM_SET_USER_MEMORY_REGION` (typically 1 page).
+* **Page Tables:** Setup by the Host to allow virtual-to-physical translation required for long mode.
+* **IDT (Interrupt Descriptor Table):** Setup by the Host to handle exceptions within the Guest.
+
+---
+
+## 3. Host-Side Implementation (`syz-executor`)
+
+The Host side is responsible for the heavy lifting of VM initialization. This is achieved through "pseudo-syscalls" - functions implemented in `syz-executor` that look like syscalls to the fuzzer but perform complex setup logic.
+
+### 3.1 VM Initialization: `syz_kvm_setup_syzos_vm()`
+This pseudo-syscall creates the VM and prepares the environment.
+* **Memory Allocation:** Calls `ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &memreg)` multiple times to map the Guest physical memory slots (Code, Stack, MMIO).
+* **Image Loading:** Copies the compiled SYZOS C library binary into the allocated Guest Code region.
+
+### 3.2 VCPU Initialization: `syz_kvm_add_vcpu()`
+This function adds a virtual CPU to the VM and configures its initial state to jump into the SYZOS entry point.
+* **Program Loading:** It parses the `syzlang`-generated argument structure, which contains the sequence of SYZOS commands, and copies them into the Guest's Command Page.
+* **Context Setup:**
+    * **x86:** Sets up SREGS (Segments, Page Tables) and IDT.
+    * **ARM64:** Sets PC to the entry point and SP to the stack.
+
+### 3.3 The Handshake Mechanism (`UEXIT`)
+The core synchronization primitive is the `UEXIT`.
+* **Trigger:** The Guest reads from a specific unmapped or read-only address (e.g., inside `0xdddd0000` on ARM) or executes a specific instruction sequence.
+* **Detection:** `KVM_RUN` returns on the Host. The Host may check `kvm_run->exit_reason`.
+* **Handling:**
+    * If the exit indicates a Page Fault (EPT violation) at the specific `UEXIT` address, the Host treats this as a voluntary yield.
+    * The Host reads the exit qualification or register state to retrieve the "return argument" passed by the Guest.
+
+---
+
+## 4. Guest-Side Implementation (SYZOS Library)
+
+### Source Organization & `GUEST_CODE`
+SYZOS guest handlers are defined directly in architecture-specific executor headers (e.g., `executor/common_kvm_amd64_syzos.h`).
+* **The `GUEST_CODE` Macro:** Functions intended to run inside the guest are marked with `GUEST_CODE` (e.g., `GUEST_CODE static void guest_handle_...`). This instructs the compiler/linker to place these functions in a specific section that `syz-executor` copies into the Guest's physical memory.
+* **Header-Based Implementation:** The entire SYZOS codebase is contained within header files included by the executor. This architecture is necessitated by `syz-prog2c`, a tool that converts `syzlang` reproducers into standalone C programs. By concatenating these headers (via `#include` expansion), `syz-prog2c` can produce a single, build-system-independent C source file that compiles anywhere without external dependencies.
+
+### The Dispatch Loop (`guest_main`)
+The entry point `guest_main` iterates through the command buffer that was populated by `syz_kvm_add_vcpu`.
+* **Command Routing (If/Else Chain):** The routing is strictly implemented as a series of `if/else if` statements rather than a `switch`. The reason for this is that a `switch` statement can be optimized by compilers into a jump table stored in the executable's `.rodata` section. Since the global data sections are not mapped into the Guest address space, accessing a jump table would cause an immediate Page Fault.
+* **Argument Parsing:** Commands are cast to specific structures (e.g., `struct api_call_5*`) to access arguments safely.
+* **Execution:** The handler performs the logic and the loop advances to the next command in the buffer.
+
+### Core Primitives
+* **`SYZOS_API_UEXIT`:** - Triggers a specific exception that the Host recognizes as a "yield". It passes a return value (1 argument) back to the Host to signal success/failure or data.
+* **`SYZOS_API_CODE`:** - Executes a raw blob of machine code supplied by the Host. This can be used to emit exact instruction sequences not covered by high-level APIs.
+
+---
+
+## 5. Platform Specifics
+
+### x86 (Intel & AMD)
+
+#### Privileged Operations
+SYZOS exposes specific APIs to fuzz privileged x86 instructions:
+* **`SYZOS_API_CPUID`:** Executes the `CPUID` instruction.
+* **`SYZOS_API_WRMSR` / `SYZOS_API_RDMSR`:** Reads/Writes Model Specific Registers.
+* **`SYZOS_API_WR_CRN` / `SYZOS_API_WR_DRN`:** Writes to Control Registers and Debug Registers.
+* **`SYZOS_API_IN_DX` / `SYZOS_API_OUT_DX`:** Executes I/O port operations.
+
+#### Nested Virtualization (NV) Engine
+SYZOS acts as a lightweight L1 hypervisor to fuzz L2 guests, abstracting the architectural differences between Intel VMX and AMD SVM. It provides a uniform API for the VM lifecycle while offering architecture-specific commands for state mutation.
+
+##### VM Lifecycle & Execution
+The following primitives control the nested guest's existence and execution flow:
+
+* **`SYZOS_API_ENABLE_NESTED`:** Enables the virtualization extensions (VMXON on Intel, EFER.SVME on AMD).
+* **`SYZOS_API_NESTED_CREATE_VM`:** Initializes the necessary control structures (VMCS for Intel, VMCB for AMD) and sets up Nested Page Tables.
+* **`SYZOS_API_NESTED_LOAD_CODE`:** Injects a sequence of instructions into the L2 guest's memory, defining what code the nested machine will execute.
+* **`SYZOS_API_NESTED_VMLAUNCH`:** Performs the initial VM Entry, transferring control to the L2 guest.
+* **`SYZOS_API_NESTED_VMRESUME`:** Resumes execution of the L2 guest after it has exited back to L1.
+
+##### State Mutation (Architecture Specific)
+To stress the host's handling of invalid or edge-case states, SYZOS allows direct mutation of the hardware control structures. This is done by applying the "set/unset/flip" mask logic: `new_val = (old_val & ~unset_mask) | set_mask ^ flip_mask`.
+The SYZOS commands are **`SYZOS_API_NESTED_INTEL_VMWRITE_MASK`** (mutates the VMCS fields on Intel) and **`SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK`** (VMCB on AMD).
+
+---
+
+### ARM64
+
+#### Device Emulation
+A significant portion of ARM64 KVM code is device emulation. SYZOS provides specialized APIs to fuzz these complex interactions.
+* **GICv3 & ITS:**
+    * **`SYZOS_API_IRQ_SETUP`:** Sets up the VGICv3 distributor and installs the guest IRQ table.
+    * **`SYZOS_API_ITS_SETUP`:** Allocates translation tables and configures the Interrupt Translation Service (ITS) base.
+    * **`SYZOS_API_ITS_SEND_CMD`:** Injects structured GIC commands (e.g., `MAPD`, `MOVI`) into the command queue.
+
+#### Hypervisor Interface
+SYZOS targets the boundary between the Guest and EL2/Firmware.
+* **Hypercalls:**
+    * **`SYZOS_API_HVC`:** Executes `hvc #0` with fuzzer-controlled parameters in registers `x0-x5`.
+    * **`SYZOS_API_SMC`:** Executes `smc #0` (Secure Monitor Call) with parameters in `x0-x5`.
+
+---
+
+## 7. Developer Guide: How to Add a New Command
+
+This guide details the process of adding a new SYZOS command, using `SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK` as a reference case.
+
+### Step 1: Define API ID and Handler Prototype
+Modify the architecture-specific executor header (e.g., `executor/common_kvm_amd64_syzos.h`) to register the new command.
+
+1.  **Add the Enum ID:** Add a new entry to the `syzos_api_id` enum.
+    ```c
+    typedef enum {
+        // ...
+        SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK = 380, // New ID
+        SYZOS_API_STOP,
+    } syzos_api_id;
+    ```
+2.  **Declare the Handler:** Add a forward declaration using the `GUEST_CODE` macro.
+    ```c
+    GUEST_CODE static void guest_handle_nested_amd_vmcb_write_mask(struct api_call_5* cmd, uint64 cpu_id);
+    ```
+
+### Step 2: Implement Guest Logic and Dispatch
+In the same file (or corresponding source), implement the guest logic.
+
+1.  **Add Dispatch Case:** Update `guest_main`.
+    ```c
+    } else if (call == SYZOS_API_NESTED_AMD_VMCB_WRITE_MASK) {
+        guest_handle_nested_amd_vmcb_write_mask((struct api_call_5*)cmd, cpu);
+    }
+    ```
+2.  **Implement Handler:** Write the function logic. Strict guest-safe code restrictions apply.
+    ```c
+    GUEST_CODE static noinline void
+    guest_handle_nested_amd_vmcb_write_mask(struct api_call_5* cmd, uint64 cpu_id)
+    {
+        if (get_cpu_vendor() != CPU_VENDOR_AMD) return;
+        // ... parse args and perform logic ...
+        vmcb_write64(vmcb_addr, offset, new_value);
+    }
+    ```
+
+### Step 3: Define syzlang Description
+Expose the new command to `syzkaller` in the description file (e.g., `sys/linux/dev_kvm_amd64.txt`).
+
+1.  **Define Structures:** Define any necessary constants or structures.
+2.  **Map Command ID:** Add the command to the `syzos_api_call` union. **Crucial:** The ID (e.g., `380`) must match the enum in the C header.
+    ```
+    syzos_api_call$x86 [
+        nested_amd_vmcb_write_mask  syzos_api$x86[380, syzos_api_nested_amd_vmcb_write_mask]
+    ] [varlen]
+    ```
+
+---
+
+## 8. Validation & Regression Testing
+
+The system includes a regression testing framework located in `sys/linux/test/`. New commands must include a test case to verify they trigger the expected Hypervisor behavior.
+
+### Test File Structure
+Tests are `syzlang` programs with special assertions.
+* **Header:** Requires metadata, e.g., `# requires: arch=amd64 -threaded`.
+* **Setup:** Standard boilerplate creates a VM and enters SYZOS.
+* **Logic:** The test configures the guest to perform a specific action (e.g., executing `HLT` in a nested L2 guest).
+
+### Assertions
+Tests use specialized pseudo-syscalls to assert the VM's exit state:
+* **`syz_kvm_assert_syzos_uexit$x86(fd, code)`:** Asserts that the guest voluntarily yielded with a specific `UEXIT` code (e.g., `0xe2e20001`).
+* **`syz_kvm_assert_syzos_kvm_exit$x86(fd, exit_reason)`:** Asserts that the guest triggered a standard KVM exit (e.g., `0x5` for `KVM_EXIT_HLT`) that was trapped by L0.
