|
| 1 | +Performance testing |
| 2 | +=================== |
| 3 | + |
| 4 | +SwiftTerm has three levels of performance measurement, from fastest to most |
| 5 | +realistic: |
| 6 | + |
| 7 | +1. **Headless feed benchmarks** — measure the terminal-emulation engine |
| 8 | + (parser + buffer) with no rendering. |
| 9 | +2. **RenderBench** — a deterministic harness that drives the real |
| 10 | + `TerminalView` render path with synthetic workloads. This is the primary |
| 11 | + tool for render-path work and for Instruments profiling. |
| 12 | +3. **In-app measurement** — vtebench or timed `cat` runs typed into the |
| 13 | + MacTerminal sample app, over a real PTY and shell. |
| 14 | + |
| 15 | +When comparing two revisions, build the second revision in a git worktree so |
| 16 | +both binaries exist at once: |
| 17 | + |
| 18 | +```bash |
| 19 | +git worktree add /tmp/swiftterm-main main |
| 20 | +# ... build the same harness in both checkouts, run them back to back ... |
| 21 | +git worktree remove --force /tmp/swiftterm-main |
| 22 | +``` |
| 23 | + |
| 24 | +1. Headless feed benchmarks |
| 25 | +--------------------------- |
| 26 | + |
| 27 | +The tests live in `Tests/SwiftTermTests/PerformanceTest.swift`. They feed |
| 28 | +byte streams into a `HeadlessTerminal` for a fixed duration and print |
| 29 | +throughput in calls/second. Release mode is required for meaningful numbers, |
| 30 | +and `@testable import` in release needs `-enable-testing`: |
| 31 | + |
| 32 | +```bash |
| 33 | +swift test -c release -Xswiftc -enable-testing --filter "PerformaceTests/testPerformance2" |
| 34 | +``` |
| 35 | + |
| 36 | +Run each test individually with `--filter` — Swift Testing runs tests |
| 37 | +concurrently by default, which corrupts throughput measurements. |
| 38 | + |
| 39 | +Two of the tests need external data files and silently skip when absent: |
| 40 | + |
| 41 | +- `repeatBigBlob` / `measureBigBlogFeed` read `~/cvs/vtebench/x`, generated |
| 42 | + with [vtebench](https://github.com/alacritty/vtebench): |
| 43 | + `target/release/vtebench --max-samples 1 -b benchmarks/medium_cells/` |
| 44 | +- `repeatDataFile` reads `~/data-file` (any large terminal capture). |
| 45 | + |
| 46 | +Duration-based tests complete a whole number of iterations, so a 10-second |
| 47 | +test that finishes ~13 iterations has ±7% quantization — treat differences |
| 48 | +smaller than that as noise. |
| 49 | + |
| 50 | +2. RenderBench (render path, Instruments) |
| 51 | +----------------------------------------- |
| 52 | + |
| 53 | +`Tools/RenderBench` is a small SPM executable that hosts a real |
| 54 | +`TerminalView` in an on-screen window and feeds it synthetic frames as fast |
| 55 | +as the main run loop accepts them — no PTY, no shell, byte-identical input on |
| 56 | +every run (fixed seed), so two builds are directly comparable. |
| 57 | + |
| 58 | +```bash |
| 59 | +cd Tools/RenderBench |
| 60 | +swift build -c release |
| 61 | +.build/release/RenderBench --seconds 10 --scenario dense |
| 62 | +``` |
| 63 | + |
| 64 | +It prints MB/s and frames/s every second and a `TOTAL` line at the end. |
| 65 | + |
| 66 | +Options: |
| 67 | + |
| 68 | +- `--scenario dense` — every cell gets its own truecolor foreground and |
| 69 | + background (vtebench dense_cells shape; stresses attribute handling, run |
| 70 | + fragmentation, and color conversion) |
| 71 | +- `--scenario medium` — a color change every 8 cells (longer runs) |
| 72 | +- `--scenario scroll` — plain scrolling ASCII (parser + scroll + full-screen |
| 73 | + redraw) |
| 74 | +- `--scenario arabic` — scrolling Arabic words (BiDi paragraph analysis, |
| 75 | + shaping, font fallback) |
| 76 | +- `--seconds N` — run duration (default 15) |
| 77 | +- `--metal` — use the Metal renderer instead of CoreGraphics |
| 78 | + |
| 79 | +The package pins its dependency identity (`.package(name: "SwiftTerm", |
| 80 | +path: "../..")`), so it also builds inside a worktree whose directory is not |
| 81 | +named `SwiftTerm` — copy `Tools/RenderBench` into the worktree if the |
| 82 | +revision under test predates it. |
| 83 | + |
| 84 | +### Profiling with Instruments |
| 85 | + |
| 86 | +```bash |
| 87 | +cd Tools/RenderBench |
| 88 | +swift build -c release |
| 89 | +xcrun xctrace record --template 'Time Profiler' --output ~/dense.trace \ |
| 90 | + --launch -- .build/release/RenderBench --seconds 20 --scenario dense |
| 91 | +open ~/dense.trace |
| 92 | +``` |
| 93 | + |
| 94 | +Each `feed` call is wrapped in an os_signpost (subsystem |
| 95 | +`org.tirania.SwiftTerm`, category `RenderBench`), so adding the os_signpost |
| 96 | +instrument splits main-thread time between the feed/parse side and the |
| 97 | +AppKit draw cycles. For A/B analysis, record the same scenario from both |
| 98 | +checkouts and diff the heaviest stacks under `buildAttributedString` and the |
| 99 | +draw loop. |
| 100 | + |
| 101 | +3. In-app measurement |
| 102 | +--------------------- |
| 103 | + |
| 104 | +For end-to-end numbers over a real PTY, build the sample app in Release |
| 105 | +(Debug builds SwiftTerm at `-Onone` and exaggerates Swift-level costs): |
| 106 | + |
| 107 | +```bash |
| 108 | +cd TerminalApp |
| 109 | +xcodebuild -project MacTerminal.xcodeproj -scheme MacTerminal \ |
| 110 | + -configuration Release -derivedDataPath /tmp/dd build |
| 111 | +``` |
| 112 | + |
| 113 | +Then, inside the running terminal window, run vtebench: |
| 114 | + |
| 115 | +```bash |
| 116 | +vtebench -b benchmarks/dense_cells --max-secs 6 --dat /tmp/results.dat |
| 117 | +``` |
| 118 | + |
| 119 | +The `.dat` file has one column per benchmark with per-sample times in ms; |
| 120 | +more samples completed in the fixed time budget = faster. The app defaults |
| 121 | +to the CoreGraphics renderer; flip `setUseMetal(false)` to `true` in |
| 122 | +`TerminalApp/MacTerminal/ViewController.swift` to measure Metal (and revert |
| 123 | +afterwards). Keep the window size identical between runs — cols × rows |
| 124 | +changes the per-frame workload. |
| 125 | + |
| 126 | +Methodology notes |
| 127 | +----------------- |
| 128 | + |
| 129 | +- **Pair your A/B runs.** Absolute numbers drift between sessions (thermal |
| 130 | + state, display state, background load). Run main and the branch back to |
| 131 | + back in the same block, and re-run any surprising result before believing |
| 132 | + it — a transient machine state can halve one configuration's numbers for |
| 133 | + minutes at a time while others look normal. |
| 134 | +- **Interpret cat/PTY timings carefully.** `time cat file` inside a terminal |
| 135 | + measures how fast the terminal drains the PTY; payloads under a few MB fit |
| 136 | + in kernel buffering and undercount. Use payloads of 10 MB+. |
| 137 | +- **vtebench sample distributions are bimodal** (fast PTY-buffered samples |
| 138 | + next to render-synced ones); compare sample counts and means, not medians, |
| 139 | + and treat differences under ~10% as noise. |
| 140 | +- **What each scenario is sensitive to:** `dense` regresses when per-cell or |
| 141 | + per-run work is added to attribute handling (dictionary copies, bridging, |
| 142 | + color conversion); `scroll` when scroll/feed or full-screen redraw gets |
| 143 | + slower; `arabic` when BiDi paragraph analysis, shaping, or font fallback |
| 144 | + gets slower. A change that only moves `arabic` costs RTL users only; a |
| 145 | + change that moves `dense`/`scroll` costs everyone. |
0 commit comments