|
| 1 | +# wasm-git Research Analysis |
| 2 | + |
| 3 | +Research on https://github.com/petersalomonsen/wasm-git for informing our clean-room Rust Git implementation targeting wasm32-wasip1. |
| 4 | + |
| 5 | +## What is wasm-git? |
| 6 | + |
| 7 | +wasm-git compiles **libgit2** (v1.7.1, a C library) to WebAssembly using **Emscripten**. It targets browser and Node.js environments, NOT wasm32-wasip1/WASI. The output is an Emscripten WASM module (`lg2.wasm` + `lg2.js` glue) that exposes libgit2's example programs as a CLI-like interface via `callMain()`. |
| 8 | + |
| 9 | +Key distinction: wasm-git is Emscripten-based (wasm32-unknown-emscripten target), not WASI-based. This means it relies heavily on Emscripten's JavaScript glue code for I/O, filesystem, and networking. Our approach (wasm32-wasip1 targeting a WASI runtime in secure-exec) is fundamentally different. |
| 10 | + |
| 11 | +## Architecture |
| 12 | + |
| 13 | +The project wraps libgit2's `examples/` directory programs into a single binary. The main entry point (`lg2.c`) dispatches subcommands to individual example implementations. It is NOT the full git CLI -- it is a subset of git operations implemented via libgit2's C API. |
| 14 | + |
| 15 | +### Build variants |
| 16 | + |
| 17 | +Three build modes exist, all using Emscripten: |
| 18 | + |
| 19 | +1. **Sync** (default) -- Runs in Web Workers using synchronous XHR for HTTP transport. Smallest binary. |
| 20 | +2. **Async** (Asyncify) -- Uses Emscripten Asyncify to allow async JS calls from synchronous C code. Doubles binary size due to stack unwinding/rewinding instrumentation. Can run on main thread. |
| 21 | +3. **OPFS** -- Uses Emscripten's WASMFS with Origin Private File System backend. Runs in Web Worker with pthreads. No Asyncify needed. |
| 22 | + |
| 23 | +### Build flags of interest |
| 24 | + |
| 25 | +``` |
| 26 | +-DREGEX_BACKEND=regcomp # Uses POSIX regcomp, not PCRE |
| 27 | +-DUSE_HTTPS=OFF # No HTTPS -- HTTP only in transport |
| 28 | +-DUSE_SSH=OFF # No SSH transport |
| 29 | +-DTHREADSAFE=OFF # Single-threaded |
| 30 | +-DBUILD_SHARED_LIBS=OFF # Static linking |
| 31 | +-s ALLOW_MEMORY_GROWTH=1 # Dynamic memory |
| 32 | +-s STACK_SIZE=131072 # 128KB stack |
| 33 | +``` |
| 34 | + |
| 35 | +## Supported Git Operations |
| 36 | + |
| 37 | +From `lg2.c` command table (these are the libgit2 example programs, some patched): |
| 38 | + |
| 39 | +| Command | Patched? | Notes | |
| 40 | +|---------|----------|-------| |
| 41 | +| add | Yes | Supports -v, -n, -u flags | |
| 42 | +| blame | No | Stock libgit2 example | |
| 43 | +| cat-file | No | Stock | |
| 44 | +| checkout | Yes | Branch switching, path checkout, -b flag, remote tracking setup | |
| 45 | +| clone | No | Stock, but HTTP-only (no SSH, no HTTPS) | |
| 46 | +| commit | Yes | Supports -m flag, merge commits, signature from .gitconfig | |
| 47 | +| config | No | Stock | |
| 48 | +| describe | No | Stock | |
| 49 | +| diff | No | Stock | |
| 50 | +| fetch | No | Stock | |
| 51 | +| for-each-ref | No | Stock | |
| 52 | +| general | No | Stock | |
| 53 | +| index-pack | No | Stock | |
| 54 | +| init | No | Stock | |
| 55 | +| log | No | Stock | |
| 56 | +| ls-files | No | Stock | |
| 57 | +| ls-remote | No | Stock | |
| 58 | +| merge | No | Stock | |
| 59 | +| push | Yes | Custom implementation, pushes HEAD to origin | |
| 60 | +| remote | No | Stock | |
| 61 | +| reset | Yes | Supports --hard, --soft | |
| 62 | +| revert | Yes | Custom implementation | |
| 63 | +| rev-list | No | Stock | |
| 64 | +| rev-parse | No | Stock | |
| 65 | +| show-index | No | Stock | |
| 66 | +| stash | Yes | push/pop/apply/list/drop | |
| 67 | +| status | Yes | Long/short/porcelain format, ahead/behind, conflict display | |
| 68 | +| tag | No | Stock | |
| 69 | + |
| 70 | +**Notable absences**: No `branch` command (branching done via checkout -b), no `rm`, no `rebase`, no `cherry-pick`, no `bisect`, no `submodule` management, no `worktree`. |
| 71 | + |
| 72 | +## Special Adaptations |
| 73 | + |
| 74 | +### 1. HTTP Transport (the biggest adaptation) |
| 75 | + |
| 76 | +libgit2's standard HTTP transport (`http.c`) is **completely replaced** with a custom Emscripten HTTP transport. The original `http.c` is deleted in `setup.sh` and replaced with `emscriptenhttp.c`. |
| 77 | + |
| 78 | +The transport implements libgit2's `git_smart_subtransport` interface and delegates actual HTTP to JavaScript: |
| 79 | + |
| 80 | +- **Sync version**: Uses `EM_ASM_INT` to call into `Module.emscriptenhttpconnect/read/write` JS functions. In browser, these use synchronous `XMLHttpRequest`. In Node.js, they use `worker_threads` with `SharedArrayBuffer` + `Atomics` for synchronous cross-thread HTTP. |
| 81 | +- **Async version**: Uses `EM_JS` with `Asyncify.handleAsync()` to wrap async fetch calls. Browser uses async XHR; Node.js uses same SharedArrayBuffer approach. |
| 82 | + |
| 83 | +**Key insight for our implementation**: Git's HTTP smart protocol uses 4 endpoints: |
| 84 | +- `/info/refs?service=git-upload-pack` (fetch discovery) |
| 85 | +- `/git-upload-pack` (fetch data) |
| 86 | +- `/info/refs?service=git-receive-pack` (push discovery) |
| 87 | +- `/git-receive-pack` (push data) |
| 88 | + |
| 89 | +These are POST requests with specific content types (`application/x-git-upload-pack-request`, etc.). Any WASI implementation needs to be able to make these HTTP requests through the host. |
| 90 | + |
| 91 | +### 2. Filesystem Layer |
| 92 | + |
| 93 | +wasm-git does NOT implement its own filesystem. It uses Emscripten's built-in FS backends: |
| 94 | + |
| 95 | +- **MEMFS**: In-memory, not persisted. Default. |
| 96 | +- **IDBFS**: IndexedDB-backed. Requires explicit `FS.syncfs()` calls to persist. |
| 97 | +- **NODEFS**: Pass-through to host Node.js filesystem. |
| 98 | +- **WASMFS + OPFS**: Emscripten's newer filesystem with Origin Private File System backend. |
| 99 | + |
| 100 | +**Key insight for our implementation**: For wasm32-wasip1, the filesystem is provided by the WASI runtime (our secure-exec kernel VFS). We don't need to implement any of this -- WASI fd_read/fd_write/path_open etc. will be handled by the kernel. This is a major simplification compared to wasm-git's approach. |
| 101 | + |
| 102 | +### 3. File Permission Patches |
| 103 | + |
| 104 | +Two patches in `setup.sh` change file modes: |
| 105 | +```c |
| 106 | +// pack.h: GIT_PACK_FILE_MODE 0444 -> 0644 |
| 107 | +// odb.h: GIT_OBJECT_FILE_MODE 0444 -> 0644 |
| 108 | +``` |
| 109 | + |
| 110 | +libgit2 creates pack files and object files as read-only (0444). Emscripten's FS doesn't handle this well -- once a file is created read-only, it can't be modified or deleted. Changing to 0644 allows normal read-write access. |
| 111 | + |
| 112 | +**Key insight for our implementation**: Our VFS supports chmod properly, so we may not need this workaround. However, if our VFS has any issues with read-only files being subsequently opened for write (e.g., during repacking), we'd hit the same issue. |
| 113 | + |
| 114 | +### 4. Integer Overflow Intrinsics |
| 115 | + |
| 116 | +`integer.h` is patched to add an Emscripten-specific case for `size_t` overflow detection: |
| 117 | +```c |
| 118 | +#if defined(__EMSCRIPTEN__) |
| 119 | + // Emscripten/WebAssembly: size_t is unsigned long |
| 120 | + #define git__add_sizet_overflow(out, one, two) __builtin_uaddl_overflow(one, two, out) |
| 121 | + #define git__multiply_sizet_overflow(out, one, two) __builtin_umull_overflow(one, two, out) |
| 122 | +``` |
| 123 | +
|
| 124 | +In Emscripten's WASM target, `size_t` is `unsigned long` (32-bit), but the existing code only matched `UINT_MAX` or `ULONG_MAX` cases that didn't apply to Emscripten's type configuration. |
| 125 | +
|
| 126 | +**Key insight for our implementation**: In wasm32-wasip1, `size_t` is 32-bit. If using libgit2's C code (we're not -- we're doing Rust), this would matter. In Rust, integer overflow is handled by the language. |
| 127 | +
|
| 128 | +### 5. C Standard Compatibility |
| 129 | +
|
| 130 | +```bash |
| 131 | +echo 'set(CMAKE_C90_STANDARD_COMPILE_OPTION "-std=gnu90")' >> CMakeLists.txt |
| 132 | +``` |
| 133 | + |
| 134 | +Forces GNU C90 standard across all libgit2 compilation units for Emscripten compatibility. |
| 135 | + |
| 136 | +### 6. WASMFS getcwd() Bug Workaround |
| 137 | + |
| 138 | +The OPFS variant has a significant workaround for a WASMFS bug where `getcwd()` returns wrong paths for directories backed by a different backend than the root. The workaround creates symlinks at the root so broken paths still resolve: |
| 139 | + |
| 140 | +```javascript |
| 141 | +// getcwd() returns '//repo' instead of '/opfs/repo' |
| 142 | +FS.symlink(workingDir + '/' + repoName, '/' + repoName); |
| 143 | +``` |
| 144 | + |
| 145 | +Additionally, CWD must be re-set before each `callMain()` call because WASMFS may reset it. |
| 146 | + |
| 147 | +### 7. chmod Workaround (now removed) |
| 148 | + |
| 149 | +The async build previously had a workaround for libgit2 calling chmod with only `S_IFREG` set (permissions 0000). This was later found to be unnecessary and removed. |
| 150 | + |
| 151 | +## Emscripten Fixes Required |
| 152 | + |
| 153 | +Four PRs to Emscripten itself were needed (all merged by 2020-03-29): |
| 154 | + |
| 155 | +1. **emscripten#10095**: Unknown (PR not detailed in README) |
| 156 | +2. **emscripten#10526**: Unknown |
| 157 | +3. **emscripten#10782**: Unknown |
| 158 | +4. **emscripten#10669**: Needed for NODEFS support |
| 159 | + |
| 160 | +These were all early-stage fixes for edge cases in Emscripten's WASM compilation and filesystem layer. |
| 161 | + |
| 162 | +## Gotchas and Lessons Learned |
| 163 | + |
| 164 | +### For any WASM git implementation: |
| 165 | + |
| 166 | +1. **No SSH transport** -- Only HTTP smart protocol is supported. SSH would require a TCP socket + SSH client implementation, which is extremely complex in WASM. |
| 167 | + |
| 168 | +2. **No HTTPS** -- wasm-git builds with `USE_HTTPS=OFF`. In browser, the browser handles TLS for XHR/fetch. In our case, the secure-exec network adapter handles TLS on the host side, so this isn't an issue. |
| 169 | + |
| 170 | +3. **No native credential handling** -- Authentication is done by the HTTP layer (basic auth in URLs or custom headers). There's no SSH key or credential helper support. |
| 171 | + |
| 172 | +4. **File permissions matter** -- libgit2 creates pack/object files as read-only. If the VFS doesn't support chmod properly, or if re-opening read-only files for write fails, git operations will break. |
| 173 | + |
| 174 | +5. **Single-threaded** -- libgit2 is compiled with `THREADSAFE=OFF`. This is fine for WASM. |
| 175 | + |
| 176 | +6. **Memory growth** -- Git operations (especially clone) can use significant memory. `ALLOW_MEMORY_GROWTH=1` is essential. |
| 177 | + |
| 178 | +7. **Stack size** -- 128KB stack is used. Git operations with deep directory trees or large diffs could potentially overflow this. |
| 179 | + |
| 180 | +8. **The "examples" are the interface** -- wasm-git doesn't wrap the full libgit2 API. It uses the example programs which are simplified implementations. Some operations (like push) have very limited option support. |
| 181 | + |
| 182 | +### For our Rust + wasm32-wasip1 approach: |
| 183 | + |
| 184 | +1. **We have a fundamental advantage**: WASI provides a proper POSIX-like filesystem interface. We don't need to implement FS backends or deal with Emscripten's FS quirks. The kernel VFS handles everything. |
| 185 | + |
| 186 | +2. **Networking is the hard part**: The git smart HTTP protocol needs HTTP client support. In WASI, we'd need to either: |
| 187 | + - Use WASI socket APIs (if available in our runtime) |
| 188 | + - Implement a custom host function for HTTP requests |
| 189 | + - Use the secure-exec kernel's network adapter |
| 190 | + |
| 191 | +3. **Consider using gitoxide (gix)**: Instead of compiling libgit2 to WASM, a pure Rust git implementation like [gitoxide](https://github.com/Byron/gitoxide) could potentially compile to wasm32-wasip1 more cleanly, since it's already Rust and doesn't have C dependencies (except for optional features). |
| 192 | + |
| 193 | +4. **Alternatively, git2-rs**: The Rust bindings for libgit2 could work, but would require cross-compiling libgit2's C code to wasm32-wasip1 (not Emscripten), which is a different challenge. Issue #77 on wasm-git discusses this but no one has done it. |
| 194 | + |
| 195 | +5. **Consider scope carefully**: wasm-git supports ~27 commands but many are stock libgit2 examples with limited options. For coding agents, the critical operations are: clone, status, add, commit, diff, log, checkout, branch, push, pull (fetch+merge), stash, reset. This is a tractable subset. |
0 commit comments