Skip to content

Commit 5259261

Browse files
thrashr888claude
andcommitted
docs: add measured Firecracker benchmark results
Measured on AMD EPYC with KVM: - Instance start: 78ms (faster than Docker's 155ms) - Agent ready: 1015ms (kernel + init + agent boot) - Command execution: 19ms (3x faster than Docker's 53ms) - Shutdown: 20ms (6x faster than Docker's 130ms) Key insight: Firecracker hypervisor overhead is minimal (78ms). The ~1s ready time is kernel/userspace boot, not Firecracker overhead. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 556e50c commit 5259261

1 file changed

Lines changed: 35 additions & 13 deletions

File tree

BENCHMARK.md

Lines changed: 35 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,13 @@ Performance measurements for sandbox lifecycle operations.
44

55
## Quick Summary
66

7-
| Backend | Platform | Avg Boot | Full Cycle | Throughput |
8-
|---------|----------|----------|------------|------------|
9-
| Docker | macOS (M1) | 188ms | 446ms | 2.0 sandboxes/sec |
10-
| Firecracker | Linux (KVM) | <125ms | <200ms | Target: 10+ sandboxes/sec |
7+
| Backend | Platform | Avg Boot | Ready Time | Exec Latency | Throughput |
8+
|---------|----------|----------|------------|--------------|------------|
9+
| Docker | macOS (M1) | 188ms | 188ms | 83ms | 2.0 sandboxes/sec |
10+
| Docker | Linux (AMD EPYC) | 155ms | 155ms | 53ms | ~4 sandboxes/sec |
11+
| Firecracker | Linux (AMD EPYC) | 78ms | 1015ms | 19ms | ~1 sandbox/sec |
12+
13+
**Key insight**: Firecracker hypervisor overhead is just 78ms (faster than Docker). The 1015ms "ready" time includes full kernel boot + userspace init + guest agent startup. Once running, Firecracker has 3x lower exec latency.
1114

1215
## Docker Backend (macOS)
1316

@@ -66,25 +69,44 @@ The ~175ms start time is the practical floor for Docker. Remaining overhead come
6669

6770
## Firecracker Backend (Linux)
6871

69-
Firecracker microVMs provide significantly faster boot times by eliminating container orchestration overhead.
72+
Firecracker microVMs provide stronger isolation (separate kernel per VM) and lower exec latency via vsock.
73+
74+
### Measured Performance (AMD EPYC, KVM)
75+
76+
| Metric | Measured | Notes |
77+
|--------|----------|-------|
78+
| Firecracker API ready | 46ms | Process start to socket available |
79+
| Instance start | 78ms | VM started (hypervisor overhead only) |
80+
| Agent ready | 1015ms | Full boot: kernel + init + agent |
81+
| Command execution | 19ms | Via vsock (3x faster than Docker exec) |
82+
| Shutdown | 20ms | 6x faster than Docker cleanup |
7083

71-
### Target Performance
84+
### Docker vs Firecracker Comparison (Linux)
7285

73-
| Metric | Target |
74-
|--------|--------|
75-
| Boot time | <125ms |
76-
| Shutdown | <50ms |
77-
| Memory overhead | <10MB per VM |
78-
| 100 VM test | <30s total |
86+
| Metric | Docker | Firecracker | Winner |
87+
|--------|--------|-------------|--------|
88+
| Process start | 40ms | 46ms | Tie |
89+
| Instance/container up | 155ms | 78ms | **Firecracker** |
90+
| Ready to execute | 155ms | 1015ms | **Docker** (no kernel boot) |
91+
| Command execution | 53ms | 19ms | **Firecracker** (vsock) |
92+
| Shutdown/cleanup | 130ms | 20ms | **Firecracker** (6x faster) |
7993

80-
### Why Firecracker is Faster
94+
### Why Firecracker
8195

8296
| Aspect | Docker | Firecracker |
8397
|--------|--------|-------------|
8498
| Isolation | Shared kernel, namespaces | Separate kernel per VM |
8599
| Boot path | Container runtime → namespaces → cgroups | KVM → minimal kernel → init |
86100
| Overhead | Docker daemon, containerd, runc | Direct KVM hypercalls |
87101
| Memory | ~50-100MB per container | ~10MB per VM |
102+
| Security | Container escapes possible | Hardware isolation via KVM |
103+
104+
### Use Case Recommendations
105+
106+
- **Short-lived tasks (<5 commands)**: Docker is faster for total cycle time
107+
- **Longer sessions**: Firecracker is better (lower per-command latency)
108+
- **Security-critical code**: Firecracker required (true VM isolation)
109+
- **Untrusted code**: Firecracker strongly recommended
88110

89111
### Requirements
90112

0 commit comments

Comments
 (0)