feat(daemon): cap concurrent sandboxes per profile

A profile bounds what ONE sandbox consumes — CPU, memory, tmpfs, PIDs, lifetime.
Nothing bounded how many exist. N sandboxes each sitting at their ceiling is N
times that ceiling of committed host memory, reached without any single request
violating a policy: a retry storm, a leaked handle, or a reaper that stops
reaping exhausts the host while the daemon does exactly what it was told to.
Concurrent count is the one resource dimension the profile model did not cover.

Adds `max_sandboxes` to a profile. Unset means unlimited, so existing
deployments are unchanged, and a negative value is rejected at load rather than
read as unlimited — "the cap silently did not apply" is the one outcome this
setting exists to prevent.

A refused create returns 429 with a new `at_capacity` error kind, distinct from
`invalid`: the request is well formed and may well succeed later, once the
reaper or an explicit Destroy frees a slot. A caller that cannot tell those
apart either retries a malformed request forever or gives up on a temporary
refusal. brokerclient maps it back through the existing ErrorFor path, so
errors.Is against ErrAtCapacity works the same as against the library.

Three things worth reviewing rather than skimming:

The count comes from the backend, not from a number this process keeps.
Sandboxes survive a daemon restart, and an in-process counter would not — it
would drift into permitting more than the cap on exactly the reboot where that
mattered. Reading the backend also means the reaper releases capacity for free:
an idle sweep, a max_age expiry or a Destroy frees a slot with no extra
bookkeeping.

Checking a live count is not enough on its own. Concurrent requests all read the
same pre-create count and all pass, which is precisely the retry storm the cap
exists for, so the check has to be atomic with taking the slot: a reservation is
held from the check until Create returns. The reservation is released only after
Create returns, so a created sandbox is already visible to the count before it
stops being counted as pending — the instant of overlap errs towards refusing
one request too many rather than admitting one too many.

An existing name does not consume a slot. Backend.Create returns the live
sandbox when the name is taken, which is how a caller reuses its own warm
sandbox; charging that against the cap would starve the callers already inside
it, and only once the host was busy enough for the cap to bind.

Stopped sandboxes do count. They hold no memory right now, but Create restarts
one under the same name, so excluding them would let a caller walk past the cap
by reviving what the reaper has not yet collected.

Capacity is per profile rather than global. A per-profile number is easier to
reason about and composes into a global one later; starting global makes the
per-profile case harder to express. Profiles with different memory ceilings also
want different counts.

Not addressed here, and deliberately: per-caller quotas. The daemon has no
caller identity — socket group membership is the entire access control list and
every connection on it is anonymous and mutually indistinguishable — so a
per-caller quota depends on first establishing identity, which is a larger and
separable piece of work. A per-profile cap needs none of it, and still bounds
total host exposure, which is the property actually at risk.

Sizing guidance in the reference config and docs/security.md says to plan
against memory_mb x max_sandboxes WITH headroom, because memory_mb is enforced
approximately: roughly 1.4x the configured figure has been observed resident
before the kill lands.

Closes #28
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Lutherwaves

CHANGELOG.md

@@ -30,6 +30,12 @@ Releases are cut automatically from [Conventional Commits](https://www.conventio
   server-side from configuration alone.
 - `pkg/brokerclient`: a drop-in `sandbox.Backend` that talks to `openbloxd`
   over its Unix socket, satisfying the same contract the Docker backend does.
+- `openbloxd`: `max_sandboxes` per profile, bounding how many sandboxes exist
+  at once — the one resource dimension a profile did not otherwise cover.
+  Exceeding it returns `429` with the new `at_capacity` error kind
+  (`brokerapi.ErrAtCapacity`), distinct from a malformed request because the
+  request is valid and may succeed once the reaper frees a slot. Unset means
+  unlimited, so existing deployments are unchanged.
 
 ### Changed
 

deploy/openbloxd.example.yaml

@@ -21,6 +21,16 @@ profiles:
     memory_mb: 2048
     disk_mb: 1024           # tmpfs, drawn from memory; must not exceed it.
     max_processes: 256      # Without this a fork bomb exhausts host PIDs.
+    # How many sandboxes this profile may hold at once. Every other bound here
+    # limits ONE sandbox; this is the only one that limits how many exist, and
+    # without it a caller looping on create exhausts host memory without ever
+    # violating a policy. Omit for unlimited.
+    #
+    # Size it as memory_mb x this number, with headroom: the runtime enforces
+    # memory_mb approximately, and roughly 1.4x the configured figure has been
+    # observed resident before the kill lands. A refused create returns 429 with
+    # kind "at_capacity", so a caller can tell it from a malformed request.
+    max_sandboxes: 32
     idle_timeout: 30m
     max_age: 4h
     default_timeout: 60s
@@ -36,5 +46,9 @@ profiles:
     memory_mb: 4096
     disk_mb: 2048
     max_processes: 256
+    # Counted per profile, not globally: browser sandboxes cost more memory
+    # each, so they get their own smaller ceiling rather than competing for
+    # code-exec's.
+    max_sandboxes: 8
     idle_timeout: 30m
     max_age: 4h

docs/security.md

@@ -54,6 +54,13 @@ Non-root by default (`1000:1000`), read-only root filesystem, and only `/tmp`,
 CPU, memory, scratch disk and PID count are capped per sandbox. Scratch is tmpfs and is
 drawn from the memory budget, so a sandbox cannot fill the host's disk by writing files.
 
+Those bound one sandbox. `openbloxd` additionally bounds how many exist at once, per
+profile, through `max_sandboxes` — without it a caller looping on create exhausts host
+memory while every individual sandbox stays inside its policy. Size a host against
+`memory_mb` × `max_sandboxes` **with headroom**: the runtime enforces `memory_mb`
+approximately rather than exactly, and roughly 1.4× the configured figure has been
+observed resident before the kill lands.
+
 ### Bounded lifetime
 
 Every sandbox has an idle timeout and a max age, enforced by a reaper.

internal/daemon/capacity.go

@@ -0,0 +1,94 @@
+package daemon
+
+import (
+	"context"
+	"fmt"
+	"sync"
+
+	"github.com/blox-eng/openblox/pkg/brokerapi"
+)
+
+// capacity bounds how many sandboxes a profile may hold at once.
+//
+// A profile bounds what ONE sandbox consumes. Nothing else bounds how many
+// exist, and N sandboxes each sitting at their ceiling is N times that ceiling
+// of committed host memory — reached without any single request violating a
+// policy. Concurrent count is the one resource dimension the profile model does
+// not otherwise cover.
+//
+// The count comes from the backend rather than from a number this process
+// keeps, because the backend is the authority: sandboxes survive a daemon
+// restart, and a counter that did not would drift into permitting more than the
+// cap on exactly the reboot where it mattered. It also means the reaper
+// releases capacity for free — an idle sweep, a max_age expiry or an explicit
+// Destroy frees a slot with no extra bookkeeping.
+type capacity struct {
+	mu sync.Mutex
+	// pending counts reservations held by requests that have passed the check
+	// but whose Create has not returned yet. Without it, several concurrent
+	// requests all read the same pre-create count and all pass — a retry storm
+	// is precisely the case the cap exists for, so the check has to be atomic
+	// with taking the slot.
+	pending map[string]int
+}
+
+// reserve takes a slot for profile, or reports ErrAtCapacity. The returned
+// release must be called once the create attempt has finished, whether or not
+// it succeeded.
+//
+// A limit of zero means unlimited, so a deployment that never configures one is
+// unchanged. In that case there is nothing to serialise and no backend call to
+// make.
+func (c *capacity) reserve(ctx context.Context, s *Server, profile string, limit int) (release func(), err error) {
+	if limit <= 0 {
+		return func() {}, nil
+	}
+
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	live, err := s.countLive(ctx, profile)
+	if err != nil {
+		return nil, err
+	}
+	if live+c.pending[profile] >= limit {
+		return nil, fmt.Errorf("%w: profile %q holds %d of %d sandboxes",
+			brokerapi.ErrAtCapacity, profile, live+c.pending[profile], limit)
+	}
+
+	if c.pending == nil {
+		c.pending = map[string]int{}
+	}
+	c.pending[profile]++
+	return func() {
+		c.mu.Lock()
+		defer c.mu.Unlock()
+		// Released only after Create has returned, so a sandbox that was
+		// created is already visible to countLive before it stops being
+		// counted here. The overlap double-counts for an instant, which errs
+		// towards refusing one request too many rather than admitting one too
+		// many.
+		if c.pending[profile]--; c.pending[profile] <= 0 {
+			delete(c.pending, profile)
+		}
+	}, nil
+}
+
+// countLive reports how many sandboxes currently exist under profile.
+//
+// Stopped sandboxes count. They hold no memory right now, but Backend.Create
+// restarts one under the same name, so excluding them would let a caller walk
+// past the cap simply by reviving what the reaper has not yet collected.
+func (s *Server) countLive(ctx context.Context, profile string) (int, error) {
+	all, err := s.backend.List(ctx)
+	if err != nil {
+		return 0, err
+	}
+	n := 0
+	for _, i := range all {
+		if i.Labels[labelProfile] == profile {
+			n++
+		}
+	}
+	return n, nil
+}

internal/daemon/capacity_test.go

@@ -0,0 +1,142 @@
+package daemon
+
+import (
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/blox-eng/openblox/pkg/brokerapi"
+)
+
+// newCappedServer builds a server whose code-exec profile admits at most limit
+// concurrent sandboxes, with `existing` already live under the named profiles.
+func newCappedServer(t *testing.T, limit int, existing map[string]string) *Server {
+	t.Helper()
+	cfg := &Config{
+		Socket: "/tmp/unused.sock",
+		Profiles: map[string]Profile{
+			"code-exec": {Image: "example.com/i@sha256:abc", Runtime: "runsc", MaxSandboxes: limit},
+			"browser":   {Image: "example.com/b@sha256:def"},
+		},
+	}
+	return New(&fakeBackend{existing: existing}, cfg)
+}
+
+func createSandbox(srv *Server, name, profile string) *httptest.ResponseRecorder {
+	rec := httptest.NewRecorder()
+	req := httptest.NewRequest(http.MethodPost, "/sandboxes",
+		strings.NewReader(fmt.Sprintf(`{"name":%q,"profile":%q}`, name, profile)))
+	srv.Handler().ServeHTTP(rec, req)
+	return rec
+}
+
+func TestLoadRejectsNegativeMaxSandboxes(t *testing.T) {
+	path := writeConfig(t, `
+socket: /tmp/s.sock
+profiles:
+  p:
+    image: example.com/i@sha256:abc
+    max_sandboxes: -1
+`)
+	_, err := Load(path)
+	if err == nil {
+		t.Fatal("expected an error: a negative cap reads as 'unlimited' but is an operator typo")
+	}
+	if !strings.Contains(err.Error(), "max_sandboxes") {
+		t.Errorf("error %q should name the offending field", err)
+	}
+}
+
+func TestCreateRefusesWhenProfileIsAtCapacity(t *testing.T) {
+	srv := newCappedServer(t, 2, map[string]string{"a": "code-exec", "b": "code-exec"})
+
+	rec := createSandbox(srv, "c", "code-exec")
+
+	if rec.Code != http.StatusTooManyRequests {
+		t.Fatalf("status = %d, want 429, body %s", rec.Code, rec.Body.String())
+	}
+	var body struct {
+		Kind string `json:"kind"`
+	}
+	if err := json.NewDecoder(rec.Body).Decode(&body); err != nil {
+		t.Fatal(err)
+	}
+	if body.Kind != brokerapi.KindAtCapacity {
+		t.Errorf("kind = %q, want %q — a caller must tell 'try later' from 'malformed'",
+			body.Kind, brokerapi.KindAtCapacity)
+	}
+}
+
+func TestCreateAtCapacityStillServesAnExistingSandbox(t *testing.T) {
+	srv := newCappedServer(t, 2, map[string]string{"a": "code-exec", "b": "code-exec"})
+
+	// "a" is already counted against the cap, so reusing it consumes no new
+	// slot. Refusing here would break warm reuse exactly when the host is busy.
+	rec := createSandbox(srv, "a", "code-exec")
+
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("status = %d, want 201, body %s", rec.Code, rec.Body.String())
+	}
+}
+
+func TestCreateCapacityCountsOnlyTheRequestedProfile(t *testing.T) {
+	srv := newCappedServer(t, 2, map[string]string{"x": "browser", "y": "browser"})
+
+	rec := createSandbox(srv, "a", "code-exec")
+
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("status = %d, want 201 — browser sandboxes must not consume code-exec capacity, body %s",
+			rec.Code, rec.Body.String())
+	}
+}
+
+func TestCreateIsUnlimitedWhenMaxSandboxesIsUnset(t *testing.T) {
+	srv := newCappedServer(t, 0, map[string]string{"a": "code-exec", "b": "code-exec", "c": "code-exec"})
+
+	rec := createSandbox(srv, "d", "code-exec")
+
+	if rec.Code != http.StatusCreated {
+		t.Fatalf("status = %d, want 201 — unset max_sandboxes means unlimited, body %s",
+			rec.Code, rec.Body.String())
+	}
+}
+
+// TestConcurrentCreatesCannotExceedCapacity is the point of the feature: a
+// retry storm arrives in parallel, and counting live sandboxes without holding
+// a reservation would let every request pass the check before any of them had
+// created anything.
+func TestConcurrentCreatesCannotExceedCapacity(t *testing.T) {
+	srv := newCappedServer(t, 1, nil)
+	fake := srv.backend.(*fakeBackend)
+
+	start := make(chan struct{})
+	fake.createBlock = start
+
+	const callers = 8
+	codes := make(chan int, callers)
+	for i := range callers {
+		go func() { codes <- createSandbox(srv, fmt.Sprintf("s%d", i), "code-exec").Code }()
+	}
+	close(start)
+
+	var created, refused int
+	for range callers {
+		switch code := <-codes; code {
+		case http.StatusCreated:
+			created++
+		case http.StatusTooManyRequests:
+			refused++
+		default:
+			t.Errorf("unexpected status %d", code)
+		}
+	}
+	if created != 1 {
+		t.Errorf("created = %d, want 1 — the cap must hold under concurrent creates", created)
+	}
+	if refused != callers-1 {
+		t.Errorf("refused = %d, want %d", refused, callers-1)
+	}
+}

internal/daemon/config.go

@@ -33,6 +33,7 @@ type Profile struct {
 	MemoryMB       int64         `yaml:"memory_mb"`
 	DiskMB         int64         `yaml:"disk_mb"`
 	MaxProcesses   int           `yaml:"max_processes"`
+	MaxSandboxes   int           `yaml:"max_sandboxes"`
 	IdleTimeout    time.Duration `yaml:"idle_timeout"`
 	MaxAge         time.Duration `yaml:"max_age"`
 	DefaultTimeout time.Duration `yaml:"default_timeout"`
@@ -145,6 +146,12 @@ func (p Profile) validate(name string) error {
 	if p.MaxProcesses < 0 {
 		return fmt.Errorf("%w: profile %q has negative max_processes %d", sandbox.ErrInvalid, name, p.MaxProcesses)
 	}
+	// Zero means unlimited, which is what an unset cap has always meant. A
+	// negative value would read the same way while being a typo, and "the cap
+	// silently did not apply" is the one outcome this setting exists to prevent.
+	if p.MaxSandboxes < 0 {
+		return fmt.Errorf("%w: profile %q has negative max_sandboxes %d; omit it for unlimited", sandbox.ErrInvalid, name, p.MaxSandboxes)
+	}
 	if p.IdleTimeout < 0 {
 		return fmt.Errorf("%w: profile %q has negative idle_timeout %s, which silently disables reaping", sandbox.ErrInvalid, name, p.IdleTimeout)
 	}

internal/daemon/sandboxes.go

@@ -33,17 +33,33 @@ func (s *Server) handleCreate(w http.ResponseWriter, r *http.Request) {
 	// be handed back as though it satisfied this request. Backend.Create returns
 	// the live sandbox when the name exists, so the check has to happen here,
 	// ahead of that call.
+	var exists bool
 	if existing, err := s.backend.Open(r.Context(), req.Name); err == nil {
 		if got := existing.Info().Labels[labelProfile]; got != req.Profile {
 			fail(w, fmt.Errorf("%w: %q exists under profile %q, requested %q",
 				brokerapi.ErrProfileConflict, req.Name, got, req.Profile))
 			return
 		}
+		exists = true
 	} else if !errors.Is(err, sandbox.ErrNotFound) {
 		fail(w, err)
 		return
 	}
 
+	// Only a NEW name consumes a slot. A sandbox that already exists is already
+	// counted, so re-creating it — the session-affinity path a caller takes to
+	// reuse its own warm sandbox — must not be refused. Charging it again would
+	// starve exactly the callers already inside the cap, and only once the host
+	// was busy enough for the cap to bind.
+	if !exists {
+		release, err := s.cap.reserve(r.Context(), s, req.Profile, profile.MaxSandboxes)
+		if err != nil {
+			fail(w, err)
+			return
+		}
+		defer release()
+	}
+
 	// Profile options first, caller options second — and the caller's are only
 	// ever env and labels. Nothing here can reach Runtime or Egress.
 	opts := profile.Options()