create_container (remote REST/gRPC path) fails with generic 500 "driver stage" error while local CLI create succeeds

[hsinatfootprintai]

Summary

Creating a container via the remote API (MCP tool create_container, which goes through the daemon's REST/gRPC path) fails consistently with a generic, unhelpful error:

API error (status 500): {"code":13,"message":"create container failed (stage: driver)","details":[]}

The exact same logical operation - creating a container on the same backend, same host - succeeds immediately when run locally on the backend host via the CLI:

sudo containarium create mlops-dev-container --no-ssh-key --podman -v

✓ Container mlops-dev-container-container created successfully!

Environment

• Backend: tunnel-ubuntu-VirtualBox (a byoc backend, tunnel-connected)
• containarium doctor on that host: all checks pass (root, capabilities, writable paths, useradd/userdel probe, incus binary present)
• incus daemon itself: healthy, running for 1+ week, storage pool default (driver dir) in CREATED state
• Disk space on the host: not the cause - confirmed healthy (39% used) at the time of the failing remote attempts

Reproduction

1. Via MCP create_container tool (or equivalent REST/gRPC call), with username=mlops-dev-container, backend_id=tunnel-ubuntu-VirtualBox, cpu=4, memory=4GB, disk=50GB (no ssh_keys passed - relying on the tool's documented ephemeral-keypair-generation fallback).
2. Request fails every time with the generic 500 "driver stage" error above. Retried twice, identical failure both times.
3. SSH'd directly into the backend host and ran the equivalent containarium create <user> --no-ssh-key --podman -v locally - succeeded on the first try, no errors.

Suspicious secondary finding: local-CLI-created containers aren't registered with the platform

After the local CLI create succeeded, the container is:

• Visible and running via incus list on the host.
• Not visible via the platform's own list_containers (MCP tool) - the container simply doesn't appear in the list.
• get_container(username=mlops-dev-container) returns 404 container not found.

This suggests the REST/gRPC create path does some additional step beyond what the local CLI path does (most likely: registering the container in a central database/control-plane, and/or the SSH-key provisioning path when ssh_keys is omitted and an ephemeral keypair needs to be generated server-side). My working theory is that this extra step is exactly what's failing with the generic "driver stage" error, since the container creation itself (the actual incus/LXC provisioning) clearly works fine when done directly.

Impact

• Any create_container call without host-local CLI access is silently broken for this backend (and possibly others using the same byoc/tunnel code path).
• The generic "stage: driver" error message gives no actionable diagnostic information - it should surface the underlying cause (e.g. which specific step failed: incus provisioning itself vs. SSH key injection vs. platform registration vs. something else).

Suggested next steps

1. Reproduce with server-side debug/verbose logging enabled for the REST/gRPC create path specifically (not just the local CLI's -v flag, which apparently takes a different code path and doesn't fail).
2. Compare the two code paths (local CLI create command vs. the daemon's REST/gRPC handler for the same operation) to find what the remote path does that the local path doesn't - the divergence is likely where the bug lives.
3. Improve the error surfaced at the "driver stage" to include the underlying error message rather than just {"code":13,"message":"create container failed (stage: driver)","details":[]}.

[hsinatfootprintai]

Root-cause investigation findings

Dug into the local Containarium OSS repo source. Summary:

The exact error shape doesn't originate in this repo. {"code":13,"message":...,"details":[]} does not appear anywhere in the codebase or git history (git log --all -S"stage: driver" / -S"create container failed" both empty). This repo's own REST error handler (internal/gateway/gateway.go:801-816, customErrorHandler) always formats errors as {"error": "<msg>", "code": <http-status-int>} — different key names, no details array, and a plain HTTP status rather than a gRPC status code. The shape actually observed (numeric gRPC code 13/Internal, message, details: []) is the default, un-customized protobuf google.rpc.Status JSON that grpc-gateway emits when no custom error handler is installed on that particular surface.

Most likely explanation: the error is being generated/wrapped outside this repo, in a closed-source cloud/control-plane component. internal/cloud/rest.go's own header comment documents a separate "managed cloud (e.g. cloud.containarium.dev)" that fronts the API as REST/grpc-gateway only for BYOC hosts — consistent with the containarium-cloud MCP server being a distinct component from this repo's own internal/mcp. "stage: driver" most plausibly labels the orchestration stage where that cloud control-plane invokes the enrolled host's driver (i.e., this OSS daemon), with the daemon's real underlying error swallowed by a generic Internal wrapper on the way back.

Separate, confirmed real bug (independent of the 500 above, worth fixing regardless): when CreateContainer targets a remote/tunnel-connected backend_id, it short-circuits into peer.ForwardCreateContainer (internal/server/peer.go:627-724) before the image-allowlist/digest/podman-privilege/SSH-key validation that the local (non-forwarded) path runs at container_server.go:331-372. The hand-built forward request body (peer.go:630-647) silently drops several fields that the direct local path supports: pool, monitoring, gitSource/gitRef/gitCredential/workspacePath, ttlSeconds, idleStopMinutes, deleteAfterStoppedSeconds. This is a genuine asymmetry between "create via REST/gRPC against a tunnel-connected backend" and "CLI create directly on that host" — the latter calls container.Manager.Create (pkg/core/container/manager.go:127) directly and never goes through this peer-forward layer at all.

Ruled out as the direct cause: omitted SSH keys. Ephemeral-key generation for a keyless request happens client-side in the MCP tool wrapper (internal/mcp/tools.go:1351-1373), not server-side — the server itself handles an empty ssh_keys list gracefully (manager.go:251-276 just skips jump-server account creation with a log warning), matching why --no-ssh-key succeeded locally.

Server-side error wrapping: the sync create path (container_server.go:572-575) wraps failures as a plain fmt.Errorf(...), not status.Errorf(...) — so grpc-go would normally surface this as codes.Unknown, not codes.Internal(13), another signal the observed 500 didn't originate from this exact call site. No interceptor was found in internal/server/internal/gateway that remaps generic errors to codes.Internal or persists extra error detail beyond the existing inline log.Printf calls.

Remaining uncertainty: confirming the true origin needs either the cloud control-plane's source (not in this repo) or live daemon/journal logs captured at the moment of a failing request — neither was available for this investigation. An alternate possibility not fully ruled out: the observed shape could be a raw peer-response body nested inside peer.go:680's "peer returned status %d: %s" wrapping, which would keep the root cause inside this repo after all.

Suggested next steps

• Add status.Errorf(codes.Internal, ...) (or similar) with a specific, non-generic message at every fallible step in the create path, so a future 500 carries enough detail to diagnose without host-side debugging.
• Fix the confirmed field-dropping bug in peer.ForwardCreateContainer (peer.go:630-647) regardless of whether it's related to this specific failure.
• If a cloud/control-plane component fronts this daemon for BYOC hosts, capture/forward the daemon's real error text through that layer rather than replacing it with a generic Internal(13).

[hsinhoyeh]

Update: workaround validated under sustained real use

Since the root-cause findings above, re-tested the exact failing path (create_container via the MCP/REST tool against a tunnel-connected byoc backend) and confirmed it still reproduces identically — same {"code":13,"message":"create container failed (stage: driver)","details":[]}, same backend. Not yet fixed.

The direct-CLI workaround (SSH to the backend host, sudo containarium create <name> --no-ssh-key --podman ... run locally on the host, bypassing the REST/gRPC path entirely) has since been used to stand up and operate a container under real, sustained load — not just a smoke test:

• Created kind-dev-container this way on the same affected backend (tunnel-ubuntu-VirtualBox).
• Installed Docker + kind + kubectl inside it, stood up a full kind Kubernetes cluster, and installed Istio (istiod + a vendored ingress-gateway) on top.
• Deployed a complete multi-service application stack into that cluster (dex, an internal authz service, Postgres x2, Redis, SeaweedFS, and six application services), wired an OIDC login flow through the Istio gateway, and verified the full login round-trip (dex LDAP auth → token exchange → session cookie) end-to-end through a single local port-forward.

All of that ran entirely on a container created via the CLI workaround, with no issues attributable to the creation path itself once the container existed. This confirms the workaround is robust enough for real (not just throwaway) work, which should lower urgency for anyone blocked on this — but the underlying REST/gRPC bug is still present and worth fixing for the intended MCP-driven workflow.

One operational note for anyone else using this workaround: CLI-created containers are not registered in the platform's database (confirmed again here — list_containers/get_container never show it), so none of the MCP tools (connect, expose_port, etc.) work against them. All interaction has to go through direct incus exec/SSH to the backend host for the container's entire lifetime. This matches the secondary finding already noted above and remains the main practical downside of relying on this workaround long-term.