Bug: --use-buildkit doesn't bypass Engine-API pre-check — unusable without a local Docker daemon

[royassis]

Summary

PR #8772 added --use-buildkit to shell out to the docker/finch CLI instead of using the Python Docker SDK. This is great for CI setups with a remote BuildKit endpoint (no local dockerd). However, ContainerClientFactory.create_client() still runs unconditionally and performs an Engine-API socket ping via the Docker SDK before --use-buildkit is evaluated. If that ping fails, the build exits before the BuildKit code path is reached — so --use-buildkit is effectively unreachable for its main target use case.

Related: #8649, whose maintainer comment foreshadowed this:

We are working on supporting BuildKit by allowing the CLI to be called directly when a parameter --use-buildkit is provided, meaning soon enough using the CLI instead of the SDK would be possible.

The SDK call is still the first thing that runs.

Environment

• SAM CLI 1.159.1 (post-feat: add cli client support for image builds to enable buildkit #8772)
• Python 3.11.10, Linux (container), x86_64
• docker CLI 29.4.2 (no local dockerd), buildx 0.33.0 with remote driver → tcp://localhost:1234
• moby/buildkit:v0.29.0 sidecar on tcp://localhost:1234

Reproducer

1. Container with docker CLI + buildx, no dockerd.
2. buildkitd sidecar on tcp://localhost:1234.
3. docker buildx create --name remote --driver remote tcp://localhost:1234 && docker buildx use remote
4. export DOCKER_HOST=tcp://localhost:1234 (required so docker buildx picks up the endpoint).
5. Minimal template with one PackageType: Image function + trivial Dockerfile.
6. sam build --use-buildkit (or --no-use-container, or neither).

Direct docker buildx build ... against the same endpoint succeeds.

Expected

With --use-buildkit, SAM shells out to docker (or finch). docker buildx handles the remote endpoint natively. Build succeeds.

Actual

Error: Running AWS SAM projects locally requires a container runtime.
Do you have Docker or Finch installed and running?

Debug log (sam build --use-buildkit --debug):

ContainerClientFactory.create_client() called
Trying Docker client creation
Creating Docker container client from environment variable.
Fall back docker api version to 1.44: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Docker daemon availability check failed with fallback: ('Connection aborted.', ...)
Docker client not created, trying creating Finch client.
Creating Finch container client with base_url=unix:///var/run/finch.sock
Finch daemon availability check failed: ('Connection aborted.', FileNotFoundError(2, ...))
No container runtime available

Telemetry: containerEngine: tcp-local, exitReason: ContainerNotReachableException.

Suggested fix

When --use-buildkit is set (or use_buildkit = true in samconfig.toml), skip the SDK-based Engine-API ping. Delegate all runtime availability checks to the CLI invocation itself — it will fail fast with a clearer error if the tooling isn't there.

Alternatively: if --use-buildkit implies "I have docker/finch on PATH," check for the CLI binary instead of pinging the Engine API socket.

Workaround

Run a docker:dind sidecar in the pod, point DOCKER_HOST=tcp://localhost:2375 only for sam build. Works, but duplicates the image-build stack (privileged DinD plus the existing BuildKit) just to satisfy the pre-check.

[reedham-aws]

Hello @royassis, thank you for raising this issue and continued support for this feature. This kind of falls under the category of a bug and a feature request. It should definitely be possible to do this.

The existing code was designed without the possibility of remote builds in mind, and it's just creating the container client as a means of checking what engine should be used (docker or finch). It's not as straight forward as the suggested fix because the real reason for creating the client is not just the engine ping, but detecting whether docker or finch should be used. That being said, this should be possible and I will try to come up with a way around this.

Edit: I guess we could make a tradeoff that when --use-buildkit is specified we only look at whether the CLIs are installed for deciding which one to use. I had originally thought not to do this because theoretically the CLIs can be used cross-engine, but in reality the current code doesn't even let you do that (which, as I recall, was also intentional). I will attempt this solution.

[reedham-aws]

One roadblock to this approach is that in a non-remote build case, CLI availability != engine should be used. For example, in my local, I have the docker CLI, but not the actual engine running. What it should see is that I have the finch engine running and use that instead, regardless of CLI availability. It might just need to be that this is a tradeoff we must accept for remote builds to succeed.

[royassis]

Follow-up on a related-but-distinct performance issue I hit with the same remote-BuildKit setup described in this issue.

This comment was investigated and drafted by an AI coding agent (Claude) working with me as operator. The conclusions below are the agent's interpretations; the measurements are from real CI runs.

Symptom

Once the pre-check from this issue was worked around (DinD sidecar just to answer SAM's Engine-API ping), sam build --use-buildkit runs — but a fully cache-hit rebuild still takes ~2 minutes per function in CI. The exact same sam build on a local Docker host finishes in ~2 seconds.

Cause

samcli/local/docker/image_build_client.py:186 hard-codes --load on every docker buildx build invocation:

if self.engine_type == "docker":
    cmd.extend(["--provenance=false", "--sbom=false", "--load"])

With a local docker buildx driver, --load is a no-op because BuildKit is the daemon. With a buildx remote driver (BuildKit running as a separate server on a Kubernetes StatefulSet), --load forces BuildKit to serialize every layer into a Docker-format tarball and stream it back over gRPC to the client's dockerd — so sam package's subsequent docker push can find the image.

That serialization is CPU-bound (gzip + framing), not network-bound. It happens even when every layer is CACHED.

Evidence

Warm-cache CI rebuild, 5 Lambda functions, BuildKit on a StatefulSet in the same cluster:

#10 CACHED
#11 CACHED
#12 CACHED
#13 CACHED
#14 CACHED
#15 CACHED
#17 exporting to docker image format
#17 DONE 54.2s    ← smaller function
#17 DONE 55.5s
#17 DONE 97.2s    ← heavy deps (scipy, numba, tiledbsoma)
#17 DONE 97.9s
#17 DONE 100.7s

All 6 layer steps CACHED — the 2-minute wall clock is entirely #17 exporting to docker image format (gzip-and-stream per layer per function).

A/B test: doubling buildkitd CPU from 2→8 moved #17 times ~15–20% (52s → 34s on the smaller function). That scaling signature means CPU-bound work inside BuildKit, not network. We didn't decompose gzip-vs-I/O-vs-framing numerically.

For comparison, docker buildx build --push (driver=remote, type=registry output) against the same BuildKit finishes in a few seconds because BuildKit pushes straight to ECR without materializing a Docker tarball.

Feature request

Expose a way to skip --load when the user is going to push anyway:

1. A --push flag on sam build that substitutes --load with --push and resolves the image URI upfront (e.g., from --image-repositories or a naming convention). sam package becomes a no-op for image functions.
2. Or, more generally, expose the buildx --output so advanced users can select type=registry,push=true,name=... instead of --load.

Either eliminates the tarball serialization entirely for the "build and ship to ECR" path that most image-packaged Lambdas follow.

AI-disclosure nitpicks (what's verified vs. estimated)

• --load is hard-coded: verified in source (link above).
• #17 is CPU-bound inside BuildKit: supported by the 2→8 CPU A/B scaling, not directly instrumented.
• Network cost is negligible: supported by cluster topology (pod-to-pod within EKS, no egress), not timed independently.
• "Gzip is ~60-70s of the 80s per function": the agent's estimate, not measured.

Happy to run follow-up tests if useful.

[royassis]

Update with better data. (Still AI-agent-driven investigation.)

After the previous comment, I instrumented pod-level CPU during a live build and found my earlier "gzip on the BuildKit side is the bottleneck" framing was wrong in the specifics. The real picture:

Per-container CPU during build-sam-artifact (128s stage, 2 CPU limit on dind)

time    buildkitd-0  dind       builder  stage
0-15s   idle         idle       3000m    (setup, layer CACHED resolution)
15-45s  2315m (peak) 833m       3000m    (#17 exporting starts on BuildKit)
45-105s idle         2003m      idle     (dind receiving + gunzipping — pegged at its 2 CPU limit)
105-128s idle        2000m      idle     (still unpacking)

So the #17 exporting to docker image format step actually splits into two CPU-bound phases:

• BuildKit side (~30s): gzip, serialize tarball, stream over gRPC. Scales with buildkitd CPU but typically not saturating.
• DinD side (~60–90s): gunzip incoming tarball, overlayfs snapshot creation. Single-threaded per layer but parallel across layers, so it scales with CPU cores on the receiver.

In our setup, dind was CPU-limit-capped at 2000m for the entire receive phase while buildkitd had headroom. The receiver was the bottleneck, not the sender.

Bump dind 2→8 CPU, same build

build-sam-artifact:  128s → 58s      (-55%)
dind peak:           2003m → 6129m   (was saturated, now using ~3× more)
buildkitd-0 peak:    2315m → 1967m   (same, never pegged)

Takeaway for the report-request above

The --load workflow problem is worse than just "gzip on BuildKit" — it moves ~gigabytes of layer tarball data across the gRPC connection and makes the Docker daemon do the full docker load work on every cache-hit rebuild. Removing --load via a --push / --output=type=registry,push=true option would skip:

1. BuildKit tarball serialization
2. gRPC stream
3. Docker daemon docker load (the actually-pegged step)

In CI where the image is going to ECR anyway, all three are pure overhead.

What's verified vs. estimated (updated)

• --load is hard-coded in image_build_client.py:186: ✅ verified
• DinD is the primary CPU bottleneck during #17: ✅ verified via kubectl top sampling
• ~50% wall-clock improvement from 2→8 CPU on dind: ✅ measured (128s → 58s)
• Removing --load entirely would eliminate this: claim — well-supported by the architecture but not measured directly, since SAM doesn't currently allow it