additional_contexts fan-in hits 4 MB gRPC limit at small N — duplicating tree serialization

[davireis]

additional_contexts fan-in hits 4 MB gRPC limit at small N — duplicating tree serialization

Summary

When a build target has many additional_contexts: <name>: service:<svc> entries (or the bake equivalent contexts: <name>: target:<X>), the dockerfile frontend's Inputs() response serializes each input as an independent pb.Definition tree. Inputs that share sub-graphs duplicate those sub-graph bytes once per reference. With realistic-size Dockerfiles, this crosses the 4 MB gRPC limit at ~15-20 cross-referenced services.

The error surfaces as the recurring:

failed to get frontend inputs: ResourceExhausted: grpc: received message larger than max (X vs. 4194304)

Same root cause as the closed-then-reopened thread: #1539 → #4948 → #5217 → still recurring at higher N in #6097, docker/buildx#2208, docker/buildx#3084. Each got patched by raising the limit; the underlying duplication wasn't addressed.

Repro

A DAG of N projects, each p_j referencing p_0..p_{j-1} via additional_contexts: ref_p_i: service:p_i. Dockerfiles include enough source-map content to make per-input Definitions non-trivial. Reproduces on Server 29.2.1, buildx 0.31.1, buildkit 0.27.1.

mkdir -p /tmp/buildkit-repro && cd /tmp/buildkit-repro
N=20
rm -rf p* compose.yaml
for i in $(seq 0 $((N-1))); do
    mkdir -p p${i}
    {
        echo "# syntax=docker/dockerfile:1.7-labs"
        for k in $(seq 1 100); do
            echo "# project p${i} comment $k filler-content-to-bulk-source-map-padding"
        done
        echo "FROM alpine:3.20 AS stage"
        echo "WORKDIR /work"
        if [ "$i" -gt 0 ]; then
            for j in $(seq 0 $((i-1))); do
                echo "COPY --from=ref_p${j} /work/marker /work/marker_p${j}"
            done
        fi
        echo "RUN echo p${i} > /work/marker"
    } > p${i}/Dockerfile
done
{
    echo "services:"
    for i in $(seq 0 $((N-1))); do
        echo "  p${i}:"
        echo "    build:"
        echo "      context: ./p${i}"
        echo "      dockerfile: Dockerfile"
        if [ "$i" -gt 0 ]; then
            echo "      additional_contexts:"
            for j in $(seq 0 $((i-1))); do
                echo "        ref_p${j}: service:p${j}"
            done
        fi
    done
} > compose.yaml

docker compose build "p$((N-1))"

Observed:

target p<X>: failed to solve: ResourceExhausted: failed to get frontend inputs: ResourceExhausted: grpc: received message larger than max (4938907 vs. 4194304)

Expected: builds the requested target.

Why the bytes blow up

The error originates at bc.client.Inputs(ctx) in frontend/dockerfile/builder/build.go (and frontend/dockerui/context.go). Inputs() returns map[string]*pb.Definition, where each pb.Definition.Def is a self-contained flat [][]byte of serialized Ops.

When N inputs reference shared upstream sub-graphs (typical when consumers all derive from a few common base targets), each input's Definition re-serializes the shared Op bytes. Buildkit's solver dedups by op-digest on receipt, so the wire bytes are the only place the duplication exists.

Per-input Definitions for realistic Dockerfiles (with COPY/RUN/cache mounts/build args and source-map data) run 10-50 KB; with overlap factor 3-5×, ~15-20 inputs crosses 4 MB. As monorepos grow this gets re-hit at ever-higher absolute thresholds.

Two paths

Easy: enable gRPC compression on the gateway connection

The duplicated Op bytes across inputs are highly compressible (10-20× for typical workloads). gRPC supports per-call compression negotiation; both ends already use the same grpc-go lib.

Server side (frontend/gateway/gateway.go):

import _ "google.golang.org/grpc/encoding/gzip"  // registers the gzip compressor
// (server already accepts compressed requests once gzip is registered)

Client side (frontend/gateway/grpcclient/client.go):

dialOpts := []grpc.DialOption{
    // ...existing options...
    grpc.WithDefaultCallOptions(grpc.UseCompressor("gzip")),
}

Backward-compatible: gRPC negotiates per-RPC; older peers get uncompressed messages. ~5-10 lines including imports. Lifts the practical ceiling by ~10× without changing the protocol shape.

Real fix: protocol that represents the DAG, not the tree

The current Inputs() shape — map[string]*pb.Definition with each Definition a self-contained tree — is the wrong representation when inputs share sub-graphs. The graph is naturally a DAG (op-digest-addressed); the protocol forces a tree-per-input expansion that duplicates shared nodes.

A new gateway capability with a shared-Op response shape — one Op pool plus per-input root-digest references — eliminates the duplication at the wire level, matching what buildkit's solver already does internally on receipt. The dockerfile frontend's c.Inputs() API can stay the same (client-side reassembly into the existing map[string]*pb.Definition), so existing frontend code is untouched.

This is the structurally correct fix; gRPC compression is the bandage. Both could land — compression first for immediate relief, DAG protocol later when there's appetite for the coordinated change.

Environment

• Docker Desktop, Server 29.2.1, BuildKit v0.27.1, buildx v0.31.1-desktop.1
• macOS arm64 (linux/arm64 native containers)
• Reproduces on both docker and docker-container drivers