docs: clarify context rules and separation of concerns. (#77)

Author: jglogan

README.md

@@ -31,24 +31,74 @@
 
 flowchart LR
   classDef soft fill:#EFF6FF,stroke:#2563EB,stroke-width:2px,rx:8,ry:8,color:#111827;
-  client["container build ..."]
+  client["container build ...\n(macOS host)"]
   subgraph boundary["Builder Container"]
     direction LR
     shim["container‑builder-shim"]
     buildkit["buildkitd"]
-    shim -- "Buildkit API (gRPC)" --> buildkit
+    shim -- "BuildKit API (gRPC)" --> buildkit
   end
 
-  client -- BuilderAPI --> shim
+  client -- "Builder API" --> shim
 
 ```
 
-1. Buildkit initiates a session via gRPC.
+1. BuildKit initiates a session via gRPC.
 2. container-builder-shim intercepts session requests (file sync, image resolution, etc.).
-3. Requests are translated to containerization's Build API format.
-4. containerization processes the build.
+3. Requests are translated to containerization's Build API format and forwarded to the macOS host over a bidirectional gRPC stream.
+4. The macOS host processes the request (serving context files, resolving images, etc.) and streams the response back.
 5. Build output and metadata flow back through container-builder-shim to BuildKit.
 
+## Build Context Transfer
+
+Build context files flow from the macOS host to BuildKit through a three-tier pipeline. Each tier has distinct responsibilities; understanding the split is important when working on file-transfer or security-related code.
+
+### Responsibilities
+
+| Tier | Responsibility |
+|---|---|
+| **macOS host** (`container` / `BuildFSSync`) | Owns the context directory. Enforces the context boundary: rejects any request that resolves outside the root. Packs requested files into a tar archive. Does **not** apply `.dockerignore`. |
+| **container-builder-shim** (`pkg/fssync`) | Bridges the host's wire format and BuildKit's `filesync` gRPC interface. Receives the tar, unpacks it to a content-addressed local cache, applies `.dockerignore` exclusions, and presents the result to BuildKit via `DiffCopy`. |
+| **BuildKit** | Owns all Dockerfile copy semantics: when to dereference symlinks, how to recurse directories, and how `.dockerignore` patterns are interpreted. Drives the transfer by sending `Walk` requests with `followpaths` and `exclude-patterns`. |
+
+### Primary data flow
+
+```
+BuildKit ──► shim Walk (followpaths, exclude-patterns)
+         ──► host: resolve globs, pack matching paths into tar
+         ◄── tar stream
+shim unpacks tar to content-addressed cache, applies .dockerignore
+BuildKit ◄── DiffCopy PACKET_STAT   (one entry per file/dir/symlink)
+BuildKit ──► PACKET_REQ             (for each regular file it needs)
+BuildKit ◄── PACKET_DATA            (shim reads from local cache)
+```
+
+There is also a fallback path (`Info` → `Read`) used by `FS.Open` when the local cache is unpopulated — a narrow race window at the start of a build. The host enforces the same boundary rules on this path.
+
+### Context boundary rules
+
+These rules govern how files are selected, transferred, and presented. They reflect standard Docker build semantics and are enforced jointly by the host and the shim.
+
+1. **Context boundary.** All files served to BuildKit must physically reside within the context root on the macOS host. Any request that resolves outside the root — whether by direct path or symlink chain — is rejected by the host.
+
+2. **Out-of-context symlinks.** A symlink whose target lies outside the context root is present in the tar as a symlink entry, but its target is not included. BuildKit will fail to dereference it during `COPY`/`ADD` because the target is absent from the unpacked context. Symlinks in a build context should refer only to paths within the context root.
+
+3. **In-context file symlink.** The host includes both the symlink entry and its target file in the tar. BuildKit dereferences the symlink during `COPY`/`ADD`; the result in the image is a regular file containing the target's content, not a symlink.
+
+4. **In-context directory symlink.** The host follows the symlink and includes the directory's contents in the tar, applying rule 1 to every entry discovered through it. BuildKit recurses the directory during `COPY`/`ADD` as if it were a plain directory.
+
+5. **Dangling symlink.** A symlink that is lexically within the context root but whose target cannot be resolved causes the target to be absent from the tar; BuildKit fails at `COPY`/`ADD` time.
+
+### `.dockerignore`
+
+`.dockerignore` filtering is the shim's responsibility, not the host's. After unpacking the tar, the shim walks the cache directory and applies the `exclude-patterns` received from BuildKit before emitting `PACKET_STAT` entries. The host has no knowledge of `.dockerignore`.
+
+### `followpaths`
+
+`followpaths` is a comma-separated list of glob patterns identifying which context paths BuildKit needs for the current build step. The shim forwards it to the host, which resolves the patterns against the context root to determine which files to pack into the tar.
+
+If BuildKit does not supply `followpaths`, the shim falls back to `addedGlobs`: source paths pre-computed by scanning the Dockerfile AST for `COPY`, `ADD`, and `RUN --mount=type=bind` instructions (see `pkg/build/buildopts.go`).
+
 ## Contributing
 
 Contributions to Containerization are welcomed and encouraged. Please see our [main contributing guide](https://github.com/apple/containerization/blob/main/CONTRIBUTING.md) for more information.

pkg/build/buildopts.go

@@ -317,8 +317,10 @@ func NewBuildOpts(ctx context.Context, basePath string, contextMap map[string][]
 		return nil, err
 	}
 
-	// do add .dockerignore support
-	// to ExcludePatterns(dockerignore) patterns
+	// addedGlobs is the fallback value for followpaths when BuildKit does not
+	// supply it. Pre-compute it by scanning the Dockerfile AST for COPY, ADD,
+	// and RUN --mount=type=bind source paths so the host packs only the files
+	// those instructions need rather than the entire context.
 	addedGlobs := []string{}
 	for _, node := range dockerfile.AST.Children {
 		if strings.EqualFold(node.Value, "COPY") || strings.EqualFold(node.Value, "ADD") {

pkg/fileutils/tarxfer.go

@@ -31,8 +31,16 @@ import (
 
 const DockerfileStaging = ".com.apple.container"
 
-// Receiver streams a remote tar archive, caches it under cacheBase and calls fn
-// for every entry that is a regular file, directory or symlink.
+// Receiver streams a tar archive from the macOS host, stores it in a
+// content-addressed cache under cacheBase, unpacks it, and walks the result.
+//
+// Symlinks are unpacked as real OS symlinks (os.Symlink). filepath.Walk does
+// not follow them, so they appear in the walked tree with their Linkname
+// intact. BuildKit decides at COPY/ADD time whether to dereference them based
+// on its own copy semantics.
+//
+// If the cache directory for the tar's content hash already exists the
+// download is skipped and the cached tree is used directly.
 type Receiver struct {
 	demux     *stream.Demultiplexer
 	cacheBase string

pkg/fssync/diffcopy.go

@@ -32,8 +32,12 @@ import (
 	"golang.org/x/sync/errgroup"
 )
 
-// DiffCopy from fsutil pkg uses a smaller buffer size. This implementation
-// is essentially the same except for larger buffers
+// DiffCopy handles BuildKit's primary file-transfer path.
+//
+// It calls Walk to request a tar of the build context from the host, then
+// concurrently serves PACKET_REQ requests from BuildKit by reading files from
+// the local unpacked cache. The buffer size is larger than fsutil's default
+// to reduce syscall overhead for large files.
 func (f *FSSyncProxy) DiffCopy(ss filesync.FileSync_DiffCopyServer) error {
 	ctx := ss.Context()
 	fs := NewFS(ctx, f, f.contextDir, f.basePath)

pkg/fssync/fssync.go

@@ -39,8 +39,14 @@ var (
 	_ filesync.FileSyncServer = &FSSyncProxy{}
 )
 
-// A fsSync that proxies requests over the bidirectional grpc stream
-// It is used by buildkit to retrieve context directory and other build artifacts
+// FSSyncProxy implements BuildKit's filesync.FileSyncServer by proxying
+// file requests over the bidirectional gRPC stream to the macOS host.
+//
+// BuildKit drives the transfer by calling DiffCopy, which delegates to Walk
+// and FS.Open. Walk is the primary path: it sends a followpaths request to
+// the host, receives a tar archive, unpacks it to a content-addressed local
+// cache, and presents the result to BuildKit. FS.Open falls back to direct
+// Info/Read host calls only when the cache is unpopulated.
 type FSSyncProxy struct {
 	stream.UnimplementedBaseStage
 	filesync.UnimplementedFileSyncServer

pkg/fssync/walk.go

@@ -32,9 +32,19 @@ import (
 )
 
 /*
-Walk is proxied over the gRPC stream to the caller.
+Walk requests build-context files from the macOS host and presents them to BuildKit.
 
-In JSON mode, the server sends file info as JSON; we walk those files directly.
+The host is asked for a tar archive containing the paths identified by
+followpaths (glob patterns BuildKit sends in the request metadata). The shim
+unpacks the tar to a content-addressed local cache and then walks the unpacked
+tree, filtering each entry through the exclude-patterns (from .dockerignore)
+before passing it to fn.
+
+Only TAR mode is supported. The JSON mode wire format is defined in
+RawFileInfo below but is not exercised by the current shim.
+
+If BuildKit does not supply followpaths, the shim falls back to addedGlobs —
+source paths pre-computed from the Dockerfile AST (see pkg/build/buildopts.go).
 
 Request Format: