docs+infra(dev-container): recommend dev container path; make Dockerf… (#49)

* docs+infra(dev-container): recommend dev container path; make Dockerfile.dev multi-arch

The dev container is the supported install path because GCO pins many
exact Python package versions and host installs frequently fail with
dependency-resolver errors. Update the docs to reflect that, and fix
two things that were undermining the recommendation.
Documentation
- README, QUICKSTART, CONTRIBUTING now lead with `docker build -f
  Dockerfile.dev` and demote pip/pipx to a collapsed "advanced" path
  with explicit guidance on the resolver-conflict failure mode.
- New "Installation Issues" section in docs/TROUBLESHOOTING.md walks
  through `ResolutionImpossible` errors and points at the dev container
  as the canonical fix.
- mcp/README.md and QUICKSTART.md now use absolute-path-in-args for
  the MCP config so the same snippet works in Cursor, Kiro, Claude
  Desktop, etc. without relying on client-specific `cwd` handling.
Dockerfile.dev — multi-arch
- Add ARG TARGETARCH and derive a GNU arch tag (x86_64 / aarch64) for
  the AWS CLI v2 and Docker static-client tarballs; pass TARGETARCH
  straight through for the kubectl URL.
- Builds natively on linux/amd64 (CI on x86_64 GitHub runners, Intel
  Macs, Linux x86_64 hosts) and linux/arm64 (Apple Silicon, Graviton,
  arm64 Linux). Native arm64 build on Apple Silicon drops from ~7-8
  min (qemu emulation under Finch/Colima) to ~2 min — matching the
  build-time claim in the docs.
- All baked-in deps verified to install from wheels on arm64 (no sdist
  fallback for any pinned package).
CI — integration:docker:dev-container
- New "Verify amd64 image actually contains amd64 binaries" step:
  asserts uname -m and AWS CLI v2 banner agree on x86_64.
- New QEMU + buildx setup followed by linux/arm64 cross-build, then
  triple-layer arch verification: uname -m == aarch64, AWS CLI banner
  contains aarch64, gco --version loads, and an inline Python ELF
  e_machine inspection of kubectl + the Docker static client to catch
  single-binary regressions that would slip past a run-time check
  under QEMU translation.
- Bumped job timeout 25→40 min to absorb the cross-build (~10× slower
  under qemu).

* updates

* Update integration-tests.yml

Author: Jmevorach

.github/workflows/integration-tests.yml

@@ -7,7 +7,7 @@
 # Triggers: push: main, pull_request, workflow_dispatch.
 #
 # Jobs (alphabetical by display name):
-#   - integration:docker:dev-container       — Dockerfile.dev + `gco --help`
+#   - integration:docker:dev-container       — Dockerfile.dev + `gco --help` + multi-arch (amd64 native, arm64 via QEMU)
 #   - integration:docker:health-monitor      — build + /healthz + /readyz
 #   - integration:docker:helm-installer      — build + `helm version`/`kubectl version`
 #   - integration:docker:inference-monitor   — build + import-based liveness
@@ -360,10 +360,17 @@ jobs:
   integration-docker-dev-container:
     name: "integration:docker:dev-container"
     runs-on: ubuntu-latest
-    timeout-minutes: 25
+    # Bumped from 25→40 to accommodate the linux/arm64 cross-build, which
+    # runs every Dockerfile.dev RUN step under QEMU emulation on the
+    # amd64 runner (vs. ~20 min for the native amd64 build on a standard
+    # 2-vCPU runner — pip-install of all the pinned deps dominates).
+    timeout-minutes: 40
     steps:
       - uses: actions/checkout@v6
       - name: Build dev container
+        # Native amd64 build on the x86_64 GitHub runner — exercises the
+        # default code path operators hit on Intel/Linux and Docker Desktop
+        # / x86_64. The arm64 path is exercised in a later step.
         run: docker build -f Dockerfile.dev -t gco-dev .
       - name: Exercise CLI inside dev container
         run: |
@@ -383,6 +390,19 @@ jobs:
             aws --version
             docker --version
           '
+      - name: Verify amd64 image actually contains amd64 binaries
+        # The Dockerfile.dev is multi-arch via $TARGETARCH (see its header).
+        # On this x86_64 runner the default build above should produce an
+        # amd64 image — confirm the GNU-arch derivation picked the right
+        # tarballs by checking what the binaries themselves report.
+        run: |
+          set -euo pipefail
+          arch=$(docker run --rm gco-dev uname -m)
+          [ "$arch" = "x86_64" ] || { echo "Expected x86_64, got: $arch"; exit 1; }
+          # AWS CLI v2 prints its build identifier (e.g. "exe/x86_64.debian.13")
+          # in --version output — the arch tag here proves we downloaded the
+          # right tarball, not just that the binary happens to run.
+          docker run --rm gco-dev aws --version | grep -q "x86_64\|amd64"
       - name: Verify Docker-in-Docker via host socket pass-through
         # `gco stacks deploy-all` drives `cdk deploy`, which shells out to
         # Docker to bundle Lambda assets. The dev container ships only the
@@ -421,6 +441,85 @@ jobs:
           # Clean up the probe image so this job leaves the runner tidy.
           docker image rm dind-probe:ci alpine:3.21 >/dev/null 2>&1 || true
 
+      # ----------------------------------------------------------------------
+      # Multi-arch cross-build verification.
+      #
+      # Dockerfile.dev is parameterized on BuildKit's $TARGETARCH so the
+      # image builds natively on both linux/amd64 (this runner, CI default)
+      # and linux/arm64 (Apple Silicon, Graviton, arm64 Linux). The amd64
+      # path is fully exercised above. The steps below cross-build the
+      # arm64 image under QEMU emulation and confirm:
+      #
+      #   1. The arm64 build succeeds (catches Dockerfile regressions where
+      #      a tool URL gets pinned back to amd64 or the GNU-arch derivation
+      #      breaks).
+      #   2. The resulting image actually contains aarch64 binaries (catches
+      #      "wrong tarball downloaded but binary happens to run under
+      #      emulation" type bugs).
+      #   3. Python dependencies still install from wheels on arm64 — i.e.
+      #      no pinned package has silently become amd64-only since the
+      #      lockfile was regenerated.
+      #
+      # Cross-build under QEMU is meaningfully slower than the native
+      # amd64 build above; if this step becomes a bottleneck, consider
+      # gating it on `paths` for Dockerfile.dev and pyproject.toml only.
+      # ----------------------------------------------------------------------
+      - name: Set up QEMU (for linux/arm64 emulation)
+        uses: docker/setup-qemu-action@v4
+        with:
+          platforms: arm64
+      - name: Set up Buildx (enables --platform)
+        uses: docker/setup-buildx-action@v4
+      - name: Cross-build dev container for linux/arm64
+        run: |
+          docker buildx build \
+            --platform linux/arm64 \
+            --load \
+            -f Dockerfile.dev \
+            -t gco-dev:arm64 \
+            .
+      - name: Verify arm64 image contains native aarch64 binaries
+        run: |
+          set -euo pipefail
+          # Under QEMU's binfmt translation, /bin/bash's `uname -m` reports
+          # the binary's expected arch, not the host kernel's. A mismatched
+          # /bin/bash here would mean the image was tagged arm64 but the
+          # base layers actually pulled amd64.
+          arch=$(docker run --rm gco-dev:arm64 uname -m)
+          [ "$arch" = "aarch64" ] || { echo "Expected aarch64, got: $arch"; exit 1; }
+          # AWS CLI v2 prints its build identifier (e.g. "exe/aarch64.debian.13")
+          # in --version output. The arch tag here proves we downloaded the
+          # awscli-exe-linux-aarch64-*.zip tarball, not the x86_64 one.
+          docker run --rm gco-dev:arm64 aws --version | grep -q "aarch64"
+          # `gco --version` actually loading proves no Python dependency fell
+          # back to an amd64-only sdist (which would fail at install or
+          # import time on an arm64 base).
+          docker run --rm gco-dev:arm64 gco --version
+          # Inspect the ELF header e_machine field for the kubectl and
+          # Docker static client binaries. We can't rely on QEMU+binfmt to
+          # surface arch mismatches by failing to run — Docker images can
+          # contain mixed-arch binaries (e.g. an amd64 kubectl alongside
+          # arm64 system binaries), and the kernel happily picks the right
+          # QEMU translator per-binary. The only durable check is the ELF
+          # header itself (e_machine = 0xB7 → AArch64; 0x3E → x86_64). We
+          # use Python's stdlib because `file`, `readelf`, and `objdump`
+          # aren't in the slim base image.
+          # `-i` attaches stdin so the heredoc reaches `python3 -` inside
+          # the container. Without it Docker drops stdin and the script
+          # runs with empty input → silent no-op (no failure, no output).
+          docker run --rm -i gco-dev:arm64 python3 - <<'PY'
+          import struct, sys, pathlib
+          EM_AARCH64 = 0xB7
+          for binary in ("/usr/local/bin/kubectl", "/usr/local/bin/docker"):
+              with pathlib.Path(binary).open("rb") as f:
+                  f.seek(0x12)  # ELF e_machine offset
+                  e_machine = struct.unpack("<H", f.read(2))[0]
+              if e_machine != EM_AARCH64:
+                  print(f"FAIL: {binary} e_machine=0x{e_machine:X} (expected 0x{EM_AARCH64:X} aarch64)", file=sys.stderr)
+                  sys.exit(1)
+              print(f"OK: {binary} is aarch64 (e_machine=0x{e_machine:X})")
+          PY
+
   # --------------------------------------------------------------------------
   # Kind cluster E2E — verifies the Kubernetes manifests apply cleanly to a
   # real API server with Calico installed so NetworkPolicy is enforced (not

CONTRIBUTING.md

@@ -6,7 +6,8 @@ Thank you for contributing to GCO (Global Capacity Orchestrator on AWS)! This gu
 
 - [Development Setup](#development-setup)
   - [Prerequisites](#prerequisites)
-  - [Local Development Environment](#local-development-environment)
+  - [Using the Dev Container (Recommended)](#using-the-dev-container-recommended)
+  - [Local Development Environment (Advanced)](#local-development-environment-advanced)
 - [Development Workflow](#development-workflow)
   - [Dependency Management](#dependency-management)
   - [Type Checking](#type-checking)
@@ -30,36 +31,30 @@ Thank you for contributing to GCO (Global Capacity Orchestrator on AWS)! This gu
 
 ### Prerequisites
 
-- AWS account with appropriate permissions
-- Python 3.10+ (required for type union syntax `str | None`)
-- Node.js 24+ (for CDK)
-- Docker or Finch
-- kubectl
-- Git
+**Recommended path — the dev container only needs:**
 
-**Alternative: Use the Dev Container** (Python 3.14, Node.js 24, CDK, kubectl, AWS CLI) to avoid local dependency issues (see below).
+- AWS account with appropriate permissions
+- Docker (or Finch / Colima) and Git
 
-### Local Development Environment
+The container itself ships Python 3.14, Node.js 24, CDK, kubectl, AWS CLI, and every Python dependency at the exact versions CI uses, so you don't need any of them on your host.
 
-```bash
-# Clone repository
-git clone <repository-url>
-cd GCO
+**Host development path additionally needs:**
 
-# Create virtual environment
-python3 -m venv .venv
-source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+- Python 3.10+ (required for type union syntax `str | None`)
+- Node.js 24+ (for CDK)
+- kubectl
+- A clean virtualenv (or pipx) for the GCO Python deps — see the warning under [Local Development Environment (Advanced)](#local-development-environment-advanced).
 
-# Install dependencies
-pip install -e ".[dev]"
-```
+> **Strong recommendation:** use the dev container. GCO pins many exact package versions (FastAPI, mypy, Ruff, AWS SDKs, CDK, etc.) so CI is reproducible. Installing those on top of an existing Python environment frequently triggers `ResolutionImpossible` / resolver errors. The dev container sidesteps this entirely and matches CI bit-for-bit.
 
 ### Using the Dev Container (Recommended)
 
-The dev container includes all dependencies pre-installed (Python 3.14, Node.js 24, CDK, kubectl, AWS CLI). This avoids "works on my machine" issues.
+The dev container includes all dependencies pre-installed (Python 3.14, Node.js 24, CDK, kubectl, AWS CLI). This avoids "works on my machine" issues and is the supported path for everything from running tests to deploying stacks.
+
+The image is **multi-arch** — Apple Silicon (`linux/arm64`), Intel/x86_64 hosts, and CI all build natively from the same `Dockerfile.dev` because every baked-in binary (kubectl, AWS CLI v2, Docker static client) is selected by `$TARGETARCH`. Native builds on Apple Silicon take ~2 min; emulated cross-builds (e.g. `--platform linux/amd64` on an arm64 host) take ~7-8 min and are only needed when you specifically want to test the amd64 image.
 
 ```bash
-# Build the container
+# Build the container (cached on subsequent runs; ~2 min the first time)
 docker build -f Dockerfile.dev -t gco-dev .
 
 # Run an interactive shell
@@ -123,6 +118,26 @@ alias gco-dev='docker run --rm -v ~/.aws:/root/.aws:ro -v $(pwd):/workspace -w /
 # Then use: gco-dev gco stacks list
 ```
 
+### Local Development Environment (Advanced)
+
+Use this path only if you specifically want to develop on your host (e.g., editor integrations like the Pyright/mypy LSP). It is not the supported path for one-off deploys or running tests — those should go through the dev container above.
+
+```bash
+# Clone repository
+git clone <repository-url>
+cd GCO
+
+# Create a *fresh* virtual environment — do not reuse one that already has
+# AWS CDK, FastAPI, mypy, or other commonly-pinned packages installed.
+python3 -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+# Install dependencies
+pip install -e ".[dev]"
+```
+
+If `pip install` fails with `ResolutionImpossible` or "the conflict is caused by..." messages, your venv is not actually clean (or you're on a Python version older than 3.10). Recreate the venv from scratch or switch to the dev container — please don't loosen the pins in `pyproject.toml` or `requirements-lock.txt` to make local install work, since CI will reject the lockfile drift.
+
 ## Development Workflow
 
 ### Dependency Management
@@ -148,7 +163,7 @@ produces a deterministic, Linux-targeted lockfile that matches CI, avoids
 host-specific path leakage, and doesn't require `pip-tools` on your machine.
 
 ```bash
-# Build the dev image once (cached between runs, ~5 minutes the first time)
+# Build the dev image once (cached between runs, ~2 minutes the first time)
 docker build -f Dockerfile.dev -t gco-dev .
 
 # Regenerate the lockfile and strip the project self-reference

Dockerfile.dev

@@ -1,9 +1,18 @@
 # GCO Development Container
 # Use this container to run CDK and CLI commands with all dependencies pre-installed
 #
+# Multi-arch: builds natively on linux/amd64 (CI on x86_64 GitHub runners,
+# Docker Desktop on Intel Macs / Linux) and linux/arm64 (Apple Silicon
+# Macs running Docker Desktop, Finch, Colima, or any arm64 Linux host).
+# kubectl, the AWS CLI v2, and the Docker static client all publish both
+# arch builds; we pick the right one via BuildKit's $TARGETARCH.
+#
 # Build:
 #   docker build -f Dockerfile.dev -t gco-dev .
 #
+# To cross-build, pass --platform explicitly:
+#   docker build --platform linux/amd64 -f Dockerfile.dev -t gco-dev .
+#
 # Run (interactive):
 #   docker run -it --rm \
 #     -v ~/.aws:/root/.aws:ro \
@@ -57,6 +66,21 @@
 
 FROM python:3.14.4-slim
 
+# Multi-arch build args. BuildKit auto-populates TARGETARCH with the build
+# platform's Go-style arch name (`amd64` / `arm64`). We additionally derive
+# the GNU/Linux arch tag (`x86_64` / `aarch64`) for tools whose download
+# URLs use that convention (AWS CLI v2, Docker static binaries).
+#
+# Building natively for the host arch is the default (`finch build .` on
+# Apple Silicon → arm64; CI on x86_64 runners → amd64). To cross-build,
+# pass `--platform linux/amd64` or `--platform linux/arm64`.
+ARG TARGETARCH
+RUN case "${TARGETARCH}" in \
+        amd64)  echo "x86_64"  > /tmp/gnu_arch ;; \
+        arm64)  echo "aarch64" > /tmp/gnu_arch ;; \
+        *)      echo "Unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
+    esac
+
 # Install system dependencies
 RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
@@ -89,17 +113,22 @@ RUN npm install -g "aws-cdk@${CDK_VERSION}" \
 # Install kubectl — pinned to the EKS cluster's minor line (see
 # ``cdk.json::kubernetes_version``). The kubectl + apiserver skew policy
 # allows ±1 minor, so this pin can lag or lead the cluster by one.
+# kubectl publishes both linux/amd64 and linux/arm64 binaries, so we
+# pass ${TARGETARCH} straight through.
 ARG KUBECTL_VERSION=v1.35.4
-RUN curl -LO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
+ARG TARGETARCH
+RUN curl -LO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
     && chmod +x kubectl \
     && mv kubectl /usr/local/bin/ \
     && kubectl version --client=true
 
 # Install AWS CLI v2 — pinned to a specific release so reproducibility is
 # preserved across rebuilds. Bump when a new feature or bugfix is needed
 # (see https://github.com/aws/aws-cli/blob/v2/CHANGELOG.rst).
+# AWS CLI v2 uses GNU arch tags (x86_64 / aarch64) in the install URL.
 ARG AWSCLI_VERSION=2.34.42
-RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip" -o "awscliv2.zip" \
+RUN GNU_ARCH=$(cat /tmp/gnu_arch) \
+    && curl "https://awscli.amazonaws.com/awscli-exe-linux-${GNU_ARCH}-${AWSCLI_VERSION}.zip" -o "awscliv2.zip" \
     && unzip awscliv2.zip \
     && ./aws/install \
     && rm -rf awscliv2.zip aws \
@@ -111,14 +140,18 @@ RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}
 # container talks to the host Docker daemon through the bind-mounted
 # socket documented in the header, so only the client binary is needed
 # here. Pinned to a specific version to keep the image reproducible —
-# bump intentionally when testing against a new Docker release.
-# Latest stable at pin time (see https://download.docker.com/linux/static/stable/x86_64/):
+# bump intentionally when testing against a new Docker release. Docker
+# static binaries also use GNU arch tags (x86_64 / aarch64) in the URL.
+# Latest stable at pin time:
+#   https://download.docker.com/linux/static/stable/x86_64/
+#   https://download.docker.com/linux/static/stable/aarch64/
 ARG DOCKER_VERSION=29.4.2
-RUN curl -fsSL "https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VERSION}.tgz" \
+RUN GNU_ARCH=$(cat /tmp/gnu_arch) \
+    && curl -fsSL "https://download.docker.com/linux/static/stable/${GNU_ARCH}/docker-${DOCKER_VERSION}.tgz" \
       -o /tmp/docker.tgz \
     && tar -xzf /tmp/docker.tgz -C /tmp \
     && mv /tmp/docker/docker /usr/local/bin/ \
-    && rm -rf /tmp/docker /tmp/docker.tgz \
+    && rm -rf /tmp/docker /tmp/docker.tgz /tmp/gnu_arch \
     && docker --version
 
 # Set working directory