Document S3 compatibility features and extend smoke coverage

Update README and configuration docs for the supported operations and
upload payload modes (including the unverified-checksum caveats), add
the design spec, extend the opt-in mutation smoke with default-settings
PutObject and an `aws s3 cp` multipart round-trip, and add a project
verify skill capturing the compose e2e recipe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: terion-name

.claude/skills/verify/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: verify
+description: Build, launch, and drive the air3 compose stack to verify edge/connector changes end-to-end.
+---
+
+# Verifying air3 changes end-to-end
+
+The runtime surface is the edge gateway's public HTTPS listener (S3 API +
+signed URLs), backed by NATS, the private connector, and versitygw.
+
+## Launch
+
+```bash
+# certs already generated in deploy/certs/generated (else: make certs)
+export MUTATIONS_ENABLED=true AIR3_S3_API_ENABLED=true \
+  AIR3_S3_API_ACCESS_KEY_ID=edge-api-key AIR3_S3_API_SECRET_ACCESS_KEY=edge-api-secret
+docker compose -f deploy/compose.yaml up -d --build
+./deploy/scripts/seed-s3.sh          # seeds s3://demo/hello.txt
+```
+
+Rebuilding one service after a code change: `docker compose -f deploy/compose.yaml up -d --build private-connector`.
+
+## Drive the S3 API (no aws CLI on host)
+
+Run `amazon/aws-cli` on the compose public network. Two settings are
+mandatory or the edge rejects mutations with InvalidRequest: path-style
+addressing and unsigned payloads (plus `when_required` checksums):
+
+```bash
+printf '[default]\ns3 =\n    addressing_style = path\n    payload_signing_enabled = false\n' > /tmp/aws-config
+docker run --rm --network air3-demo_public \
+  -v "$PWD/deploy/certs/generated:/certs:ro" -v /tmp:/work \
+  -e AWS_CONFIG_FILE=/work/aws-config \
+  -e AWS_ACCESS_KEY_ID=edge-api-key -e AWS_SECRET_ACCESS_KEY=edge-api-secret \
+  -e AWS_DEFAULT_REGION=us-east-1 \
+  -e AWS_REQUEST_CHECKSUM_CALCULATION=when_required \
+  -e AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
+  amazon/aws-cli --endpoint-url https://edge-gateway:8443 --ca-bundle /certs/dev-ca.crt \
+  s3api get-object --bucket demo --key hello.txt /dev/stdout
+```
+
+## Drive signed URLs (public HMAC path)
+
+```bash
+URL=$(go run ./cmd/signurl -method GET -base-url https://localhost:8443 \
+  -bucket demo -key hello.txt -secret dev-signing-secret-change-me \
+  -expiration 2m -default-bucket-path)
+curl --cacert deploy/certs/generated/dev-ca.crt \
+  --resolve edge-gateway:8443:127.0.0.1 "${URL/localhost/edge-gateway}"
+```
+
+Raw SigV4 probes (odd methods/headers) work with
+`curl --aws-sigv4 "aws:amz:us-east-1:s3" --user edge-api-key:edge-api-secret`.
+
+## Gotchas
+
+- Logs are intentionally redacted (`safeLogError`) — both services log
+  almost nothing. Debug by watching `docker logs air3-demo-versitygw-1`,
+  which access-logs every backend request; if an operation never appears
+  there, it failed before reaching S3.
+- A status-only error (no XML `<Code>`) comes from the connector's
+  fetch-error path; XML errors come from the edge or connector mutation
+  paths. The aws CLI prints `(503)` for status-only and `(ServiceUnavailable)`
+  for XML — useful to localize failures.
+- Tear down: `docker compose -f deploy/compose.yaml down --remove-orphans`
+  (volumes persist; delete test objects via the `aws-cli` tools profile
+  against `http://versitygw:10000`).

README.md

@@ -350,8 +350,9 @@ While air3 primarily relies on its highly-secure HMAC signed URLs, you can optio
 ### Security First
 
 - **No Real S3 Credentials at the Edge:** You must define a *brand new, gateway-only* set of credentials (`AIR3_S3_API_ACCESS_KEY_ID` and `AIR3_S3_API_SECRET_ACCESS_KEY`). The Edge Gateway only uses these to verify incoming AWS SigV4 requests. It **does not** give the Edge access to the backend storage. These are completely separate from your real, private S3 credentials and your HMAC `AIR3_SIGNING_SECRET`.
-- **Supported v1 Operations:** The API supports read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, and `HeadBucket`, plus gated `PutObject` and `DeleteObject`.
-- **Mutations Off by Default:** S3-compatible `PUT`/`DELETE` handling must be explicitly gated with `MUTATIONS_ENABLED=true`. `PutObject` requires `Content-Length` and `x-amz-content-sha256: UNSIGNED-PAYLOAD`; multipart upload, ACLs, tagging, versioning, AWS chunked/streaming payloads, and signed-payload `PUT` requests are unsupported.
+- **Supported Operations:** The API supports read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, and `HeadBucket`, plus gated mutations `PutObject`, `DeleteObject`, and multipart upload (`CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`) — enough for `aws s3 cp` of large files. ACLs, tagging, versioning, `ListParts`, and `ListMultipartUploads` are unsupported.
+- **Mutations Off by Default:** S3-compatible mutation handling must be explicitly gated with `MUTATIONS_ENABLED=true`.
+- **Works with Default Client Settings:** Uploads accept `x-amz-content-sha256: UNSIGNED-PAYLOAD`, a signed SHA-256 payload hash, or the `STREAMING-UNSIGNED-PAYLOAD-TRAILER` aws-chunked mode that modern AWS CLI/SDK versions use by default (the framing is decoded at the edge). Two caveats: a signed payload hash is verified as part of the request signature but the streamed body is **not** re-hashed against it, and aws-chunked checksum trailers (`x-amz-checksum-*`) are discarded, not verified — end-to-end body integrity relies on TLS. Signed-chunk streaming (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD*`) stays rejected; clients only use it on plain-HTTP endpoints.
 - **Independent Edge/Connector Gates:** Routed mutations require `MUTATIONS_ENABLED=true` on both the Edge and the target Connector. Direct-server aliases bypass the Connector, so direct alias mutations require only the Edge gate. `AIR3_MUTATIONS_ENABLED` remains as a compatibility alias; if both names are non-empty, their boolean values must match.
 
 ### Using the API (Path-Style)

deploy/scripts/smoke.sh

@@ -107,6 +107,17 @@ aws_s3api() {
   return "$rc"
 }
 
+# Like aws_s3api but with stock CLI settings (no payload-signing or checksum
+# overrides), proving default-configured clients work against the edge.
+aws_default() {
+  AWS_CONFIG_FILE=/dev/null \
+    AWS_ACCESS_KEY_ID="$AIR3_S3_API_ACCESS_KEY_ID" \
+    AWS_SECRET_ACCESS_KEY="$AIR3_S3_API_SECRET_ACCESS_KEY" \
+    AWS_DEFAULT_REGION="${AIR3_S3_API_REGION:-us-east-1}" \
+    AWS_PAGER="" \
+    aws --endpoint-url "$BASE_URL" --ca-bundle "$CERT_DIR/dev-ca.crt" "$@"
+}
+
 mutation_gate_enabled() {
   local primary=${MUTATIONS_ENABLED:-}
   local alias=${AIR3_MUTATIONS_ENABLED:-}
@@ -207,6 +218,81 @@ run_optional_s3_mutation_smoke() {
 
   deleted_url=$(sign_url GET "$mutation_key" 2m)
   assert_status "signed GET after S3 API DeleteObject" "404" "$deleted_url"
+
+  run_default_client_mutation_smoke
+  run_multipart_mutation_smoke
+}
+
+run_default_client_mutation_smoke() {
+  echo "Running default-settings AWS CLI mutation smoke checks..."
+  local key body readback
+  key="air3-smoke-default-$(date +%s)-$$.txt"
+  body=$(mktemp)
+  readback=$(mktemp)
+  printf 'default client settings\n' >"$body"
+
+  if ! aws_default s3api put-object --bucket "$BUCKET" --key "$key" --body "$body" --content-type text/plain >/dev/null; then
+    rm -f "$body" "$readback"
+    echo "error: default-settings PutObject (aws-chunked) failed" >&2
+    exit 1
+  fi
+  echo "ok: default-settings PutObject (aws-chunked trailer mode) succeeded"
+
+  if ! aws_default s3api get-object --bucket "$BUCKET" --key "$key" "$readback" >/dev/null; then
+    rm -f "$body" "$readback"
+    echo "error: default-settings GetObject failed" >&2
+    exit 1
+  fi
+  if ! cmp -s "$body" "$readback"; then
+    rm -f "$body" "$readback"
+    echo "error: default-settings readback did not match uploaded content" >&2
+    exit 1
+  fi
+  rm -f "$body" "$readback"
+  echo "ok: default-settings readback matched uploaded content"
+
+  if ! aws_default s3api delete-object --bucket "$BUCKET" --key "$key" >/dev/null; then
+    echo "error: default-settings DeleteObject failed" >&2
+    exit 1
+  fi
+  echo "ok: default-settings DeleteObject removed temporary object"
+}
+
+run_multipart_mutation_smoke() {
+  echo "Running multipart upload smoke checks (aws s3 cp)..."
+  local key big_body big_readback size_mb
+  key="air3-smoke-multipart-$(date +%s)-$$.bin"
+  size_mb=${AIR3_DEMO_MULTIPART_MB:-16}
+  big_body=$(mktemp)
+  big_readback=$(mktemp)
+  dd if=/dev/urandom of="$big_body" bs=1048576 count="$size_mb" status=none
+
+  # aws s3 cp switches to multipart upload above the 8 MiB threshold.
+  if ! aws_default s3 cp "$big_body" "s3://$BUCKET/$key" >/dev/null; then
+    rm -f "$big_body" "$big_readback"
+    echo "error: multipart upload via aws s3 cp failed" >&2
+    exit 1
+  fi
+  echo "ok: multipart upload via aws s3 cp succeeded (${size_mb} MiB)"
+
+  if ! aws_default s3 cp "s3://$BUCKET/$key" "$big_readback" >/dev/null; then
+    rm -f "$big_body" "$big_readback"
+    echo "error: multipart download via aws s3 cp failed" >&2
+    exit 1
+  fi
+  if ! cmp -s "$big_body" "$big_readback"; then
+    rm -f "$big_body" "$big_readback"
+    echo "error: multipart readback did not match uploaded content" >&2
+    exit 1
+  fi
+  rm -f "$big_body" "$big_readback"
+  echo "ok: multipart readback matched uploaded content"
+
+  if ! aws_default s3api delete-object --bucket "$BUCKET" --key "$key" >/dev/null; then
+    echo "error: multipart cleanup DeleteObject failed" >&2
+    exit 1
+  fi
+  echo "ok: multipart cleanup removed temporary object"
 }
 
 wait_for_edge() {

docs/configuration.md

@@ -102,8 +102,8 @@ The Edge Gateway can expose a **path-style S3-compatible API** for clients and S
 You must provide a brand new set of *gateway verifier credentials*: `AIR3_S3_API_ACCESS_KEY_ID` and `AIR3_S3_API_SECRET_ACCESS_KEY`.
 - These credentials **only** live on the Edge and are used merely to authenticate incoming AWS SigV4 requests.
 - They **do not** grant access to backend S3 storage, and they are completely separate from your real S3 backend credentials.
-- Supported v1 operations are read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, `HeadBucket`, and gated mutations `PutObject` and `DeleteObject`. S3-compatible mutations (`PUT`/`DELETE`) are disabled unless `MUTATIONS_ENABLED=true` is set.
-- `PutObject` accepts only known-length, unsigned-payload uploads: clients must send `Content-Length` and use `x-amz-content-sha256: UNSIGNED-PAYLOAD`. Multipart upload, ACLs, tagging, versioning, AWS chunked transfer (`aws-chunked`), streaming payloads, and signed-payload `PUT` requests are not supported.
+- Supported operations are read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, `HeadBucket`, and gated mutations `PutObject`, `DeleteObject`, `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, and `AbortMultipartUpload`. All mutations are disabled unless `MUTATIONS_ENABLED=true` is set. ACLs, tagging, versioning, `ListParts`, and `ListMultipartUploads` are not supported.
+- Uploads (`PutObject`, `UploadPart`, `CompleteMultipartUpload`) need a known decoded length and accept three payload-hash modes: `x-amz-content-sha256: UNSIGNED-PAYLOAD` with `Content-Length`; a signed SHA-256 payload hash (verified as part of the SigV4 signature, but the streamed body is not re-hashed against it); or `STREAMING-UNSIGNED-PAYLOAD-TRAILER` with `X-Amz-Decoded-Content-Length`, the aws-chunked default of modern AWS CLI/SDK releases — the edge decodes the framing and discards (does not verify) checksum trailers. Signed-chunk streaming (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD*`) is rejected. The `CompleteMultipartUpload` part-list body is capped at 1 MiB.
 - Routed S3 mutations require `MUTATIONS_ENABLED=true` on both the Edge and the target Connector. Direct-server aliases bypass the Connector, so direct alias mutations require only the Edge gate. `AIR3_MUTATIONS_ENABLED` is a compatibility alias; if both names are non-empty, their boolean values must match.
 
 ### Path-Style Mapping Examples

docs/superpowers/specs/2026-07-06-s3-compatibility-design.md

@@ -0,0 +1,93 @@
+# S3 compatibility upgrades: aws-chunked, signed payloads, multipart upload
+
+Date: 2026-07-06. Approved scope: features 1, 2, 4 from the compatibility
+review; all defaults confirmed (signed payloads always-on, trailers parsed
+and discarded, four core multipart operations).
+
+## Goal
+
+A stock AWS CLI/SDK with no custom configuration can upload through the
+edge's S3-compatible API: `aws s3api put-object` works with default
+checksum/signing settings, and `aws s3 cp` of files >8 MiB works via
+multipart upload.
+
+## Feature 1 — aws-chunked upload decoding (edge)
+
+Modern clients send upload bodies as
+`x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER` with
+aws-chunked framing and a checksum trailer (e.g. `x-amz-checksum-crc32`).
+
+- The edge accepts this mode for upload operations (PutObject, UploadPart,
+  CompleteMultipartUpload body). `X-Amz-Decoded-Content-Length` is
+  required and becomes the object/part content length.
+- Unsigned aws-chunked framing is identical to HTTP/1.1 chunked framing;
+  decoding wraps `httputil.NewChunkedReader` plus exact decoded-length
+  enforcement (`internal/s3api`).
+- Trailers are parsed for framing validity and discarded. By the time the
+  trailer arrives the payload has already streamed to the backend, so a
+  checksum mismatch cannot fail the upload; this is documented.
+- Signed-chunk streaming modes (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD*`)
+  remain rejected: clients use them only on plain-HTTP endpoints and the
+  edge is HTTPS-only.
+- One body-resolution step in the edge feeds both the routed
+  (upload-source) and direct-server paths.
+
+## Feature 2 — signed-payload uploads (edge)
+
+`ValidatePayloadHashForOperation` additionally accepts a 64-hex-digit
+payload hash for mutation operations. The hash is covered by the verified
+SigV4 signature; the edge does not re-hash the streaming body (impossible
+without buffering). Transport integrity remains TLS's job. Always-on, no
+config flag. Documented as "accepted but not verified".
+
+## Feature 4 — multipart upload
+
+Operations: CreateMultipartUpload, UploadPart, CompleteMultipartUpload,
+AbortMultipartUpload — the set `aws s3 cp` needs. ListParts and
+ListMultipartUploads are deferred.
+
+- **Classification** (`internal/s3api`): by query params — POST `?uploads`
+  → Create; PUT `?partNumber&uploadId` → UploadPart; POST `?uploadId` →
+  Complete; DELETE `?uploadId` → Abort.
+- **Gating**: all four are mutations, gated by `MUTATIONS_ENABLED` on both
+  edge and connector, same as PutObject/DeleteObject.
+- **Tickets** (`internal/tickets`): new operations plus a `Multipart`
+  envelope: `UploadID`, `PartNumber`, and `Rewrite{Bucket,Key}` carrying
+  the public names for response XML. Part lists never ride NATS tickets,
+  keeping message size bounded.
+- **Body transport**: UploadPart streams its body through the existing
+  upload-source channel exactly like PutObject. CompleteMultipartUpload's
+  part-list XML travels the same way, capped at 1 MiB (S3 allows at most
+  10,000 parts ≈ <1 MiB of XML).
+- **Backend calls** (`internal/s3fetch`): SDK CreateMultipartUpload /
+  UploadPart (UNSIGNED-PAYLOAD, same as PutObject) / Complete (parses the
+  XML via `internal/s3api`) / Abort. Create and Complete responses are
+  rendered deterministically with public bucket/key (same pattern as the
+  ListObjectsV2 rewrite).
+- **Statuses**: Create 200 + XML, UploadPart 200 + ETag header,
+  Complete 200 + XML, Abort 204.
+- **Direct-server aliases** get all four operations through the shared
+  s3fetch code (edge-only gate, as with other direct mutations).
+
+## Error handling
+
+- Malformed chunked framing or decoded-length mismatch surfaces as a read
+  error mid-stream; the backend PUT aborts (short body) and the client
+  receives the standard stream-failure error.
+- Unknown/oversized Complete bodies → 400 InvalidRequest at the edge.
+- Multipart requests while mutations are disabled → 405, `Allow`
+  reflecting the gate.
+
+## Testing
+
+Unit tests per package (chunked decoder incl. malformed framing and length
+mismatches; classification; ticket validation; s3fetch operations against
+stub servers; edge/connector wiring). End-to-end: compose stack driven by
+a default-configured `aws` CLI — put-object without custom config, and
+`aws s3 cp` of a >8 MiB file (multipart), plus abort and gate-off probes.
+
+## Docs
+
+README limitations updated (multipart/aws-chunked/signed-payload lines),
+`docs/configuration.md` operation list, and a short "Using the AWS CLI"
+note. `deploy/scripts/smoke.sh` gains default-config and multipart checks.