Add opt-in S3 mutation smoke coverage

Author: terion-name

README.md

@@ -343,14 +343,15 @@ If you need to serve partial files (like for streaming video), `air3` fully supp
 
 If you're using signed URLs and expect clients to send `Range` headers, you must include the exact range claim when generating the URL (e.g., `cmd/signurl -range 'bytes=0-99'`). The Connector forwards authorized ranges to S3, seamlessly returning a `206 Partial Content` response to the client.
 
-## Optional: Read-Only S3-Compatible API
+## Optional: S3-Compatible API
 
-While air3 primarily relies on its highly-secure HMAC signed URLs, you can optionally enable a **read-only S3-compatible API** (`AIR3_S3_API_ENABLED=true`). This is extremely useful if your clients or internal tools already use standard S3 SDKs (like `aws-cli` or `boto3`) and you want them to fetch files without modifying their code.
+While air3 primarily relies on its highly-secure HMAC signed URLs, you can optionally enable a **path-style S3-compatible API** (`AIR3_S3_API_ENABLED=true`). It is read-only by default and is useful if your clients or internal tools already use standard S3 SDKs (like `aws-cli` or `boto3`) and you want them to fetch files without modifying their code.
 
 ### Security First
 
 - **No Real S3 Credentials at the Edge:** You must define a *brand new, gateway-only* set of credentials (`AIR3_S3_API_ACCESS_KEY_ID` and `AIR3_S3_API_SECRET_ACCESS_KEY`). The Edge Gateway only uses these to verify incoming AWS SigV4 requests. It **does not** give the Edge access to the backend storage. These are completely separate from your real, private S3 credentials and your HMAC `AIR3_SIGNING_SECRET`.
-- **Mutations Off by Default:** The API defaults to safe read operations: `GetObject`, `HeadObject`, `ListObjectsV2`, and `HeadBucket`. S3-compatible `PUT`/`DELETE` handling must be explicitly gated with `MUTATIONS_ENABLED=true`.
+- **Supported v1 Operations:** The API supports read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, and `HeadBucket`, plus gated `PutObject` and `DeleteObject`.
+- **Mutations Off by Default:** S3-compatible `PUT`/`DELETE` handling must be explicitly gated with `MUTATIONS_ENABLED=true`. `PutObject` requires `Content-Length` and `x-amz-content-sha256: UNSIGNED-PAYLOAD`; multipart upload, ACLs, tagging, versioning, AWS chunked/streaming payloads, and signed-payload `PUT` requests are unsupported.
 - **Independent Edge/Connector Gates:** Routed mutations require `MUTATIONS_ENABLED=true` on both the Edge and the target Connector. Direct-server aliases bypass the Connector, so direct alias mutations require only the Edge gate. `AIR3_MUTATIONS_ENABLED` remains as a compatibility alias; if both names are non-empty, their boolean values must match.
 
 ### Using the API (Path-Style)

deploy/scripts/smoke-multiserver.sh

@@ -80,19 +80,26 @@ assert_status() {
   echo "ok: $label returned HTTP $expected"
 }
 
-assert_body() {
+assert_body_equals() {
   local label=$1
-  local url=$2
+  local expected=$2
+  local url=$3
   local body
   body=$(curl --silent --show-error --fail --cacert "$CERT_DIR/dev-ca.crt" "$url")
-  if [ "$body"$'\n' != "$EXPECTED" ] && [ "$body" != "$EXPECTED" ]; then
+  if [ "$body"$'\n' != "$expected" ] && [ "$body" != "$expected" ]; then
     echo "error: $label body did not match expected content" >&2
-    printf 'expected: %q\nactual:   %q\n' "$EXPECTED" "$body" >&2
+    printf 'expected: %q\nactual:   %q\n' "$expected" "$body" >&2
     exit 1
   fi
   echo "ok: $label returned expected content"
 }
 
+assert_body() {
+  local label=$1
+  local url=$2
+  assert_body_equals "$label" "$EXPECTED" "$url"
+}
+
 assert_head_no_body() {
   local label=$1
   local url=$2
@@ -162,18 +169,42 @@ aws_s3api() {
   local aws_config rc
   aws_config=$(mktemp)
   temp_files+=("$aws_config")
-  printf '[default]\ns3 =\n    addressing_style = path\n' >"$aws_config"
+  printf '[default]\ns3 =\n    addressing_style = path\n    payload_signing_enabled = false\n' >"$aws_config"
   AWS_CONFIG_FILE="$aws_config" \
     AWS_ACCESS_KEY_ID="$AIR3_S3_API_ACCESS_KEY_ID" \
     AWS_SECRET_ACCESS_KEY="$AIR3_S3_API_SECRET_ACCESS_KEY" \
     AWS_DEFAULT_REGION="${AIR3_S3_API_REGION:-us-east-1}" \
     AWS_PAGER="" \
+    AWS_REQUEST_CHECKSUM_CALCULATION="when_required" \
+    AWS_RESPONSE_CHECKSUM_VALIDATION="when_required" \
     aws --endpoint-url "$BASE_URL" --ca-bundle "$CERT_DIR/dev-ca.crt" s3api "$@"
   rc=$?
   rm -f "$aws_config"
   return "$rc"
 }
 
+mutation_gate_enabled() {
+  local primary=${MUTATIONS_ENABLED:-}
+  local alias=${AIR3_MUTATIONS_ENABLED:-}
+  if [ -n "$primary" ] && [ -n "$alias" ] && [ "$primary" != "$alias" ]; then
+    return 1
+  fi
+  if [ -n "$primary" ]; then
+    [ "$primary" = "true" ]
+    return
+  fi
+  [ "$alias" = "true" ]
+}
+
+mutation_smoke_ready() {
+  s3_api_smoke_ready || return 1
+  if ! mutation_gate_enabled; then
+    echo "skip: optional multi-server S3-compatible mutation smoke checks need MUTATIONS_ENABLED=true (or matching AIR3_MUTATIONS_ENABLED=true)"
+    return 1
+  fi
+  return 0
+}
+
 assert_s3_api_body() {
   local label=$1
   local bucket=$2
@@ -227,6 +258,39 @@ run_optional_s3_api_smoke() {
   assert_s3_api_list_contains "direct default-bucket mapping" "$DIRECT_SERVER" "$KEY" "$KEY"
 }
 
+run_optional_s3_mutation_smoke() {
+  mutation_smoke_ready || return 0
+
+  echo "Running optional multi-server S3-compatible mutation smoke checks..."
+  local mutation_key mutation_content mutation_body get_url head_url deleted_url
+  mutation_key=${AIR3_DEMO_MUTATION_KEY:-"air3-smoke-mutation-$(date +%s)-$$.txt"}
+  mutation_content=${AIR3_DEMO_MUTATION_CONTENT:-$'air3 routed mutation smoke\n'}
+  mutation_body=$(mktemp)
+  temp_files+=("$mutation_body")
+  printf '%s' "$mutation_content" >"$mutation_body"
+
+  if ! aws_s3api put-object --bucket "$BLUE_SERVER" --key "$mutation_key" --body "$mutation_body" --content-type text/plain >/dev/null; then
+    echo "error: blue routed S3 API PutObject failed" >&2
+    exit 1
+  fi
+  echo "ok: blue routed S3 API PutObject created temporary object"
+
+  get_url=$(sign_default_bucket_url GET "$BLUE_SERVER" "$mutation_key" 2m)
+  assert_body_equals "blue signed GET after routed S3 API PutObject" "$mutation_content" "$get_url"
+
+  head_url=$(sign_default_bucket_url HEAD "$BLUE_SERVER" "$mutation_key" 2m)
+  assert_head_no_body "blue signed HEAD after routed S3 API PutObject" "$head_url"
+
+  if ! aws_s3api delete-object --bucket "$BLUE_SERVER" --key "$mutation_key" >/dev/null; then
+    echo "error: blue routed S3 API DeleteObject failed" >&2
+    exit 1
+  fi
+  echo "ok: blue routed S3 API DeleteObject removed temporary object"
+
+  deleted_url=$(sign_default_bucket_url GET "$BLUE_SERVER" "$mutation_key" 2m)
+  assert_status "blue signed GET after routed S3 API DeleteObject" "404" "$deleted_url"
+}
+
 wait_for_blue() {
   echo "Waiting for edge gateway at $BASE_URL with server '$BLUE_SERVER'..."
   local url
@@ -272,6 +336,7 @@ direct_head_url=$(sign_default_bucket_url HEAD "$DIRECT_SERVER" "$KEY" 2m)
 check_optional_head "direct signed" "$direct_head_url"
 
 run_optional_s3_api_smoke
+run_optional_s3_mutation_smoke
 
 mutated_url=${blue_get_url/\/$BLUE_SERVER\//\/$GREEN_SERVER\/$BUCKET\/}
 if [ "$mutated_url" = "$blue_get_url" ]; then

deploy/scripts/smoke.sh

@@ -93,18 +93,42 @@ s3_api_smoke_ready() {
 aws_s3api() {
   local aws_config rc
   aws_config=$(mktemp)
-  printf '[default]\ns3 =\n    addressing_style = path\n' >"$aws_config"
+  printf '[default]\ns3 =\n    addressing_style = path\n    payload_signing_enabled = false\n' >"$aws_config"
   AWS_CONFIG_FILE="$aws_config" \
     AWS_ACCESS_KEY_ID="$AIR3_S3_API_ACCESS_KEY_ID" \
     AWS_SECRET_ACCESS_KEY="$AIR3_S3_API_SECRET_ACCESS_KEY" \
     AWS_DEFAULT_REGION="${AIR3_S3_API_REGION:-us-east-1}" \
     AWS_PAGER="" \
+    AWS_REQUEST_CHECKSUM_CALCULATION="when_required" \
+    AWS_RESPONSE_CHECKSUM_VALIDATION="when_required" \
     aws --endpoint-url "$BASE_URL" --ca-bundle "$CERT_DIR/dev-ca.crt" s3api "$@"
   rc=$?
   rm -f "$aws_config"
   return "$rc"
 }
 
+mutation_gate_enabled() {
+  local primary=${MUTATIONS_ENABLED:-}
+  local alias=${AIR3_MUTATIONS_ENABLED:-}
+  if [ -n "$primary" ] && [ -n "$alias" ] && [ "$primary" != "$alias" ]; then
+    return 1
+  fi
+  if [ -n "$primary" ]; then
+    [ "$primary" = "true" ]
+    return
+  fi
+  [ "$alias" = "true" ]
+}
+
+mutation_smoke_ready() {
+  s3_api_smoke_ready || return 1
+  if ! mutation_gate_enabled; then
+    echo "skip: optional S3-compatible mutation smoke checks need MUTATIONS_ENABLED=true (or matching AIR3_MUTATIONS_ENABLED=true)"
+    return 1
+  fi
+  return 0
+}
+
 run_optional_s3_api_smoke() {
   s3_api_smoke_ready || return 0
 
@@ -148,6 +172,43 @@ run_optional_s3_api_smoke() {
   echo "ok: S3 API ListObjectsV2 included $KEY"
 }
 
+run_optional_s3_mutation_smoke() {
+  mutation_smoke_ready || return 0
+
+  echo "Running optional S3-compatible mutation smoke checks..."
+  local mutation_key mutation_content mutation_body get_url deleted_url
+  mutation_key=${AIR3_DEMO_MUTATION_KEY:-"air3-smoke-mutation-$(date +%s)-$$.txt"}
+  mutation_content=${AIR3_DEMO_MUTATION_CONTENT:-$'air3 mutation smoke\n'}
+  mutation_body=$(mktemp)
+  printf '%s' "$mutation_content" >"$mutation_body"
+
+  if ! aws_s3api put-object --bucket "$BUCKET" --key "$mutation_key" --body "$mutation_body" --content-type text/plain >/dev/null; then
+    rm -f "$mutation_body"
+    echo "error: S3 API PutObject failed" >&2
+    exit 1
+  fi
+  rm -f "$mutation_body"
+  echo "ok: S3 API PutObject created temporary object"
+
+  get_url=$(sign_url GET "$mutation_key" 2m)
+  assert_body "signed GET after S3 API PutObject" "$mutation_content" "$get_url"
+
+  if ! aws_s3api head-object --bucket "$BUCKET" --key "$mutation_key" >/dev/null; then
+    echo "error: S3 API HeadObject after PutObject failed" >&2
+    exit 1
+  fi
+  echo "ok: S3 API HeadObject found temporary object"
+
+  if ! aws_s3api delete-object --bucket "$BUCKET" --key "$mutation_key" >/dev/null; then
+    echo "error: S3 API DeleteObject failed" >&2
+    exit 1
+  fi
+  echo "ok: S3 API DeleteObject removed temporary object"
+
+  deleted_url=$(sign_url GET "$mutation_key" 2m)
+  assert_status "signed GET after S3 API DeleteObject" "404" "$deleted_url"
+}
+
 wait_for_edge() {
   echo "Waiting for edge gateway at $BASE_URL..."
   local url
@@ -203,6 +264,7 @@ short_form_url=$(sign_url GET "$SHORT_FORM_COLLISION_KEY" 2m)
 assert_body "short-form /$SHORT_FORM_COLLISION_KEY default-bucket routing" "$SHORT_FORM_EXPECTED" "$short_form_url"
 
 run_optional_s3_api_smoke
+run_optional_s3_mutation_smoke
 
 bad_url=$(printf '%s' "$get_url" | sed 's/sig=[^&]*/sig=deadbeef/')
 assert_status "bad signature rejection" "403" "$bad_url"

docs/configuration.md

@@ -23,7 +23,7 @@ The Edge Gateway is your public-facing entry point. In the default recommended t
 | `S3_{SUFFIX}_BUCKET` | unset | Optional per-server default bucket for multi-server public paths. For alias `blue`, `S3_BLUE_BUCKET=demo` lets signed URLs use `/blue/{key}` instead of `/blue/demo/{key}`. |
 | `DIRECT_SERVERS` | unset | Bare fallback name for `AIR3_DIRECT_SERVERS`. If both are set, the values must match exactly or startup fails. |
 | `AIR3_EDGE_ALLOWED_CONNECTOR_IDENTITIES` | unset | (Optional) Comma-separated list of allowed Connector certificate identities for mTLS ingest connections. |
-| `AIR3_S3_API_ENABLED` | `false` | Opt in to the public read-only S3-compatible API for AWS SigV4-shaped requests. When `false`, normal air3 HMAC signed URLs continue unchanged and S3 API credentials are not required. |
+| `AIR3_S3_API_ENABLED` | `false` | Opt in to the public S3-compatible API for AWS SigV4-shaped requests. It is read-only by default; gated mutation operations require `MUTATIONS_ENABLED=true`. When `false`, normal air3 HMAC signed URLs continue unchanged and S3 API credentials are not required. |
 | `MUTATIONS_ENABLED` | `false` | Enables S3-compatible mutation methods at the Edge when mutation handling is installed. S3 API mutations are off by default. `AIR3_MUTATIONS_ENABLED` is a compatibility alias; if both names are non-empty, they must parse to the same boolean. Routed Connector mutations require this gate on both Edge and Connector; direct-server alias mutations require only the Edge gate. |
 | `AIR3_S3_API_REGION` | `us-east-1` | Region clients must use in the AWS SigV4 credential scope. Region matching is strict. |
 | `AIR3_S3_API_ACCESS_KEY_ID` | unset | Public gateway SigV4 verifier access key ID. Required only when `AIR3_S3_API_ENABLED=true`; not a backend connector/direct S3 credential. |
@@ -94,16 +94,17 @@ Use `AIR3_MULTI_SERVER=true` when a single Edge Gateway needs to route requests
 **Combining Multi-Server with Short URLs:**
 You can assign default buckets to specific server aliases using `S3_{SUFFIX}_BUCKET`. For example, setting `S3_BLUE_BUCKET=demo` allows the Edge to serve `/blue/file.txt` and automatically map it to the `demo` bucket for the "blue" connector.
 
-## 3. Optional Read-Only S3-Compatible API
+## 3. Optional S3-Compatible API
 
-The Edge Gateway can expose a **read-only, path-style S3-compatible API** for clients and SDKs (like `aws-cli` or `boto3`) that natively speak AWS SigV4. Enable it with `AIR3_S3_API_ENABLED=true`.
+The Edge Gateway can expose a **path-style S3-compatible API** for clients and SDKs (like `aws-cli` or `boto3`) that natively speak AWS SigV4. Enable it with `AIR3_S3_API_ENABLED=true`.
 
 ### Security First
 You must provide a brand new set of *gateway verifier credentials*: `AIR3_S3_API_ACCESS_KEY_ID` and `AIR3_S3_API_SECRET_ACCESS_KEY`.
 - These credentials **only** live on the Edge and are used merely to authenticate incoming AWS SigV4 requests.
 - They **do not** grant access to backend S3 storage, and they are completely separate from your real S3 backend credentials.
-- By default, the API is read-only (`GetObject`, `HeadObject`, `ListObjectsV2`, `HeadBucket`). S3-compatible mutations (`PUT`/`DELETE`) are disabled unless `MUTATIONS_ENABLED=true` is set.
-- Routed S3 mutations require `MUTATIONS_ENABLED=true` on both the Edge and the target Connector. Direct-server aliases bypass the Connector, so direct alias mutations require only the Edge gate.
+- Supported v1 operations are read paths `GET`/`GetObject`, `HEAD`/`HeadObject`, `ListObjectsV2`, `HeadBucket`, and gated mutations `PutObject` and `DeleteObject`. S3-compatible mutations (`PUT`/`DELETE`) are disabled unless `MUTATIONS_ENABLED=true` is set.
+- `PutObject` accepts only known-length, unsigned-payload uploads: clients must send `Content-Length` and use `x-amz-content-sha256: UNSIGNED-PAYLOAD`. Multipart upload, ACLs, tagging, versioning, AWS chunked transfer (`aws-chunked`), streaming payloads, and signed-payload `PUT` requests are not supported.
+- Routed S3 mutations require `MUTATIONS_ENABLED=true` on both the Edge and the target Connector. Direct-server aliases bypass the Connector, so direct alias mutations require only the Edge gate. `AIR3_MUTATIONS_ENABLED` is a compatibility alias; if both names are non-empty, their boolean values must match.
 
 ### Path-Style Mapping Examples
 
@@ -243,13 +244,20 @@ Our included `deploy/compose.yaml` demo clearly illustrates the required network
 | `broker` | `edge-gateway`, `nats`, `private-connector` | The middle ground for NATS control messages and Edge ingest routing. |
 | `private` | `private-connector`, `versitygw`, `aws-cli` | Deeply isolated S3 access. Marked `internal: true`. The Edge Gateway cannot reach this. |
 
-The Compose demo wires the optional S3-compatible API variables through to the Edge with `AIR3_S3_API_ENABLED=false`, `MUTATIONS_ENABLED=false`, and empty verifier credentials by default, so credentials and mutation support are not required unless you explicitly enable them:
+The Compose demo wires the optional S3-compatible API variables through to the Edge with `AIR3_S3_API_ENABLED=false`, `MUTATIONS_ENABLED=false`, and empty verifier credentials by default, so credentials and mutation support are not required unless you explicitly enable them. Leave `MUTATIONS_ENABLED` unset/false for the normal read-only demos; set it only when you intentionally want the opt-in mutation smoke checks to run:
 
 ```sh
 AIR3_S3_API_ENABLED=true \
 AIR3_S3_API_ACCESS_KEY_ID=demo-gateway-access \
 AIR3_S3_API_SECRET_ACCESS_KEY=demo-gateway-secret \
 make compose-up
+
+# Optional mutation demo/smoke (requires services started with the same gate).
+AIR3_S3_API_ENABLED=true \
+AIR3_S3_API_ACCESS_KEY_ID=demo-gateway-access \
+AIR3_S3_API_SECRET_ACCESS_KEY=demo-gateway-secret \
+MUTATIONS_ENABLED=true \
+make compose-up
 ```
 
 ## Release artifacts and images