Remove underscores from object metadata keys

[samantha-earthmover]

What

Renames the five object-metadata keys Icechunk stamps on writes so they contain letters only:

before	after
ic_spec_ver	icspecver
ic_client	icclient
ic_file_type	icfiletype
ic_comp_alg	iccompalg
icechunk_write_id	icechunkwriteid

The old names stay available as *_DEPRECATED constants, for scripts that read repositories written before this change.

Why

nginx sets underscores_in_headers off by default, so an S3 gateway behind nginx silently drops a request header whose name contains an underscore. Icechunk signs its metadata headers, so the gateway then sees a header in SignedHeaders that is not in the request. MinIO-derived gateways answer:

AccessDenied: There were headers present in the request which were not signed

The message points at the wrong thing. Nothing extra arrived, something signed went missing (extractSignedHeaders, cmd/signature-v4-utils.go).

Tigris serves t3.storage.dev from a mixed fleet, so only some nodes are affected. That made it look like a random flake in Arraylake's integration tests: a whole job failed or passed together, across every client version and environment, and only ever on writes, because reads carry no metadata.

66.93.0.25 runs nginx, 66.93.0.22 does not. Same request, same second, 10 runs each:

NODE             SERVER         HEADER                     RESULT
66.93.0.22       Tigris OS      x-amz-meta-probe_key       . . . . . .
66.93.0.22       Tigris OS      x-amz-meta-probekey        . . . . . .
66.93.0.25       nginx          x-amz-meta-probe_key       R R R R R R
66.93.0.25       nginx          x-amz-meta-probekey        . . . . . .

Script that produces that table (needs no credentials)

MinIO-derived code validates SignedHeaders before it looks up the access key, so a fake key still reaches the branch under test.

#!/usr/bin/env bash
#
# Does an S3 gateway drop user-metadata header names that contain "_"?
#
# nginx sets `underscores_in_headers off` by default and discards such a header.
# When that header is also in the SigV4 SignedHeaders list, the gateway sees a
# signed header that is not in the request, and MinIO-derived gateways answer
# "AccessDenied: There were headers present in the request which were not signed".
# The message points at the wrong thing: nothing extra arrived, something signed
# went missing. Icechunk therefore keeps every object-metadata key alphanumeric.
#
# The probe needs no credentials, because MinIO-derived code validates
# SignedHeaders before it looks up the access key.
#
# Usage:
#   scripts/gateway-metadata-header-probe.sh [endpoint] [bucket]
#   REPS=20 scripts/gateway-metadata-header-probe.sh https://t3.storage.dev
#   IPS="1.2.3.4 5.6.7.8" scripts/gateway-metadata-header-probe.sh
#
# DNS hands out one slice of an anycast fleet at a time, so a single run can miss
# the affected nodes. Pass IPS to pin the addresses you want to compare.
#
# Reads "." as "header survived" and "R" as "header dropped".

set -uo pipefail

ENDPOINT=${1:-https://t3.storage.dev}
BUCKET=${2:-icechunk-nonexistent-probe-bucket}
REPS=${REPS:-10}

HOST=${ENDPOINT#https://}
HOST=${HOST#http://}
HOST=${HOST%%/*}

DATE=$(date -u +%Y%m%dT%H%M%SZ)
SCOPE_DATE=${DATE%%T*}
EMPTY_SHA=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
FAKE_SIG=0000000000000000000000000000000000000000000000000000000000000000

probe_once() {
  local ip=$1 header=$2 resolve=()
  [ -n "$ip" ] && resolve=(--resolve "${HOST}:443:${ip}")
  curl -s --max-time 20 "${resolve[@]}" -X PUT "${ENDPOINT}/${BUCKET}/probe.txt" \
    -H "x-amz-date: ${DATE}" \
    -H "x-amz-content-sha256: ${EMPTY_SHA}" \
    -H "${header}: probe" \
    -H "Authorization: AWS4-HMAC-SHA256 Credential=probe/${SCOPE_DATE}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;${header}, Signature=${FAKE_SIG}" \
    --data-binary "" \
  | grep -oE "<Code>[^<]*" | head -1 | cut -d'>' -f2
}

server_of() {
  local ip=$1 resolve=()
  [ -n "$ip" ] && resolve=(--resolve "${HOST}:443:${ip}")
  curl -s --max-time 15 "${resolve[@]}" -o /dev/null -D - "${ENDPOINT}/" \
  | grep -i '^server:' | tr -d '\r' | cut -d' ' -f2- | head -1
}

# One anycast name can front a mixed fleet, so probe each address on its own.
ips=${IPS:-}
if [ -z "$ips" ] && command -v dig >/dev/null 2>&1; then
  ips=$(dig +short "$HOST" A | grep -E '^[0-9.]+$' | sort -u)
fi

printf '%-16s %-14s %-26s %s\n' NODE SERVER HEADER RESULT
for ip in ${ips:-""}; do
  server=$(server_of "$ip")
  for header in x-amz-meta-probe_key x-amz-meta-probekey; do
    row=""
    for _ in $(seq 1 "$REPS"); do
      case "$(probe_once "$ip" "$header")" in
        AccessDenied) row="${row} R" ;;
        InvalidAccessKeyId | SignatureDoesNotMatch) row="${row} ." ;;
        *) row="${row} ?" ;;
      esac
    done
    printf '%-16s %-14s %-26s%s\n' "${ip:-default}" "${server:-?}" "$header" "$row"
  done
done

cat <<'LEGEND'

  .  header survived, the request reached signature or key validation
  R  header dropped, gateway answered "headers present ... which were not signed"
  ?  unexpected response, inspect by hand

A gateway that prints R for the underscore row and . for the plain row mangles
metadata header names. Icechunk avoids it by keeping keys alphanumeric.
LEGEND

Why not hyphens

Azure requires metadata names to be valid C# identifiers, which forbids -. That is why the keys used _ in the first place (design doc 017). Letters and digits are the only characters both back ends accept, so the new names carry no separator at all. Design doc 017 is updated to record both constraints.

Compatibility

The four format keys have no reader in this repo; they are informational. icechunkwriteid is read back, but only to recognise a write the same process just issued, so it is always written and read with the new name. Objects written by older versions keep the old keys and stay readable; they simply read as "not ours" in the readback path, which is already the correct answer for another writer's object.

Testing

cargo test -p icechunk-format -p icechunk-storage passes. A test in each crate asserts the keys stay ASCII alphanumeric so this cannot regress. cargo fmt --check, cargo clippy --all-targets and cargo check --workspace --all-targets are clean.

Related: earth-mover/arraylake#7414.

---

Draft on purpose. Samantha, per AI_USAGE_POLICY.md the description should be in your own words along with your review attestation, so please rewrite this before marking it ready.

[This is Claude Code on behalf of Samantha Hughes]

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.96%. Comparing base (d0d25e2) to head (ecff41b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2354   +/-   ##
=======================================
  Coverage   84.95%   84.96%           
=======================================
  Files          88       88           
  Lines       39594    39611   +17     
=======================================
+ Hits        33638    33655   +17     
  Misses       5956     5956           

Flag	Coverage Δ
python	90.09% <ø> (ø)
rust	84.37% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[paraseba]

can we also keep constants for the old names, maybe with suffix _DEPRACATED or something? In case we ever need to write some scripts that use them with old repos

[paraseba]

Let's put less detail, just what happened not why

[paraseba]

Let's wait on Luiz's +1 too, to have two people thinking if we depend on this anywhere.

[li-em]

Unfortunate to drop _ and - because it is harder to read, but it is the right solution. Thanks!