`Expect: 100-continue` + zero-length PUT poisons the pooled connection: next response parsed against a stale status tuple (silent 60s stall + hidden retry + double-write)

[ThomasBurgess2000]

Describe the bug

When a PutObject with an empty body is sent over a pooled keep-alive connection to an S3-compatible server built on Go's net/http (SeaweedFS, MinIO, …), botocore's Expect: 100-continue handling permanently corrupts that connection object, and the next request on the connection has its response parsed against a stale, stashed status tuple instead of the socket.

Root cause walk-through (botocore 1.42.70):

1. handlers.add_expect_header adds Expect: 100-continue to every S3 PUT (registered on before-call.s3) — including zero-length bodies.
2. Go's net/http is RFC 9110-compliant: for a request with Content-Length: 0 it never emits an interim 100 Continue (server.go: if req.ProtoAtLeast(1, 1) && req.ContentLength != 0). RFC 9110 §10.1.1 explicitly permits omitting 100 "if the framing indicates that there is no content". So the final 200 arrives in the interim-response slot, within botocore's 1-second wait_for_read window (awsrequest.py, AWSConnection._send_output).
3. AWSConnection._handle_expect_response reads that early status line and — in its non-100 branch — permanently swaps self.response_class to functools.partial(AWSHTTPResponse, status_tuple=(...)) so the already-consumed status line can be replayed for this response.
4. The bug: AWSConnection.request() resets response_class back to the default only in the code path taken by requests that do not carry Expect. In a burst of PUTs, every request carries Expect, so the poisoned response_class persists on the pooled connection.
5. On the next (non-empty) PUT, the server does send 100 Continue + a final response — but AWSHTTPResponse._read_status replays the stale stashed tuple without reading the socket. Header parsing then starts at the real (unread) status line and the client is left holding a status-only response with no headers.

The wire is clean throughout — we verified with a byte-logging TCP tap that the server sends exactly one well-formed response per request. The "corrupt" bytes exist only in the client's parse.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

• A zero-length PUT carrying Expect: 100-continue against a server that (per RFC 9110 §10.1.1) skips the interim 100 Continue should complete cleanly and leave the pooled connection in a clean state.
• Every subsequent response on a reused connection should be parsed from the socket; no parsing state stashed for one response should ever survive into the next request on that connection.
• If a response cannot be parsed, the failure should surface as an error attributable to the request — not as a silent full-read_timeout stall followed by an invisible retry.

Current Behavior

On the first non-empty PUT after a zero-length PUT on the same keep-alive connection:

• http.client records MissingHeaderBodySeparatorDefect; urllib3 logs Failed to parse headers with the entire real, well-formed response captured as unparsed_data.
• The stale tuple yields a status-only 200 with no headers: no Content-Length → framing falls back to read-until-EOF on a kept-alive socket → the call blocks for the full read_timeout (60s with defaults).
• botocore then raises ReadTimeoutError internally and silently retries, re-executing the request on a fresh connection (close() restores response_class, which is why retries always succeed). The operation reports success with RetryAttempts=1; the object is written twice server-side.
• Nothing surfaces above one WARNING log line.

Observed output from the reproduction below (SeaweedFS 4.18, read_timeout=5 to keep the stall short):

python   3.13.12
boto3    1.42.70
botocore 1.42.70
urllib3  2.6.3
server   seaweedfs   mitigation: none

--- 3 PUTs over one keep-alive connection (read_timeout=5s) ---
PUT a-4kb     4096B -> HTTP 200  RetryAttempts=0  wall=  1.01s
PUT b-empty      0B -> HTTP 200  RetryAttempts=0  wall=  0.00s
PUT c-4kb     4096B -> HTTP 200  RetryAttempts=1  wall=  5.76s

--- urllib3 warnings captured ---
Failed to parse headers (url=http://127.0.0.1:18333/expect-repro/ecd81fc9/c-4kb):
[MissingHeaderBodySeparatorDefect()], unparsed data: 'HTTP/1.1 200 OK\r\nAccept-Ranges: bytes\r\n
Content-Length: 0\r\nETag: "20439f79e4e9dc95be34b21029221f80"\r\nServer: SeaweedFS 30GB 4.18\r\n
Vary: Origin\r\nX-Amz-Request-Id: 18C460A53951A5004F798F6D\r\nX-Amz-Version-Id: 673b9f5a...\r\n
Date: Tue, 21 Jul 2026 18:13:35 GMT\r\n\r\n'

--- server-side write count for the final key ---
bucket versioning shows 2 version(s) of c-4kb

--- verdict ---
BUG REPRODUCED: header-parse failure + 5s stall + silent retry on final PUT, object written 2x

Reproduction Steps

Deterministic (3/3 consecutive runs, byte-identical artifact shape). The script performs three PUTs over one keep-alive connection: PUT 4KB → PUT 0B → PUT 4KB.

Option A — real S3-compatible Go server (SeaweedFS 4.18):

# terminal 1
docker run --rm -p 18333:8333 --entrypoint sh chrislusf/seaweedfs:4.18 -c \
  'printf "{\"identities\":[{\"name\":\"repro\",\"credentials\":[{\"accessKey\":\"repro\",\"secretKey\":\"repro\"}],\"actions\":[\"Admin\",\"Read\",\"Write\",\"List\",\"Tagging\"]}]}" > /tmp/s3.json \
   && exec weed server -s3 -s3.config=/tmp/s3.json -volume.max=100 -master.volumeSizeLimitMB=100'

# terminal 2
python repro.py

Option B — no docker needed:

python repro.py --fake-server

runs a ~45-line in-process Python socket server that mimics exactly Go net/http's RFC-compliant Expect handling (interim 100 only when the body is non-empty). Same artifact.

Sanity checks that isolate the trigger — both of these make the same run fully clean (no warning, RetryAttempts=0, single version written):

BOTO_EXPERIMENTAL__NO_EMPTY_CONTINUE=true python repro.py   # skip Expect on empty bodies -> CLEAN
REPRO_STRIP_EXPECT=1 python repro.py                        # before-send Expect strip    -> CLEAN

repro.py (self-contained)

#!/usr/bin/env python3
"""Repro: botocore's Expect: 100-continue handling permanently poisons a pooled
keep-alive connection against RFC 9110-compliant servers (Go net/http: SeaweedFS,
MinIO, ...) that answer a zero-byte PUT with the final response and no interim
"100 Continue".

Mechanism (botocore 1.42.70, awsrequest.py):
  1. add_expect_header() puts "Expect: 100-continue" on every S3 PUT/POST with a
     file-like body, including zero-byte bodies (handlers.py:376).
  2. Go net/http never sends an interim 100 when Content-Length == 0 -- the final
     200 arrives in the interim slot.  AWSConnection._handle_expect_response()
     reads that early final status line and swaps self.response_class to
     functools.partial(AWSHTTPResponse, status_tuple=(..., 200, ...)) on the
     pooled connection.
  3. AWSConnection.request() resets response_class only for requests WITHOUT an
     Expect header, so in a burst of PUTs the poison persists.
  4. The next non-empty PUT on that connection replays the stale status tuple
     without reading the socket: the real status line is fed to the header
     parser -> urllib3 "Failed to parse headers" (MissingHeaderBodySeparatorDefect,
     full real response visible in `unparsed data`) -> header-less 200 with no
     Content-Length -> read-until-EOF -> read timeout stall -> silent retry on a
     fresh connection -> success.  The object is written twice server-side.

Server (terminal 1) -- SeaweedFS 4.18 requires an identity config, injected inline:
    docker run --rm -p 18333:8333 --entrypoint sh chrislusf/seaweedfs:4.18 -c \
      'printf "{\\"identities\\":[{\\"name\\":\\"repro\\",\\"credentials\\":[{\\"accessKey\\":\\"repro\\",\\"secretKey\\":\\"repro\\"}],\\"actions\\":[\\"Admin\\",\\"Read\\",\\"Write\\",\\"List\\",\\"Tagging\\"]}]}" > /tmp/s3.json \
       && exec weed server -s3 -s3.config=/tmp/s3.json -volume.max=100 -master.volumeSizeLimitMB=100'

Client (terminal 2):
    python repro.py                                            # bug reproduced
    BOTO_EXPERIMENTAL__NO_EMPTY_CONTINUE=true python repro.py  # mitigation 1: clean
    REPRO_STRIP_EXPECT=1 python repro.py                       # mitigation 2: clean
    python repro.py --fake-server   # no docker needed; in-process python server
                                    # that mimics Go's RFC-compliant behavior

Tested with Python 3.13.12, boto3 1.42.70, botocore 1.42.70, urllib3 2.6.3.
"""

import collections
import io
import logging
import os
import socket
import sys
import threading
import time
import uuid

import boto3
import botocore
import urllib3
from botocore.config import Config
from botocore.exceptions import ClientError

ENDPOINT = os.environ.get("REPRO_ENDPOINT", "http://127.0.0.1:18333")
READ_TIMEOUT = 5
FAKE = "--fake-server" in sys.argv
STRIP_EXPECT = os.environ.get("REPRO_STRIP_EXPECT", "") == "1"
NO_EMPTY_CONTINUE = os.environ.get("BOTO_EXPERIMENTAL__NO_EMPTY_CONTINUE", "")


def start_fake_go_server():
    """Mimic Go net/http's RFC 9110 Expect handling: interim 100 only when the
    request has a non-empty body; for Content-Length: 0 the final response is
    sent immediately (no 100).  Counts server-side PUTs per path."""
    put_counts = collections.Counter()
    srv = socket.create_server(("127.0.0.1", 0))

    def handle(conn):
        f = conn.makefile("rb")
        try:
            while True:
                request_line = f.readline()
                if not request_line:
                    return
                method, path, _ = request_line.decode().split(" ", 2)
                headers = {}
                while (line := f.readline()) not in (b"\r\n", b"\n", b""):
                    name, _, value = line.decode().partition(":")
                    headers[name.strip().lower()] = value.strip()
                clen = int(headers.get("content-length", 0))
                if headers.get("expect", "").lower() == "100-continue" and clen > 0:
                    conn.sendall(b"HTTP/1.1 100 Continue\r\n\r\n")
                if clen:
                    f.read(clen)
                if method == "PUT":
                    put_counts[path] += 1
                conn.sendall(
                    b"HTTP/1.1 200 OK\r\n"
                    b'Etag: "d41d8cd98f00b204e9800998ecf8427e"\r\n'
                    b"Content-Length: 0\r\n\r\n"
                )
        except (ConnectionError, ValueError, OSError):
            pass
        finally:
            conn.close()

    def accept_loop():
        while True:
            try:
                c, _ = srv.accept()
            except OSError:
                return
            threading.Thread(target=handle, args=(c,), daemon=True).start()

    threading.Thread(target=accept_loop, daemon=True).start()
    return f"http://127.0.0.1:{srv.getsockname()[1]}", put_counts


def make_client(endpoint):
    return boto3.client(
        "s3",
        endpoint_url=endpoint,
        aws_access_key_id="repro",
        aws_secret_access_key="repro",
        region_name="us-east-1",
        config=Config(
            s3={"addressing_style": "path"},
            max_pool_connections=1,  # everything over ONE keep-alive connection
            connect_timeout=5,
            read_timeout=READ_TIMEOUT,
            retries={"mode": "standard", "max_attempts": 3},
            request_checksum_calculation="when_required",
            response_checksum_validation="when_required",
        ),
    )


def main():
    mode = "fake-go-server" if FAKE else "seaweedfs"
    mitigation = (
        "BOTO_EXPERIMENTAL__NO_EMPTY_CONTINUE"
        if NO_EMPTY_CONTINUE
        else ("strip-Expect-hook" if STRIP_EXPECT else "none")
    )
    print(f"python   {sys.version.split()[0]}")
    print(f"boto3    {boto3.__version__}")
    print(f"botocore {botocore.__version__}")
    print(f"urllib3  {urllib3.__version__}")
    print(f"server   {mode}   mitigation: {mitigation}")

    # Capture urllib3's "Failed to parse headers" warning.
    records = []

    class Capture(logging.Handler):
        def emit(self, record):
            records.append(record)

    # urllib3 2.x emits it from urllib3.connection; 1.x from urllib3.connectionpool.
    urllib3_logger = logging.getLogger("urllib3")
    urllib3_logger.addHandler(Capture())
    urllib3_logger.setLevel(logging.WARNING)

    run = uuid.uuid4().hex[:8]  # unique key prefix so reruns don't collide
    put_counts = None
    if FAKE:
        endpoint, put_counts = start_fake_go_server()
        bucket, versioning = "any-bucket", False
    else:
        endpoint, bucket = ENDPOINT, "expect-repro"
        # Separate client for setup: bucket creation/versioning use non-file-like
        # bodies, get no Expect header, and stay off the demo connection.
        setup = make_client(endpoint)
        try:
            setup.create_bucket(Bucket=bucket)
        except ClientError:
            pass  # BucketAlreadyExists / BucketAlreadyOwnedByYou on rerun
        try:
            setup.put_bucket_versioning(
                Bucket=bucket, VersioningConfiguration={"Status": "Enabled"}
            )
            versioning = True
        except ClientError:
            versioning = False  # server build without versioning support
        # Warm up volume assignment so demo PUT timings are clean.
        setup.put_object(Bucket=bucket, Key=f"{run}/warmup", Body=b"w" * 16)
        setup.close()

    s3 = make_client(endpoint)
    if STRIP_EXPECT:

        def strip_expect(request, **kwargs):
            request.headers.pop("Expect", None)

        s3.meta.events.register("before-send.s3", strip_expect)

    results = []

    def put(name, size):
        # io.BytesIO: add_expect_header only fires for file-like bodies
        # (hasattr(body, "read")) -- exactly what upload streams look like.
        t0 = time.monotonic()
        r = s3.put_object(Bucket=bucket, Key=f"{run}/{name}", Body=io.BytesIO(b"x" * size))
        dt = time.monotonic() - t0
        md = r["ResponseMetadata"]
        print(
            f"PUT {name:<8} {size:>5}B -> HTTP {md['HTTPStatusCode']}  "
            f"RetryAttempts={md['RetryAttempts']}  wall={dt:6.2f}s"
        )
        results.append((name, md["RetryAttempts"], dt))
        return r

    print(f"\n--- 3 PUTs over one keep-alive connection (read_timeout={READ_TIMEOUT}s) ---")
    put("a-4kb", 4096)  # healthy connection baseline
    put("b-empty", 0)  # zero-byte PUT: no interim 100 -> connection poisoned
    put("c-4kb", 4096)  # replays stale status tuple -> warning + stall + retry

    print("\n--- urllib3 warnings captured ---")
    parse_warnings = [r for r in records if "Failed to parse headers" in r.getMessage()]
    if parse_warnings:
        for r in parse_warnings:
            print(r.getMessage())
    else:
        print("(none)")

    print("\n--- server-side write count for the final key ---")
    if FAKE:
        n = put_counts[f"/{bucket}/{run}/c-4kb"]
        print(f"fake server observed {n} PUT(s) for /{bucket}/{run}/c-4kb")
        double_write = n
    elif versioning:
        v = s3.list_object_versions(Bucket=bucket, Prefix=f"{run}/c-4kb").get(
            "Versions", []
        )
        print(f"bucket versioning shows {len(v)} version(s) of c-4kb")
        double_write = len(v)
    else:
        print("(versioning unavailable; retry count above implies duplicate PUT)")
        double_write = None

    stalled = results[2][2] >= READ_TIMEOUT
    retried = results[2][1] > 0
    print("\n--- verdict ---")
    if parse_warnings and retried and stalled:
        extra = f", object written {double_write}x" if double_write else ""
        print(
            f"BUG REPRODUCED: header-parse failure + {READ_TIMEOUT}s stall + "
            f"silent retry on final PUT{extra}"
        )
        sys.exit(0)
    if not parse_warnings and not any(retry for _, retry, _ in results):
        extra = f", object written {double_write}x" if double_write else ""
        print(f"CLEAN: no warnings, no stalls, no retries{extra}")
        sys.exit(0)
    print("UNEXPECTED: partial reproduction, see output above")
    sys.exit(1)


if __name__ == "__main__":
    main()

Possible Solution

Reset self.response_class unconditionally at the start of AWSConnection.request() (or clear the stashed tuple once it has been consumed by the response of the request that stashed it). The stashed-tuple mechanism is only valid for the response of the one request that stashed it; it must never survive into the next request on the pooled connection.

Workarounds we validated in the meantime:

• BOTO_EXPERIMENTAL__NO_EMPTY_CONTINUE=true (skips Expect for zero-length bodies) eliminates the empty-body trigger.
• Stripping Expect entirely via a before-send handler (request.headers.pop("Expect", None)) eliminates the whole class; the header is not SigV4-signed (SignedHeaders=host;x-amz-content-sha256;x-amz-date), so this is signature-safe.

Additional Information/Context

• Production impact that led us here: 114 occurrences in 30 days, each a 60-second stall inside a latency-critical path; one 182-file sequential upload spent 242 of its 254 seconds in these stalls. Every occurrence mapped 1:1 to zero-byte→non-empty PUT transitions (directory trees full of empty __init__.py files).
• The silent retry re-sends the request. In our case the PUTs were idempotent overwrites, but the poisoned parse — not the operation's semantics — decides what gets replayed.
• Side observation visible in the repro output: wall=1.01s on the first non-empty PUT is botocore's wait_for_read interim wait burning a full second per PUT against servers that send 100 Continue lazily — a separate cost of the same code path.
• Related reports: MissingHeaderBodySeparatorDefect on PUT request #1833 (same artifact signature, unresolved); MissingHeaderBodySeparatorDefect Warning After Uploading Empty File with Boto3 minio/minio#6540 and MissingHeaderBodySeparatorDefect (again) minio/minio#11245 (same class against MinIO — also a Go net/http server; closed there as client-side).

SDK version used

1.42.70

Environment details (OS name and version, etc.)

• Python 3.13.12 (uv venv), urllib3 2.6.3 - OS: Linux x86_64 (Docker 29.5.2 for the server container) - Server: chrislusf/seaweedfs:4.18 (digest sha256:37ff8b1c2aff…, weed version "30GB 4.18 24805ff47 linux amd64") — but reproducible against any Go net/http server, and against the included pure-Python mock (--fake-server)

[adev-code]

Thanks for reaching out, @ThomasBurgess2000. We tested this using boto3 with a real request to S3 (sent 3 PUTs over one reused connection - 4KB, then empty, then 4KB) and the issue did not appear. It seems S3 sends the 100 Continue even for empty PUTs. We wanted to check if you have tested this directly against S3 with boto3 to see if the issue emerges there as well?

[ThomasBurgess2000]

Thanks for reaching out, @ThomasBurgess2000. We tested this using boto3 with a real request to S3 (sent 3 PUTs over one reused connection - 4KB, then empty, then 4KB) and the issue did not appear. It seems S3 sends the 100 Continue even for empty PUTs. We wanted to check if you have tested this directly against S3 with boto3 to see if the issue emerges there as well?

Thanks for testing! No — we have not tested against AWS S3 directly, and your result is exactly what we'd expect: this cannot reproduce against AWS S3, because S3's front-end chooses to send the interim 100 Continue even for zero-length PUTs. With a 100 in the interim slot, botocore's _handle_expect_response takes its happy path and the connection state stays clean.

The report is about what happens against servers that exercise the omission RFC 9110 §10.1.1 explicitly permits:

A server MAY omit sending a 100 (Continue) response if it has already received some or all of the content for the corresponding request, or if the framing indicates that there is no content.

Go's net/http — the HTTP server under SeaweedFS, MinIO, and a large share of the S3-compatible endpoints people use via endpoint_url — implements exactly that: it never sends an interim 100 when Content-Length: 0 (server.go: if req.ProtoAtLeast(1, 1) && req.ContentLength != 0). So for a zero-length PUT, the final response arrives in the interim slot, _handle_expect_response's non-100 branch swaps self.response_class to a functools.partial(AWSHTTPResponse, status_tuple=...) on the pooled connection — and because request() only resets response_class on the path taken by requests without an Expect header, the stale tuple survives into the next request on that connection and is replayed instead of reading the socket. We verified with a byte-level TCP tap that the server side is fully well-formed throughout (one valid response per request — the server is exercising a MAY, not misbehaving); the corruption exists only in the client's parse state.

That's why we'd frame this as a botocore state-machine bug rather than a server bug, even though AWS S3 itself can never elicit it: any final response consumed in the interim window poisons the connection object for the following request, and the same artifact was previously reported against MinIO (minio/minio#6540, minio/minio#11245 — closed there as client-side) and in #1833.

Two things that may help reproduce without any S3-compatible infrastructure:

• python repro.py --fake-server in the attached script needs no docker — it spins a ~45-line in-process server that implements only the RFC-permitted behavior (interim 100 iff the body is non-empty) and reproduces the full artifact (urllib3 Failed to parse headers with the real response in unparsed_data, read-until-read_timeout stall, silent retry, object written twice).
• The suggested fix — resetting self.response_class unconditionally at the start of AWSConnection.request() — is a no-op against AWS S3 (the stash is already clean there) and only removes the cross-request leak for servers that omit the interim 100.

For impact context: we hit this in production against SeaweedFS at ~114 occurrences/30 days, each a silent 60s stall (default read_timeout) plus a hidden retry, triggered deterministically by zero-byte files (__init__.py) in sequential directory uploads.