RequestSigner holds its event emitter as a weakref.proxy, so a signer outliving its session raises ReferenceError (deterministic reproducer) #3794

Describe the bug

RequestSigner.__init__ stores its event_emitter argument as a weakref.proxy (botocore/signers.py L88-89):

        # We need weakref to prevent leaking memory in Python 2.6 on Linux 2.6
        self._event_emitter = weakref.proxy(event_emitter)

The emitter is normally owned by a botocore.session.Session. If the caller lets the session go out of scope but keeps the RequestSigner, then every later sign() or generate_presigned_url() call raises ReferenceError: weakly-referenced object no longer exists from _choose_signer.

The caller has no way to know this from the API. The constructor docstring types the parameter as botocore.hooks.BaseEventHooks and says only "Extension mechanism to fire events". It does not say the caller must keep the emitter's owner alive for the lifetime of the signer.

The failure is delayed and looks random, because a Session participates in reference cycles. Dropping the last strong reference is not enough. The object survives until a cyclic GC pass runs, which in a long-lived process can be minutes or months after the signer was built. This matches the "worked for months, then failed 266 times in a row" reports in #2970.

This matters most for ElastiCache IAM authentication, because the two documented samples both build the signer and drop the session:

• redis-py, docs/examples/connection_examples.ipynb, "Connecting to a redis instance with ElastiCache IAM credential provider": session = botocore.session.get_session() is a local in __init__, self.request_signer is stored on the instance, and the session is never stored.
• AWS, Tutorial: Configuring Lambda to access ElastiCache in a VPC, step 2: the same shape, in the ElastiCacheIAMProvider sample.

botocore.session.get_session() returns a new Session per call, so nothing at module level keeps that session alive.

#2970 ("very rare ReferenceError") collected reports of this over two years and was closed by the stale bot on 2025-11-24 after a request for a minimal reproducer went unanswered. The closing message asked people to open a new issue that references it. This is that issue: the reproducer below is deterministic and needs no AWS account and no network.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

A RequestSigner built with a valid event_emitter keeps working for as long as the signer itself is alive. Signing should not depend on the caller also keeping a second object alive, and it should not start failing because a GC pass happened to run.

Both documented ElastiCache IAM samples were written against that expectation. Both are broken by the current behaviour.

Current Behavior

Signing raises ReferenceError once the session is garbage-collected. Real traceback from the reproducer below on botocore 1.43.89 and Python 3.13.15 (local paths replaced with /path/to):

Traceback (most recent call last):
  File "/path/to/repro.py", line 39, in <module>
    print(signer.generate_presigned_url(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        request_dict=REQUEST, operation_name="connect", expires_in=900, region_name="eu-west-1",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ))
    ^
  File "/path/to/venv/lib/python3.13/site-packages/botocore/signers.py", line 355, in generate_presigned_url
    self.sign(
    ~~~~~~~~~^
        operation_name,
        ^^^^^^^^^^^^^^^
    ...<4 lines>...
        signing_name,
        ^^^^^^^^^^^^^
    )
    ^
  File "/path/to/venv/lib/python3.13/site-packages/botocore/signers.py", line 152, in sign
    signature_version = self._choose_signer(
        operation_name, signing_type, request.context
    )
  File "/path/to/venv/lib/python3.13/site-packages/botocore/signers.py", line 234, in _choose_signer
    handler, response = self._event_emitter.emit_until_response(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ReferenceError: weakly-referenced object no longer exists

Reproduction Steps

Save as repro.py and run. It fails 10 out of 10 times on botocore 1.43.89. It needs no AWS account and no network: presigning is local HMAC-SHA256, and the credentials are the example pair from the AWS documentation. I confirmed the offline part by patching socket.socket.connect to raise, which changes nothing.

import gc
import os

import botocore.session
from botocore.model import ServiceId
from botocore.signers import RequestSigner

os.environ.update(
    AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE",
    AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
)
os.environ.pop("AWS_PROFILE", None)
os.environ.pop("AWS_DEFAULT_PROFILE", None)

REQUEST = {
    "method": "GET",
    "url": "https://my-cache/?Action=connect&User=my-user",
    "body": {},
    "headers": {},
    "context": {},
}


def build_signer():
    """Build a signer the way the ElastiCache IAM auth samples do."""
    session = botocore.session.Session()
    return RequestSigner(
        service_id=ServiceId("elasticache"),
        region_name="eu-west-1",
        signing_name="elasticache",
        signature_version="v4",
        credentials=session.get_credentials(),
        event_emitter=session.get_component("event_emitter"),
    )


signer = build_signer()   # the session was a local, so it is now unreachable
gc.collect()              # in a real process this happens whenever the GC feels like it
print(signer.generate_presigned_url(
    request_dict=REQUEST, operation_name="connect", expires_in=900, region_name="eu-west-1",
))

Two variants pin down the cause:

• Return (session, signer) from build_signer and keep both. The same call then succeeds and prints a signed URL. The only difference is the strong reference.
• Remove the gc.collect() and run with gc.disable(). The call succeeds. Re-enable the GC, call gc.collect(), and the next call raises. So the trigger is a cyclic GC pass, not dropping the last strong reference, which is why the failure looks random in production.

The redis-py docs example also fails when run verbatim, with the redis.Redis client and the ping() removed so that no network is needed, and gc.collect() added after the provider is built.

Possible Solution

I am not asking for a specific fix, but one constraint is worth stating up front: #3590 already tried deleting the weakref.proxy line and was closed because tests/functional/leak/test_resource_leaks.py::TestDoesNotLeakMemory exceeded its MAX_GROWTH_BYTES budget without it.

One thing that may help separate the two cases: what that leak test exercises is client churn. It calls create_client and free_clients in a loop, so the emitter and the signer are created and dropped together, and the signer never needs to outlive the emitter. A bare RequestSigner that the caller constructed and owns is the opposite case. If those two lifetimes were handled separately, rather than by one proxy covering both, the leak budget and a usable standalone signer would not be in conflict.

Additional Information/Context

Related history:

• very rare ReferenceError #2970, "very rare ReferenceError". Same failing line. Opened 2023-06-15 for a DynamoDB call in a patched Lambda. Four later commenters reported it from ElastiCache IAM auth, three of them naming the redis-py or AWS sample. One of them, in very rare ReferenceError #2970 (comment), found the same workaround: keep a reference to the session. Closed by the stale bot on 2025-11-24.
• Very rare ReferenceError aws/aws-xray-sdk-python#394, opened from very rare ReferenceError #2970.
• [Fix] Remove weakref usage in RequestSigner #3590, "[Fix] Remove weakref usage in RequestSigner". Opened 2025-11-12, closed unmerged 2025-11-20 on the memory-growth grounds above.

What I did instead, in case it helps anyone arriving from a search: in my own ElastiCache IAM credential provider for redis-py, the botocore session is a cached_property on the provider, and the RequestSigner is built inside get_credentials and used in the same scope. The signer therefore never outlives the session.

SDK version used

botocore 1.43.89 (also reproduced on botocore 1.35.99)

Environment details (OS name and version, etc.)

macOS 26.6.2 (Darwin 25.6.0, arm64), Python 3.13.15 (also reproduced on Python 3.14.5). Code inspected on develop at commit b4c8dd9, where botocore/signers.py L88-89 are identical to the released 1.43.89.