Unguarded getsockname()[1] on async connection open → unretryable TypeError: 'NoneType' object is not subscriptable (deterministic repro; re: #730, #1057)

[riley-deliberately]

AI DISCLOSURE

I used Claude Code to generate this repro. Its very hard to verify that the socketname is really "None" in the running system, but by patching this code in my production deployment the error did go away...

The Issue

AsyncBolt.__init__ (and several sibling sites in the async connect path) call self.socket.getsockname()[1] with no None guard. For the async driver, BoltSocket.getsockname() is self._writer.transport.get_extra_info("sockname"), which asyncio returns as None once the transport's socket is gone — i.e. when a load balancer drops a freshly-opened connection (we see this constantly on Neo4j Aura, most often on the connection opened for a routing-table refresh).

Because the result is a bare TypeError rather than ServiceUnavailable/SessionExpired, the driver's transaction-retry never catches it, so a transient connection drop becomes a hard, non-retryable crash with a misleading message.

This is the same defect previously reported in #730 (2022) and #1057 (2024); both were closed because the condition could not be reproduced (#1057: "I was not able to reproduce this condition locally", and "feel free to reopen … while providing the additional information requested"). This report provides a deterministic, minimal reproduction — the missing piece — plus a verification that guarding the None fixes it.

Versions

• Driver: reproduced on neo4j==5.28.4; the unguarded line is identical in the latest 6.2.0 (_bolt.py:159).
• Server: Neo4j Aura in production; the repro below works against any local Neo4j.
• Python 3.12.

Production traceback (matches #730 / #1057)

File ".../neo4j/_async/io/_pool.py", line 792, in fetch_routing_info
    cx = await self._acquire(address, auth, deadline, None)
File ".../neo4j/_async/io/_pool.py", line 711, in opener
    return await AsyncBolt.open(...)
File ".../neo4j/_async/io/_bolt.py", line 413, in open
    connection = bolt_cls(...)
File ".../neo4j/_async/io/_bolt.py", line 156, in __init__
    self.local_port = self.socket.getsockname()[1]
                      ~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

Deterministic reproduction

The real-world trigger (an LB dropping a brand-new connection) is hard to race — presumably why the prior issues stalled. But the condition asyncio actually exposes is simple: get_extra_info("sockname") returns None once the transport's socket is gone. The script marks the first freshly-opened socket as "dropped" right after its handshake — exactly the production sequence — so the crash lands on the same line as #730/#1057. It then applies a one-line guard and shows the same drop become retryable and self-heal. Needs only pip install neo4j:

docker run --rm -d -p 7687:7687 -e NEO4J_AUTH=neo4j/password neo4j:5
NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=password python repro.py

# repro.py
import asyncio
import os
import traceback

import neo4j._async.io._bolt as bolt_mod
import neo4j._async.io._bolt_socket as io_sock
import neo4j._async_compat.network._bolt_socket as base_sock
from neo4j import AsyncGraphDatabase
from neo4j.exceptions import ServiceUnavailable

URI = os.environ.get("NEO4J_URI", "bolt://localhost:7687")
USER = os.environ.get("NEO4J_USER", "neo4j")
PASSWORD = os.environ.get("NEO4J_PASSWORD", "password")

# Make the sockets we choose report sockname == None -- the value asyncio yields
# for a transport whose socket is already gone.
_orig_getsockname = base_sock.AsyncBoltSocketBase.getsockname
_orig_connect = io_sock.AsyncBoltSocket.connect.__func__  # unwrap classmethod
_dead_sockets: set[int] = set()
_arm = {"on": False}


def _getsockname(self):
    return None if id(self) in _dead_sockets else _orig_getsockname(self)


async def _connect(cls, *args, **kwargs):
    sock, *rest = await _orig_connect(cls, *args, **kwargs)
    if _arm["on"]:
        _dead_sockets.add(id(sock))  # LB drops THIS freshly-opened connection
        _arm["on"] = False           # one-shot: only the first connection
    return (sock, *rest)


base_sock.AsyncBoltSocketBase.getsockname = _getsockname
io_sock.AsyncBoltSocket.connect = classmethod(_connect)


async def _run_query() -> int:
    driver = AsyncGraphDatabase.driver(URI, auth=(USER, PASSWORD))
    try:
        records, _, _ = await driver.execute_query("RETURN 1 AS n")
        return records[0]["n"]
    finally:
        await driver.close()


async def main() -> int:
    print(f"neo4j driver {__import__('neo4j').__version__}  |  target {URI}\n")

    print("SCENARIO 1 - stock driver, first connection's socket dropped:")
    _dead_sockets.clear(); _arm["on"] = True
    try:
        await _run_query()
        print("  UNEXPECTED: query succeeded\n"); s1 = False
    except TypeError as exc:
        line = next((l.strip() for l in traceback.format_exc().splitlines()
                     if "getsockname()[1]" in l), "?")
        print(f"  REPRODUCED -> TypeError: {exc}\n  at: {line}\n"
              "  (a bare TypeError -- the driver's retry never catches it)\n")
        s1 = True

    # The fix: reclassify the None-sockname subscript crash as retryable.
    _orig_init = bolt_mod.AsyncBolt.__init__

    def _guarded_init(self, *args, **kwargs):
        try:
            _orig_init(self, *args, **kwargs)
        except TypeError as exc:
            if "subscriptable" not in str(exc):
                raise
            raise ServiceUnavailable(
                "socket dropped before use (sockname unavailable); retrying"
            ) from exc

    bolt_mod.AsyncBolt.__init__ = _guarded_init

    print("SCENARIO 2 - same drop, with the None-guard applied:")
    _dead_sockets.clear(); _arm["on"] = True
    try:
        result = await _run_query()
        print(f"  HEALED -> driver retried on a fresh connection, returned {result}\n")
        s2 = result == 1
    finally:
        bolt_mod.AsyncBolt.__init__ = _orig_init

    print("RESULT:", "PASS" if (s1 and s2) else "FAIL")
    return 0 if (s1 and s2) else 1


if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

Output (driver 5.28.4, local Neo4j):

SCENARIO 1 - stock driver, first connection's socket dropped:
  REPRODUCED -> TypeError: 'NoneType' object is not subscriptable
  at: self.local_port = self.socket.getsockname()[1]
  (a bare TypeError -- the driver's retry never catches it)

SCENARIO 2 - same drop, with the None-guard applied:
Transaction failed and will be retried in 0.83s (socket dropped before use (sockname unavailable); retrying)
  HEALED -> driver retried on a fresh connection, returned 1

RESULT: PASS

All unguarded getsockname()[1] sites (any can be hit, depending on when the socket dies)

In 5.28.4 (same pattern in 6.2.0):

• _async/io/_bolt.py:156 — AsyncBolt.__init__ (the production crash site)
• _async/io/_bolt.py:381, 399, 407 — AsyncBolt.open close / auth-failure debug logging
• _async/io/_bolt_socket.py:225 — _handshake (a one-liner getsockname = lambda self: None lands here instead)
• _async/io/_bolt_socket.py:343, 361 — connect
• _async/io/_common.py:41 — AsyncOutbox

Why it matters

This is a transient condition (connection dropped mid/just-after open — normal during Aura leader elections / routing refresh; #1057 also noted it "manifested itself under high load, and when leader elections were happening"). The driver already retries transient connection failures, but a bare TypeError is not in the retryable set, so it escapes and kills the operation instead of re-opening.

Note: get_extra_info("sockname") returns None simply because the transport's socket is already None/closed — there isn't necessarily an OSError at the getsockname call itself (which may be why the async-sockname OSError-surfacing branch from #1057 didn't pan out).

Suggested fix

Guard the None and raise a retryable driver error instead of subscripting, e.g. in AsyncBolt.__init__:

sockname = self.socket.getsockname()
if sockname is None:
    raise ServiceUnavailable(
        "Connection's socket was closed before it could be used "
        "(sockname unavailable); will retry on a fresh connection"
    )
self.local_port = sockname[1]

and treat the debug-logging sites defensively (local_port = sockname[1] if sockname else -1). As Scenario 2 shows, this lets the existing retry transparently re-open — the behavior every reporter of this defect actually wants.

[riley-deliberately]

I feel like catching the NoneType exception is a BAD fix to this issue, which is why i'm not proposing a PR. Probably best to detect an empty socketname and raise something specific?

[robsdedude]

Hi and thanks for the report.

Can you please

1. let me know which platform you're running on: OS + version (maybe even architecture),
2. let me know if you can reproduce the issue consistently (without mocking driver internals) or if it's intermittent (if so, under which circumstances?),
3. as well as share driver debug logs with me from around when the exception occurred?
4. Further, I'd like you to confirm that you're using asyncio and not any other async runtime.

[riley-deliberately]

1. Platform: OS + version + architecture

• Runtime: AWS ECS Fargate, base image python:3.12-slim → Debian 12 (bookworm), Python 3.12, Linux.
• Architecture: amd64... system architecture is that this is a distributed processing engine that scales horizontally (pretty large)
• Driver version: we run neo4j 5.28.4

2. Reproducible vs intermittent

Intermittent, and you cannot reproduce it consistently against a live DB without controlling the socket. It only fires when Neo4j Aura (or its LB) drops a brand-new connection mid-open, which the driver hits most often during a routing-table refresh

3. Driver debug logs

I don't have these. It would cost me a non-0 amount of money to have this running until the next error happens. If you have ideas to do this cheap i'm open to it.

4. asyncio confirmation — yes

Confirmed asyncio, no other runtime.

[robsdedude]

Thanks for the added details.

First for completeness' and documentation's sake my analysis:

Root cause analysis

I think your LLM hallucinated in regards to the reason for the error. I've tried all sorts of mean things with the connection during the opening: dropping packets, delaying them indefinitely, injecting RST/FIN packets. No None to be observed.

asyncio (at least on Linux) never appears to be setting extras["sockname"] to None retroactively. When the transport is created (which is not async and happens after the socket connection has been established), asyncio will attempt to call socket.getsockname and cache the return value. See one of the following code snippets (depending on which IO event solution is being chosen):

https://github.com/python/cpython/blob/v3.12.13/Lib/asyncio/proactor_events.py#L31-L36
https://github.com/python/cpython/blob/v3.12.13/Lib/asyncio/selector_events.py#L781-L785

Both silently swallow OSErrors, which is different from the sync driver that lets these errors bubble up.

As said, I was not even able to reproduce getting such an OSError. So I'll have to rely on docs here: https://www.man7.org/linux/man-pages/man2/getsockname.2.html#ERRORS

This describes 5 error cases:

• EBADF: The argument sockfd is not a valid file descriptor.
• EFAULT: The addr argument points to memory not in a valid part of the process address space.
• EINVAL: addrlen is invalid (e.g., is negative).
• ENOBUFS: Insufficient resources were available in the system to perform the operation.
• ENOTSOCK: The file descriptor sockfd does not refer to a socket.

EBADF and ENOTSOCK cannot happen (assuming CPython doesn't have weird bugs), since the driver only ever uses socket handlers as received from CPython which encapsulate the file descriptor. Neither does the driver ever mess with the raw FDs.

EFAULT and EINVAL would be straight up bugs in CPython.

So I bet my money on ENOBUFS: the client's network buffers are too full to serve the request. So this might be about too much load on the client rather than the DBMS cluster.

Proposed solution

I've crated a PR that calls getsockname on the raw socket before passing it into asyncio and caches the result. This way any OSError will just propagate instead of leading to a deferred TypeError.

If you have a staging environment I'd really appreciate if you could give this driver branch a spin and see if it makes the issue go away for you. As written in the analysis section: this error is hard for me to reproduce, so I'm hoping you can help with verifying the fix.

You can install the patched driver with

pip install git+https://github.com/neo4j/neo4j-python-driver.git@fix/own-sockname-caching

[riley-deliberately]

I can try it give me a couple days.