Uncaught exception in socket code: "Cannot read properties of null (reading 'finishWrite')"

[danfuzz]

Version

v18.4.0

Platform

Linux i-00f751a18edbe3eb6.us-west-2.compute.internal 5.15.73-45.135.amzn2022.x86_64 #1 SMP Fri Oct 14 17:47:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

http2 (maybe actually net or tls)

What steps will reproduce the bug?

Apologies that I don't have an easy repro case yet. It's one of those "run my server for a while and then it does this." Near as I can tell, it's along these lines:

1. Connection accepted to an Http2Server (via TLS socket).
2. Session established.
3. Session remains idle for five minutes. (No requests ever made.)
4. My code notices the idleness and calls close() on the session. (Tick/turn concludes.)
5. Boom! Uncaught exception: TypeError: Cannot read properties of null (reading 'finishWrite')

When I get a chance (hopefully soon) I will try to distill down a real example. Thanks for any help you can offer in the meantime.

How often does it reproduce? Is there a required condition?

Unclear. This shows up about once a day on my not-very-active server. It does correspond one-to-one with a call to session.close(). That is, every time my server does this right now, this uncaught exception seems to follow.

What is the expected behavior?

No uncaught exception. That is, either no exception at all, or an exception that can be safely caught and handled without shutting the system down.

What do you see instead?

This uncaught exception:

TypeError: Cannot read properties of null (reading 'finishWrite')
    at JSStreamSocket.finishWrite (node:internal/js_stream_socket:211:12)
    at Immediate.<anonymous> (node:internal/js_stream_socket:196:14)
    at process.processImmediate (node:internal/timers:471:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)

Additional information

This is the actual code which sets up the timeout. The timeout time is 5 minutes, and the logger... call simply writes a structured log line.

    session.setTimeout(Http2Wrangler.#SESSION_TIMEOUT_MSEC, () => {
      logger?.idleTimeout();
      session.close();
    });

Http2Session.close() does accept an optional callback, which is documented as being bound to the close event. However, this code already binds a handler to close (which does not seem to be getting called before the uncaught exception is generated).

[danfuzz]

I have now also managed to reproduce this error on v19.3.0 (on the Mac).

In addition to the error noted in the original report, my repro-case-in-progress also manages to get a second error to be thrown (moments [CORRECTION] after before the original error is thrown), though this one ends up getting emitted in an error event on a socket. Unclear if it is truly meaningful with regards to the original report, but FWIW:

Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at new NodeError (node:internal/errors:400:5)
    at _write (node:internal/streams/writable:322:11)
    at Writable.write (node:internal/streams/writable:337:10)
    at JSStreamSocket.doWrite (node:internal/js_stream_socket:176:19)
    at JSStream.onwrite (node:internal/js_stream_socket:33:57)
    at JSStream.callbackTrampoline (node:internal/async_hooks:130:17)
    at Socket._final (node:net:495:28)
    at callFinal (node:internal/streams/writable:700:12)
    at prefinish (node:internal/streams/writable:712:7)
    at finishMaybe (node:internal/streams/writable:722:5) {
  code: 'ERR_STREAM_WRITE_AFTER_END'

[danfuzz]

Hi again! I'm afraid I still don't have a succinct repro, but I can tell you a bit more about what's going on.

In my codebase, I'm wrapping the raw socket that gets accepted from a net.Server, in order to implement rate-limiting on the write side. The wrapped Duplex stream is set up with allowHalfOpen: true.

When the session times out, (per original report) I call session.close(). This has the effect of doing a little bit of IO to the socket including reading, and when reading is done, endReadableNT() is ultimately called inside internal/streams/readable.js. Surprisingly, this calls through to endWritableNT(), which is only supposed to happen if allowHalfOpen is false. I never set it to false, but in fact at some point, something has indeed set it so, though not in my code. I confirmed that socket.allowHalfOpen === true just before the call to session.close() and that socket.allowHalfOpen === false by the time my wrapped Duplex is asked to close.

After the call to endWritableNT(), something in the HTTP2 code still wants to write some data, and this is what causes the write after end exception to be thrown (per my previous comment). And at that point, I think the internals of the library are kinda messed up, which is what then quickly causes the uncaught exception.

I have an "investigatory workaround" which "fixes" the problem by delaying my code acting on the socket.end() call so that the session can write its final bits. That said, I'm guessing it would be better to figure out why allowHalfOpen suddenly becomes false, as that seems at least wrong-ish. And, even with that fix maybe there is still a bug here in that the original exception should arguably be catch-able (even though it is a consequence of another problem).

[danfuzz]

I figured out what's going on in my code, and though I still think there's a Node bug worth addressing, I am no longer triggering it:

I made a mistake with my log-based instrumentation (from the previous comment), and I was wrong about allowHalfOpen switching from true to false. The actual case was that the unwrapped Socket was always allowHalfOpen: false, and my Duplex wrapper was always allowHalfOpen: true.

What I was right about was that HTTP2 tacitly expects allowHalfOpen: true of its socket/stream, and I am pretty sure simply setting it to false and then proactively closing an Http2Session will ultimately reproduce the uncaught exception described in my original report.

The solution in my code was (of course?) to specify allowHalfOpen: true in my server socket constructor options.

I checked the Node docs for the http2 module, and didn't see anything about allowHalfOpen: true being required. So, in addition to any other action on this bug, maybe it makes sense to note this effective requirement in the docs. (To be clear, I don't know the protocol well enough to understand whether it's truly required by the protocol.)

Thank you for going on this journey with me, and wishing you all a happy new year.

[danfuzz]

Maybe this is the same bug, or at least related?: #35695

That one is closed, but it sure looks similar.

[danfuzz]

BTW, I just ran into another way to get this error:

• Connection accepted.
• Set up a short (e.g. a few seconds) socket timeout.
• Session established.
• Set up a longer (e.g. a minute) session timeout.
• Handle a request on the session.
• Idle.
• Socket timeout fires; callback does socket.destroy().
• Idle.
• Session timeout fires; callback does session.close().
• Exception thrown, Cannot read properties of null (reading 'finishWrite').

[ywave620]

node does not allow manipulation againt http2session.socket, how did you manage to call socket.destroy() @danfuzz

https://nodejs.org/docs/latest-v14.x/api/http2.html#http2_http2session_socket

[danfuzz]

node does not allow manipulation againt http2session.socket, how did you manage to call socket.destroy() @danfuzz

I'm accepting sockets in my code directly, wrapping them (to do rate limiting, timeout, and logging at the low level), then handing the wrapped sockets off to an Http2Server instance. I set up event listeners on the wrapped socket at my layer (or the original socket directly, depending on configuration), not on the one wrapped inside the HTTP2 library. You can find the salient code here:

• https://github.com/danfuzz/lactoserv/blob/main/src/network-protocol/private/TcpWrangler.js#L105
• https://github.com/danfuzz/lactoserv/blob/main/src/network-protocol/private/TcpWrangler.js#L192
• https://github.com/danfuzz/lactoserv/blob/main/src/network-protocol/private/Http2Wrangler.js#L58
• https://github.com/danfuzz/lactoserv/blob/main/src/builtin-services/private/RateLimitedStream.js#L337

[ywave620]

How did you

handing the wrapped sockets off to an Http2Server instance

could you point out which lines of code is responsible for this? (I glanced your code but couldn't find out)

[ywave620]

Actually, I have managed to do the same thing but in a different way.

1. create a http(s) server without listening at any port, called httpServer
2. given a connected socket(maybe from the 'connection' of net.Server), called sock, invoke httpServer.emit('connection', sock)