Skip to content

fix(asyncengine): wake both callbacks on Event.Error (#709) - #710

Open
cryo2010 wants to merge 2 commits into
status-im:masterfrom
cryo2010:fix/709-event-error-wakes-both-callbacks
Open

fix(asyncengine): wake both callbacks on Event.Error (#709)#710
cryo2010 wants to merge 2 commits into
status-im:masterfrom
cryo2010:fix/709-event-error-wakes-both-callbacks

Conversation

@cryo2010

@cryo2010 cryo2010 commented Aug 22, 2026

Copy link
Copy Markdown

Fixes #709.

Problem

poll() only resumes an I/O callback for its own direction: Event.Error wakes a side only when the event is exactly {Error}. When the kernel coalesces the error with the opposite direction — e.g. {Read, Error} (send buffer full, so no EPOLLOUT) or {Write, Error} (nothing to read) — the other side's callback is never scheduled. Its readLoop/writeStreamLoopnever resumes, so a parkedreadOnce/write` future is never completed: the consumer hangs forever and the future leaks.

This is the leak fixed by #672 and re-introduced by #700, which reverted #672 because its blunt form caused a kqueue double-callback crash and deferred "a proper fix." This is an attempt at a final fix.

Change

  1. chronos/internal/asyncengine.nimpoll() schedules the reader on {Read, Error} and the writer on {Write, Error}, so neither side is left un-resumed:
   if {Event.Read,  Event.Error} * events != {}: ...schedule reader...
   if {Event.Write, Event.Error} * events != {}: ...schedule writer...
  1. chronos/transports/stream.nimreaderCb now tolerates the resulting spurious/duplicate wakeups (kqueue reports {Read, Error} and {Write, Error} as separate events for the same fd): it skips reading when {Closed, ReadPaused} is already set, and uses if readIntoBuffer(): instead of doAssert progress. completeReader() still runs unconditionally so the close-injected event and genuine EOF/error still complete the parked read. writeStreamLoop is already idempotent (an empty queue simply re-pauses), so it needs no change.
  2. tests/testbugs.nim — restores the dispatcher-level regression test (both reader and writer callbacks must fire on {Read, Error}). It fails on master (writerFlag == false) and passes with this change, on both the epoll and poll engines. Revert "Always report Event.Error to both read/write callbacks" #700's kqueue "Reader notification after buffer got full" test continues to pass.

Verification

Deterministic dispatcher test, before/after:

# master:          readerFlag=true writerFlag=false   -> FAILED
# with this fix:   readerFlag=true writerFlag=true     -> OK   (epoll and poll)

Reproduced the leak end-to-end under Valgrind (Docker nimlang/nim:2.2.10, Linux aarch64, Valgrind 3.24.0, ORC + -d:useMalloc, epoll engine). A consumer coroutine awaits a future resolved by a writer callback (standing in for
writeStreamLoop), with the fd held in {Read, Error}; the repro is in #709.

Before (on master): parked-future.finished = false

==119== 376 (136 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 9 of 15
==119==    by 0x10D963: nimNewObj (arc.nim:116)
==119==    by 0x125F03: asyncfutures::newFutureImpl (asyncfutures.nim:80)
==119==    by 0x128D4F: leakrepro::run (leakrepro.nim:20)
==119==
==119== LEAK SUMMARY:
==119==    definitely lost: 136 bytes in 1 blocks
==119==    indirectly lost: 240 bytes in 3 blocks
==119==      possibly lost: 12,864 bytes in 11 blocks
==119==    still reachable: 0 bytes in 0 blocks
==119== ERROR SUMMARY: 12 errors from 12 contexts (suppressed: 0 from 0)

After (this PR): consumer resumed, parked-future.finished = true

==119== LEAK SUMMARY:
==119==    definitely lost: 0 bytes in 0 blocks
==119==    indirectly lost: 0 bytes in 0 blocks
==119==      possibly lost: 12,864 bytes in 11 blocks
==119==    still reachable: 0 bytes in 0 blocks
==119== ERROR SUMMARY: 11 errors from 11 contexts (suppressed: 0 from 0)

The definitely/indirectly lost blocks (the orphaned future and its suspended
consumer frame) go from 136/240 to 0/0. The possibly lost
12,864 bytes in 11 blocks is unchanged between the two builds — ORC/dispatcher
interior-pointer bookkeeping, unrelated to this bug.

Suites pass on both engines: testbugs, teststream, testasyncstream,
testhttpclient, testshttpserver (TLS), testserver, testfut, testsoon,
testsignal, testdatagram.

poll() only resumed an I/O callback for its own direction: Event.Error
woke a side only when the event was exactly {Error}. When the kernel
coalesced the error with the opposite direction (e.g. {Read, Error} with
a full send buffer, or {Write, Error} with no data to read), the other
side's callback was never scheduled. Its readLoop/writeStreamLoop never
resumed, leaking the parked readOnce/write future and hanging the
consumer.

This restores the fix from status-im#672 (reverted by status-im#700 for causing a kqueue
double-callback crash) without the crash:

- poll(): schedule the reader on {Read, Error} and the writer on
  {Write, Error}, so neither side is left un-resumed.
- readerCb: tolerate the resulting spurious/duplicate wakeups (kqueue
  reports {Read, Error} and {Write, Error} as separate events for the
  same fd). Skip reading when {Closed, ReadPaused} is already set and
  replace `doAssert progress` with `if readIntoBuffer():`;
  completeReader() still runs unconditionally so close-injected events
  and genuine EOF/error still complete the parked read. writeStreamLoop
  is already idempotent.

Adds the dispatcher-level regression test back (both callbacks must fire
on {Read, Error}); status-im#700's kqueue "Reader notification after buffer got
full" test continues to pass.
@cryo2010
cryo2010 marked this pull request as draft August 22, 2026 05:33
@cryo2010
cryo2010 marked this pull request as ready for review August 22, 2026 05:40
@arnetheduck

Copy link
Copy Markdown
Member

Thanks for the PR - this however ended up a duplicate of #705 - the new test looks interesting though, if you would care to resolve the conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Closing a transport with a parked readOnce and data in flight leaks the read future

2 participants