perf(dump): refactor async iterator merging with promise slots

[MrOrz]

Refactor the async iterator merging logic to use a more efficient promise slot pattern with NEVER_RESOLVE and index-based tracking. This improves performance by avoiding O(N) searches and frequent array allocations.

[gemini-code-assist]

Code Review

This pull request replaces the mergeAsyncIterables function with mergeAsyncGenerators, implementing a promise slot pattern to merge multiple async generators. The review feedback identifies opportunities to improve throughput by refilling promise slots before yielding and to prevent resource leaks by ensuring source generators are properly closed via a try...finally block.

[MrOrz]

Why it keeps OOM? So frustrating
https://github.com/cofacts/opendata/actions/runs/24854282461
https://github.com/cofacts/opendata/actions/runs/24856258425

[MrOrz]

@claude do you have any idea why it keeps OOM when scanning the index and output?
The original loop does not OOM, but my implementation OOMs.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[nonumpa]

The likely cause of the OOM is the NEVER_RESOLVE sentinel in mergeAsyncGenerators().

In the previous implementation, finished iterators were removed from the Promise.race() set with entries.splice(...).

In the new implementation, a finished slot is replaced with a promise that never settles:

const NEVER_RESOLVE = new Promise(() => {});
promises[winner.idx] = NEVER_RESOLVE;

while the loop continues to run:

await Promise.race(promises);

That means the finished slot stays in the race forever. If Promise.race(promises) keeps attaching reactions to every still-pending promise, those reactions can accumulate on the same never-settling promise over the full scan/output run, which would explain the memory growth and the OOM.

The old loop did not have this behavior because finished iterators were removed entirely instead of being kept as permanently pending slots.

[MrOrz]

But NEVER_RESOLVE gets only instantiated once, all loop executions share exactly the same NEVER_RESOLVE instance. It should only occupy 1 slot of memory, the others are just references to the same promise.