Skip to content

fix(payload): prevent parallel prewarming from dropping transactions - #7351

Open
teyrebaz33 wants to merge 1 commit into
tempoxyz:mainfrom
teyrebaz33:fix/prewarming-exhaustion-race
Open

fix(payload): prevent parallel prewarming from dropping transactions#7351
teyrebaz33 wants to merge 1 commit into
tempoxyz:mainfrom
teyrebaz33:fix/prewarming-exhaustion-race

Conversation

@teyrebaz33

Copy link
Copy Markdown

Closes #7284

Problem

With parallel payload building enabled (--builder.parallel, currently hidden/experimental), BestTransactionsPrewarming's coordinator eagerly calls advance() up to 2 * num_threads times during startup. In parallel mode, a worker's Some(tx) result is only sent on transactions_tx once prewarm_transaction finishes (asynchronously), while the coordinator's exhaustion sentinel (None) is sent synchronously the moment the source iterator returns None.

For a source smaller than the eager-fill batch (e.g. a single pending transaction), the coordinator can reach the exhaustion branch and send None before an earlier, still in-flight worker has sent its Some(tx). Since messages are delivered in send order, the consumer can observe the premature None first and stop -- silently dropping a valid, still-pending transaction. The transaction remains known to the pool (eth_getTransactionByHash still returns it) but is never included by that or any subsequent payload, since every payload build hits the same race.

Fix

Tracks how many parallel prewarm workers are currently in flight (in_flight: usize on the coordinator context). The exhaustion sentinel is only sent once the source is exhausted and no workers are in flight. Workers now report completion via a new WorkerDone command, distinct from the existing Advance (which also represents a consumer pulling the next item), so the coordinator can tell the two apart and decide whether it's safe to finalize. The sequential (non-parallel) path is unchanged.

Tests

Added a regression test that deterministically reproduces the original timing: saturates every worker thread with a blocking sleep before submitting a source smaller than the eager-fill batch, forcing the two real transactions' prewarm workers to queue behind the busy threads -- mirroring a slow worker finishing after the coordinator has already observed exhaustion. Verified this test fails reliably against the pre-fix logic and passes reliably (15+ consecutive runs) against the fix. Full cargo test -p tempo-payload-builder prewarming, cargo clippy -p tempo-payload-builder --lib, and cargo fmt --check all clean.

Note: while iterating I also noticed the pre-existing prewarming_does_not_use_shared_worker_state_slot test has some intrinsic flakiness (~15-20%) fully independent of this change (reproducible in isolation on current main). CI's retries = 2 on the default profile absorbs this in practice; flagging it here for visibility rather than trying to fix an unrelated test as part of this PR.

Scope note

This didn't reach a full end-to-end reproduction (real node, real RPC transaction, advance_block()) matching the exact repro in #7284, since parallel payload building is currently a hidden/in-development flag (--builder.parallel, with an explicit "should not be used" warning) not exposed by the existing e2e test harness's node-construction path. The included test instead reproduces the exact race deterministically at the coordination-logic level, which is where the root cause lives.

Thanks to @Osraka for the original diagnosis, reproduction, and fix direction in #7284 -- this implements and verifies that approach.

BestTransactionsPrewarming's coordinator eagerly calls advance() up to
2x the thread count during startup. In parallel mode, a worker's
Some(tx) result is only sent once prewarm_transaction finishes
(asynchronously), while the coordinator's exhaustion sentinel (None) is
sent synchronously the moment the source iterator returns None.

For a source smaller than the eager-fill batch, the coordinator could
reach the exhaustion branch and send None before an earlier, still
in-flight worker sent its Some(tx). Since messages are delivered in
send order, the consumer could observe the premature None first and
stop, silently dropping a valid, still-pending transaction. The
transaction remained known to the pool (visible via
eth_getTransactionByHash) but was never included by that or any
subsequent payload, since every payload build hits the same race.

Fixes this by tracking how many parallel workers are currently in
flight. The exhaustion sentinel is only sent once the source is
exhausted AND no workers are in flight. Workers now report completion
via a distinct WorkerDone command (rather than the existing Advance,
which also represents a consumer pulling the next item), so the
coordinator can tell the two apart and decide whether it's safe to
finalize. The sequential (non-parallel) path is unchanged.

Adds a regression test that deterministically reproduces the original
timing: saturates every worker thread with a blocking sleep before
submitting a source smaller than the eager-fill batch, forcing the
real transactions' prewarm workers to queue behind the busy threads --
mirroring a slow worker finishing after the coordinator has already
observed exhaustion. Verified this test fails reliably against the
pre-fix logic and passes reliably (15+ consecutive runs) against the
fix.

Originally diagnosed by @Osraka in tempoxyz#7284, including root cause and a
narrow fix direction; this implements and verifies that fix along with
a regression test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

payload: parallel prewarming can report exhaustion before in-flight transactions

1 participant