feat(engine): add persistence completion wake channel

[cakevm]

Based on #20616 I noticed that there is kind of a fixed delay until a block(s) is finally persisted ~500ms. The idea in the PR is to add channel to notify the listener when the persistence is finished. This allows to wake up before the timeout is triggered, since it may that there is no new engine event that will trigger the wakeup. The timeout is still there as safety net. The latency is still higher than I expected, but not fixed and better as before.

Overall the next message is proceed when:

• An incoming engine message
• A persistence completion signal <-- NEW
• A fallback timeout (500ms)

reth/crates/engine/tree/src/tree/mod.rs

Lines 1203 to 1218 in ad37490

	fn try_recv_engine_message(
	&self,
	) -> Result<Option<FromEngine<EngineApiRequest<T, N>, N::Block>>, RecvError> {
	if self.persistence_state.in_progress() {
	// try to receive the next request with a timeout to not block indefinitely
	match self.incoming.recv_timeout(std::time::Duration::from_millis(500)) {
	Ok(msg) => Ok(Some(msg)),
	Err(err) => match err {
	RecvTimeoutError::Timeout => Ok(None),
	RecvTimeoutError::Disconnected => Err(RecvError),
	},
	}
	} else {
	self.incoming.recv().map(Some)
	}
	}

---

I used https://github.com/cakevm/reth-stale-direct-db to test the latency:
Before (usually it is around 500ms, I picked some worst case). latency_ms=501 is the latency between newHead and the persisted event from #20616 is received.

2025-12-24T09:06:50.079711Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081475 hash=0x7bc412b263752e1e213f333d075e11ca417753e19ded09baaf7486e09198eb66 pending=0
2025-12-24T09:06:50.581421Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081475 hash=0x7bc412b263752e1e213f333d075e11ca417753e19ded09baaf7486e09198eb66 blocks_to_flush=1 max_latency_ms=501
2025-12-24T09:06:50.581441Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081475 rpc_block_hash=0x7bc412b263752e1e213f333d075e11ca417753e19ded09baaf7486e09198eb66 latency_ms=501
2025-12-24T09:06:50.581674Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081475 db_last_block=24081475
2025-12-24T09:06:50.581682Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081475 db_block_hash=0x7bc412b263752e1e213f333d075e11ca417753e19ded09baaf7486e09198eb66
2025-12-24T09:07:01.925165Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081476 hash=0xeded8f015ed2de0688cfd26a9948dfb875aa63392b25c8b0ebcd15cff468b9f3 pending=0
2025-12-24T09:07:04.430847Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081476 hash=0xeded8f015ed2de0688cfd26a9948dfb875aa63392b25c8b0ebcd15cff468b9f3 blocks_to_flush=1 max_latency_ms=2505
2025-12-24T09:07:04.430910Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081476 rpc_block_hash=0xeded8f015ed2de0688cfd26a9948dfb875aa63392b25c8b0ebcd15cff468b9f3 latency_ms=2505
2025-12-24T09:07:04.431275Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081476 db_last_block=24081476
2025-12-24T09:07:04.431289Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081476 db_block_hash=0xeded8f015ed2de0688cfd26a9948dfb875aa63392b25c8b0ebcd15cff468b9f3
2025-12-24T09:07:14.299734Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081477 hash=0x6b4c9efb0c0f34adfa65c1adc30995e93cd89a4e41668847f7fafe6c4b3ab903 pending=0
2025-12-24T09:07:15.301366Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081477 hash=0x6b4c9efb0c0f34adfa65c1adc30995e93cd89a4e41668847f7fafe6c4b3ab903 blocks_to_flush=1 max_latency_ms=1001
2025-12-24T09:07:15.301412Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081477 rpc_block_hash=0x6b4c9efb0c0f34adfa65c1adc30995e93cd89a4e41668847f7fafe6c4b3ab903 latency_ms=1001
2025-12-24T09:07:15.301819Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081477 db_last_block=24081477

After:

2025-12-24T09:22:25.221447Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081552 hash=0xb2fbe9c65230f2d5ffdff80354059577b4b53fc8a9d54636678ab97dcb79c78b pending=0
2025-12-24T09:22:25.635592Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081552 hash=0xb2fbe9c65230f2d5ffdff80354059577b4b53fc8a9d54636678ab97dcb79c78b blocks_to_flush=1 max_latency_ms=414
2025-12-24T09:22:25.635610Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081552 rpc_block_hash=0xb2fbe9c65230f2d5ffdff80354059577b4b53fc8a9d54636678ab97dcb79c78b latency_ms=414
2025-12-24T09:22:25.635781Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081552 db_last_block=24081552
2025-12-24T09:22:25.635786Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081552 db_block_hash=0xb2fbe9c65230f2d5ffdff80354059577b4b53fc8a9d54636678ab97dcb79c78b
2025-12-24T09:22:37.255259Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081553 hash=0xe50fc20cb85f865d8693621041c12a661bb52a237bd56e91643794aa74d7c09c pending=0
2025-12-24T09:22:38.216540Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081553 hash=0xe50fc20cb85f865d8693621041c12a661bb52a237bd56e91643794aa74d7c09c blocks_to_flush=1 max_latency_ms=961
2025-12-24T09:22:38.216558Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081553 rpc_block_hash=0xe50fc20cb85f865d8693621041c12a661bb52a237bd56e91643794aa74d7c09c latency_ms=961
2025-12-24T09:22:38.216746Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081553 db_last_block=24081553
2025-12-24T09:22:38.216754Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081553 db_block_hash=0xe50fc20cb85f865d8693621041c12a661bb52a237bd56e91643794aa74d7c09c
2025-12-24T09:22:49.634410Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081554 hash=0xd6d28038b8c59d609426c5d5ce3d7a0bc186160da4f986e7f101dd58188afcca pending=0
2025-12-24T09:22:49.981407Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081554 hash=0xd6d28038b8c59d609426c5d5ce3d7a0bc186160da4f986e7f101dd58188afcca blocks_to_flush=1 max_latency_ms=346
2025-12-24T09:22:49.981455Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081554 rpc_block_hash=0xd6d28038b8c59d609426c5d5ce3d7a0bc186160da4f986e7f101dd58188afcca latency_ms=347
2025-12-24T09:22:49.981659Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081554 db_last_block=24081554
2025-12-24T09:22:49.981666Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081554 db_block_hash=0xd6d28038b8c59d609426c5d5ce3d7a0bc186160da4f986e7f101dd58188afcca
2025-12-24T09:23:00.847461Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081555 hash=0xbf9c4d260bcb1cb4b1f4a7005ed1a797a2b2a376638dc7f5d2bba8b3904e77a1 pending=0
2025-12-24T09:23:01.269566Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081555 hash=0xbf9c4d260bcb1cb4b1f4a7005ed1a797a2b2a376638dc7f5d2bba8b3904e77a1 blocks_to_flush=1 max_latency_ms=422
2025-12-24T09:23:01.269586Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081555 rpc_block_hash=0xbf9c4d260bcb1cb4b1f4a7005ed1a797a2b2a376638dc7f5d2bba8b3904e77a1 latency_ms=422
2025-12-24T09:23:01.269834Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081555 db_last_block=24081555
2025-12-24T09:23:01.269844Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081555 db_block_hash=0xbf9c4d260bcb1cb4b1f4a7005ed1a797a2b2a376638dc7f5d2bba8b3904e77a1
2025-12-24T09:23:14.324459Z  INFO reth_stale_direct_db::monitor: New head received block_number=24081556 hash=0x6b4574354f091ef6c257b73a148dead9a0c60ce9175a25c5257638d201f01838 pending=0
2025-12-24T09:23:14.967566Z  INFO reth_stale_direct_db::monitor: New latest persisted block received block_number=24081556 hash=0x6b4574354f091ef6c257b73a148dead9a0c60ce9175a25c5257638d201f01838 blocks_to_flush=1 max_latency_ms=643
2025-12-24T09:23:14.967596Z  INFO reth_stale_direct_db::monitor: Verifying block block_number=24081556 rpc_block_hash=0x6b4574354f091ef6c257b73a148dead9a0c60ce9175a25c5257638d201f01838 latency_ms=643
2025-12-24T09:23:14.967878Z  INFO reth_stale_direct_db::monitor: Consistency check passed (256 block hashes accessible) block_number=24081556 db_last_block=24081556
2025-12-24T09:23:14.967886Z  INFO reth_stale_direct_db::monitor: Block hash matches block_number=24081556 db_block_hash=0x6b4574354f091ef6c257b73a148dead9a0c60ce9175a25c5257638d201f01838

Switched around 13:30 to this branch (before #20616). I do not have a good explanation for the outliners. But they seem not related to this PR and have been around before. Overall the access time to new state looks better.

PS: to_tree_tx use now crossbeam to make use of select!.