Summary
When the messaging service fails to insert a gathered L1-handler message into the pool (or commit_message fails), it breaks the batch without committing the checkpoint — but the stream's in-memory cursor has already advanced past the whole gather range. Within the running process the skipped message (and the rest of that batch's tail) is never re-gathered; only a restart (which re-reads the older persisted DB checkpoint) picks it back up.
The service.rs comment ("will retry on next gather") is misleading: the next tick gathers from the advanced in-memory from_block, not the checkpoint.
Where
crates/messaging/src/service.rs (batch loop, ~L186-220) — break on pool/commit error without reconciling the cursor.crates/messaging/src/stream/mod.rs:246 — in-memory from_block advances to to_block + 1 as soon as gather yields, regardless of whether the consumer accepted the messages.
Impact
A single transiently-dropped message is silently lost until the next process restart. (The permanent cascade where every subsequent message is then rejected was fixed separately by ungating the L1-handler nonce in the pool; this issue is the remaining silent single-message loss.)
Fix idea
Don't advance the in-memory cursor past an uncommitted message — reconcile the stream cursor with the persisted checkpoint on batch failure (e.g. signal the consumed high-water back to the stream, or re-seed from_block/from_tx_index from the checkpoint on the next tick). Add a stream test where the consumer rejects a message and assert it's re-covered on the next gather.
Follow-up split from the L1-handler nonce-gate fix (case B3a in the investigation).
Summary
When the messaging service fails to insert a gathered L1-handler message into the pool (or
commit_messagefails), itbreaks the batch without committing the checkpoint — but the stream's in-memory cursor has already advanced past the whole gather range. Within the running process the skipped message (and the rest of that batch's tail) is never re-gathered; only a restart (which re-reads the older persisted DB checkpoint) picks it back up.The
service.rscomment ("will retry on next gather") is misleading: the next tick gathers from the advanced in-memoryfrom_block, not the checkpoint.Where
crates/messaging/src/service.rs(batch loop, ~L186-220) —breakon pool/commit error without reconciling the cursor.crates/messaging/src/stream/mod.rs:246— in-memoryfrom_blockadvances toto_block + 1as soon asgatheryields, regardless of whether the consumer accepted the messages.Impact
A single transiently-dropped message is silently lost until the next process restart. (The permanent cascade where every subsequent message is then rejected was fixed separately by ungating the L1-handler nonce in the pool; this issue is the remaining silent single-message loss.)
Fix idea
Don't advance the in-memory cursor past an uncommitted message — reconcile the stream cursor with the persisted checkpoint on batch failure (e.g. signal the consumed high-water back to the stream, or re-seed
from_block/from_tx_indexfrom the checkpoint on the next tick). Add a stream test where the consumer rejects a message and assert it's re-covered on the next gather.Follow-up split from the L1-handler nonce-gate fix (case B3a in the investigation).