[Rust] Lack-of-ack timeout resets on any frame, not on durable progress

[zlata-stefanovic-db]

Summary

The receiver's server-lack-of-ack timeout is implemented as a per-read timeout wrapping each response_grpc_stream.message(). Any received frame resets it — including a stale or duplicate durability_ack_up_to_offset that does not advance durability. A server that dribbles repeated, non-advancing acks keeps the timer alive indefinitely and defers recovery, even though no real progress is being made.

Location

rust/sdk/src/stream/grpc/receiver.rs:75-90 (and the timeout trip at :195-207)

res = tokio::time::timeout(
    Duration::from_millis(options.server_lack_of_ack_timeout_ms),
    response_grpc_stream.message(),
) => res,

Only a full window of total silence trips the timeout; any inbound message — regardless of whether it advances the ack watermark — resets the clock.

Why it matters

The lack-of-ack timeout exists to detect a stalled/half-dead stream where in-flight records are not being durably acknowledged. As written, a server repeating the same cumulative ack (or sending any other frame) postpones recovery forever while later offsets remain unacknowledged — the exact failure the timeout is meant to catch.

Suggested fix

Gate the timer reset on durable progress (the ack watermark advancing), not on frame arrival. A stale/duplicate cumulative ack should not extend the budget.

Reference: how the pure-Go core handles this

The pure-Go reimplementation restarts the ack-silence budget only when the watermark advances:

purego/internal/stream/core.go:1005-1018

// syncLackTimer realigns the ack-silence budget with the in-flight set. Only
// durable progress restarts it: a stale or duplicate cumulative ack must not
// extend the budget, or a server repeating one offset could postpone recovery
// indefinitely while later offsets stay unacknowledged.
syncLackTimer := func(progressed bool) {
    if cs.buf.inFlight() == 0 { disarmLackTimer(); return }
    if progressed { disarmLackTimer() }
    armLackTimer()
}

Both cores correctly gate the timer on having in-flight records; the divergence is purely in what resets it.