Skip to content

Backoff Configuration Bug in WorkerServiceClient #35

@aptgetgit

Description

@aptgetgit

Description

Duplicate line in WorkerServiceClient::new() where with_max_elapsed_time gets called twice, second call just overwrites the first. This sets max_elapsed_time to None and retries never stop:

let backoff = ExponentialBackoffBuilder::new()
    .with_initial_interval(Duration::from_millis(100))
    .with_max_elapsed_time(Some(Duration::from_secs(10)))
    .with_max_elapsed_time(None)  // overwrites previous value
    .build();

Breaks

Breaks the failover logic in wait_tasks():

async fn wait_tasks(&self, proof_id: String, ids: &[String]) -> anyhow::Result<()> {
    // Try to create a subscriber first, otherwise use the failover mechanism.
    match self.create_subscriber(proof_id.clone()).await {
        Ok((sub_tx, mut sub_rx)) => { /* use subscriber */ }
        Err(err) => {
            tracing::warn!("Failed to create subscriber, using failover mechanism");
            self.wait_tasks_failover(proof_id, ids).await
        }
    }
}

Inside create_subscriber(), the buggy backoff is used for open_sub:

let response = backoff::future::retry(backoff.clone(), || async {
    connection
        .clone()
        .open_sub(request.clone())
        .await
        .map_err(status_to_backoff_error)
})
.await?;  // blocks here indefinitely with max_elapsed_time: None

When open_sub hits transient errors, it should timeout and fall back to wait_tasks_failover(). But with infinite retries, the error branch never runs and the polling fallback never kicks in.

Fix

Just remove the duplicate line:

let backoff = ExponentialBackoffBuilder::new()
    .with_initial_interval(Duration::from_millis(100))
    .with_max_elapsed_time(Some(Duration::from_secs(10)))
    .build();

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions