Skip to content

A throttle never says how long, anywhere an operator will look #13

Description

@sshine

From docs/FINDINGS.md. Found by running the Go provider this replaces, and correlated
against this codebase rather than acted on. Sources: external-dns-desec-provider#26,
a year of production fixes, its review, and sshine/external-dns-desec-provider#2,
another operator's account of the same failure from a four-cluster deployment.
This is not a decision.

The mechanism

desec::Error::RateLimited:

#[error("still rate limited after {attempts} attempts")]
RateLimited {
    attempts: u32,
    retry_after: Option<Duration>,
    #[source] body: ApiError,
},

retry_after appears in no format string, and body is a #[source], so it is not in Display
either. apply.rs logs error = %error, which therefore renders as "still rate limited after 1
attempts"
and nothing else. The read path is quieter still:

if error.is_rate_limited() {
    tracing::warn!(zone = %name, "throttled while re-reading zones; publishing what was read");
    break;
}

— no error, no duration. Meanwhile deSEC's own body, sitting unread in RateLimited::body, says
{"detail": "Request was throttled. Expected available in 45000 seconds."}.

Why it matters

This is the Go finding in a different mechanism. There:

Without a logger retryablehttp swallows the 429 and its retry wait, which is how the ~12.5h
sleep was invisible in the logs.

Nothing hangs here — that is what #1 is for — so the consequence is smaller. But the number that
tells an operator whether they are thirty seconds or twelve hours from working is equally absent,
and it is the first thing anyone will want. The Retry-After does reach external-dns as a response
header, where nobody reads it, and reaches /metrics as nothing at all.

Where we stand

Two tracing fields in this crate, and arguably one line upstream to put retry_after into
RateLimited's Display — which would fix it for every consumer of the crate rather than this
one.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA feature or enhancementfindingRecorded in docs/FINDINGS.mdupstreamNeeds or would be better served by a change in desec-rs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions