From docs/FINDINGS.md. Found by running the Go provider this replaces, and correlated
against this codebase rather than acted on. Sources: external-dns-desec-provider#26,
a year of production fixes, its review, and sshine/external-dns-desec-provider#2,
another operator's account of the same failure from a four-cluster deployment.
This is not a decision.
The mechanism
desec::Error::RateLimited:
#[error("still rate limited after {attempts} attempts")]
RateLimited {
attempts: u32,
retry_after: Option<Duration>,
#[source] body: ApiError,
},
retry_after appears in no format string, and body is a #[source], so it is not in Display
either. apply.rs logs error = %error, which therefore renders as "still rate limited after 1
attempts" and nothing else. The read path is quieter still:
if error.is_rate_limited() {
tracing::warn!(zone = %name, "throttled while re-reading zones; publishing what was read");
break;
}
— no error, no duration. Meanwhile deSEC's own body, sitting unread in RateLimited::body, says
{"detail": "Request was throttled. Expected available in 45000 seconds."}.
Why it matters
This is the Go finding in a different mechanism. There:
Without a logger retryablehttp swallows the 429 and its retry wait, which is how the ~12.5h
sleep was invisible in the logs.
Nothing hangs here — that is what #1 is for — so the consequence is smaller. But the number that
tells an operator whether they are thirty seconds or twelve hours from working is equally absent,
and it is the first thing anyone will want. The Retry-After does reach external-dns as a response
header, where nobody reads it, and reaches /metrics as nothing at all.
Where we stand
Two tracing fields in this crate, and arguably one line upstream to put retry_after into
RateLimited's Display — which would fix it for every consumer of the crate rather than this
one.
The mechanism
desec::Error::RateLimited:retry_afterappears in no format string, andbodyis a#[source], so it is not inDisplayeither.
apply.rslogserror = %error, which therefore renders as "still rate limited after 1attempts" and nothing else. The read path is quieter still:
— no error, no duration. Meanwhile deSEC's own body, sitting unread in
RateLimited::body, says{"detail": "Request was throttled. Expected available in 45000 seconds."}.Why it matters
This is the Go finding in a different mechanism. There:
Nothing hangs here — that is what #1 is for — so the consequence is smaller. But the number that
tells an operator whether they are thirty seconds or twelve hours from working is equally absent,
and it is the first thing anyone will want. The
Retry-Afterdoes reach external-dns as a responseheader, where nobody reads it, and reaches
/metricsas nothing at all.Where we stand
Two
tracingfields in this crate, and arguably one line upstream to putretry_afterintoRateLimited'sDisplay— which would fix it for every consumer of the crate rather than thisone.