Skip to content

DNS-based producer registration should query DNS more frequently #6158

Open
@bnaecker

Description

@bnaecker

When starting an oximeter_producer::Server, one can ask the server to use internal DNS to resolve Nexus's address for registration. That looks like this:

async fn registration_task(
find_nexus: FindNexus,
log: Logger,
endpoint: ApiProducerEndpoint,
) {
loop {
debug!(
log,
"registering / renewing oximeter producer lease with Nexus"
);
let address = match &find_nexus {
FindNexus::ByAddr(addr) => *addr,
FindNexus::WithResolver(resolver) => {
resolve_nexus_with_backoff(&log, resolver).await
}
};
debug!(log, "using nexus address for registration"; "addr" => ?address);
let lease_duration =
register_with_backoff(address, &log, &endpoint).await;
debug!(log, "registered with nexus successfully");
// Wait for a reasonable fraction of the renewal period, and then hit
// 'em again.
let wait =
lease_duration.checked_div(RENEWAL_RATE).unwrap_or(lease_duration);
debug!(
log,
"pausing until time to renew lease";
"lease_duration" => ?lease_duration,
"wait_period" => ?wait,
);
tokio::time::sleep(wait).await;
}
}

That loop will first look up the address in a loop, and then attempt to register in a loop. Those should probably be interleaved instead, so that on each pass through the loop we look up the address and register. Otherwise, it's possible that the address we resolve becomes stale before we can successfully register with it. This is an unlikely event -- we'd only see that if internal DNS gave us an address we're partitioned from or that immediately goes away as part of service reconfiguration. However, it is certainly still possible, and we should restructure this loop.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions