DNS-based producer registration should query DNS more frequently

[bnaecker]

When starting an oximeter_producer::Server, one can ask the server to use internal DNS to resolve Nexus's address for registration. That looks like this:

omicron/oximeter/producer/src/lib.rs

Lines 312 to 345 in 38575ab

	async fn registration_task(
	find_nexus: FindNexus,
	log: Logger,
	endpoint: ApiProducerEndpoint,
	) {
	loop {
	debug!(
	log,
	"registering / renewing oximeter producer lease with Nexus"
	);
	let address = match &find_nexus {
	FindNexus::ByAddr(addr) => *addr,
	FindNexus::WithResolver(resolver) => {
	resolve_nexus_with_backoff(&log, resolver).await
	}
	};
	debug!(log, "using nexus address for registration"; "addr" => ?address);
	let lease_duration =
	register_with_backoff(address, &log, &endpoint).await;
	debug!(log, "registered with nexus successfully");
	// Wait for a reasonable fraction of the renewal period, and then hit
	// 'em again.
	let wait =
	lease_duration.checked_div(RENEWAL_RATE).unwrap_or(lease_duration);
	debug!(
	log,
	"pausing until time to renew lease";
	"lease_duration" => ?lease_duration,
	"wait_period" => ?wait,
	);
	tokio::time::sleep(wait).await;
	}
	}

That loop will first look up the address in a loop, and then attempt to register in a loop. Those should probably be interleaved instead, so that on each pass through the loop we look up the address and register. Otherwise, it's possible that the address we resolve becomes stale before we can successfully register with it. This is an unlikely event -- we'd only see that if internal DNS gave us an address we're partitioned from or that immediately goes away as part of service reconfiguration. However, it is certainly still possible, and we should restructure this loop.