CD daemon: process manager: backoff on daemon restarts

[jgehrcke]

For config failures and similar, right now we busy-spin on daemon restarts managed by process manager.

Let's introduce an adaptive wait time here:

https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/7b5e2cd428f3c654812ec9d21347437bb6f099e7/cmd/compute-domain-daemon/process.go#L195

Example, showing unreasonably high rate:

W1014 09:04:42.467224       1 process.go:189] Watchdog: child terminated unexpectedly
[...]
W1014 09:04:42.467236       1 process.go:195] Watchdog: start process again
[...]
W1014 09:04:43.467276       1 process.go:189] Watchdog: child terminated unexpectedly
[...]
W1014 09:04:44.467334       1 process.go:195] Watchdog: start process again

[jgehrcke]

I misread the logs -- the timestamps above show that there's a one second delay between attempts. That's expected. We're not busy-spinning.

We can add a back-off later, of course. But there's nothing to do here for the 25.8.0 release.

[klueska]

Yes, that is what I pointed out in our call yesterday.

[jgehrcke]

what I pointed out in our call yesterday.

Misunderstanding. I thought we were both seeing two restarts within the same millisecond.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale