Feature: Move away from builder TTL

[Despire]

Description

Currently when the config is picked up by builder, it has a certain amount of TIME in which it has to complete the build of the cluster. This might work for small to medium clusters. For larger cluster or even with long enough timeouts this TTL will time out and a new config (possibly with changes) can be enqueued and now two configs for the same infra will be build by builder which can result in undefined infa

Exit criteria

• Move away from builder TTL

[Despire]

Other possible solution would be to have a Lease instead of a TTL.

Builder would periodically renew the lease and if there is no renewal for time X the task would be rescheduled.

So as long as there would be a builder running (that is able to connect to the manager pod) this would be okay. Even in rollouts of a new version, as builder always finishes the current task before exiting. And the newly spawned pod would then wait until the lease runs out and/or pickup the next available task.

But there is a possible edge with this (as is with builder TTL), What if there is some outage between builder/manager where the lease renewal fails and the task gets rescheduled, the old running builder pod would know that it doesn't need to be worked on, but it could happen that the request would be routed to another builder pod (if spawned) and this one would pick up the task and would end up with 2 workflows again.

  We can also have a lease in individual services, and when a link is broken have a gracefull shutdown period after which     we report an error and enter reconciliation