Flux reconciliation dependencies

[jvcdk]

Hi Community :)

Topic:
I would like to understand better how Flux sees dependencies in order to make our pipeline faster.

Background:
I am working on a project where we are building and maintaining two Kubernetes clusters, and we are using Flux for our gitops. As part of our CI/CD pipeline, we use Kind to spin up the two clusters, run some tests, and then spin them down again. Currently this process takes nearly 30 minutes to complete, and is kind of blocking our pipeline. Hence I am investigating how to make this process faster.

The main thing that takes time is reconciliation of our two clusters. To speed that up, we have (some time ago) implemented a small program that help speed up the reconciliation process. It functions basically like this:

1. Our pipeline builds an OCI artifact and pushes it to the cluster.
2. Then we build a Kustomization dependency tree (based on our yaml source).
3. Starting with the Kustomizations that have no dependencies, we run flux reconcile kustomization <kustomization name>
4. When a reconciliation is (successfully) done, we will select Kustomizations which have all their dependencies marked done, then repeat 3.

The observation was this: If we run a flux reconcile ... on a Kustomization, it would trigger a reconciliation faster than just waiting for the trigger timeout to happen.

Our observed issue:
Our system also prints out status changes from the Kustomizations, so we can track a bit what is going on. This revealed that while a given Kustomization could be marked Done, the dependent Kustomization(s) would still see it as Pending.

This is what the log file shows:

• At 10:35:14 we start reconciling kyverno-system-controllers.
• At 10:37:22 it is done.
• At the same instant, we start reconciling kyverno-system.
• At 10:37:24 We get the message from kyverno-system: Dependency kyverno-system-controllers is not ready.
• At 10:38:12 kyverno-system starts fetching its manifest, indicating that it discovered that kyverno-system-controllers was ready.

Log file snippet:

10:35:14.423  Reconciling flux-system/kyverno-system-controllers: Reconciling...
10:35:16.519  Reconciling flux-system/kyverno-system-controllers: Running health checks for revision shared-services@sha256:<cut> with a timeout of 10m0s
10:37:21.153  Reconciling flux-system/kyverno-system-controllers: Applied revision: shared-services@sha256:<cut>
10:37:22.480  Reconciling flux-system/kyverno-system-controllers: Done
10:37:22.480  Reconciling flux-system/kyverno-system: Reconciling...
10:37:24.543  Reconciling flux-system/kyverno-system: dependency 'flux-system/kyverno-system-controllers' is not ready
10:38:12.009  Reconciling flux-system/kyverno-system: Fetching manifests for revision shared-services@sha256:<cut> with a timeout of 10m0s
10:38:18.219  Reconciling flux-system/kyverno-system: Detecting drift for revision shared-services@sha256:<cut> with a timeout of 10m0s
10:39:05.732  Reconciling flux-system/kyverno-system: Running health checks for revision shared-services@sha256:<cut> with a timeout of 10m0s
10:39:46.524  Reconciling flux-system/kyverno-system: Done
10:39:48.596  Reconciling flux-system/dex-system: dependency 'flux-system/kyverno-system' is not ready

Notes on polling Kustomization messages:

• We poll for changes every 2 seconds; hence the time difference between Reconciling and Dependency Not Ready may be smaller than the listed 2 seconds.
• We use the command kubectl get kustomizations.kustomize.toolkit.fluxcd.io <name> --output jsonpath={.status.conditions[0].message} to retrieve the latest message.

Questions / help needed:

• Why does it take 50 seconds between kyverno-system-controllers being ready, and kyverno-system figuring that out?
• What can we do to reduce this latency?

Thank you in advance 🙂

Ps:

I know that my thread here is very similar to Faster reconciliation of dependency chains, but seeing it did not get a lot of attention, I thought I might bring some attention on the issue again.

Also... I would like to know that is going on behind the scenes... that may help me either fix things, or have patience with it 🙂

[stefanprodan]

You can speed this up by setting the --requeue-dependency=5s

See the docs here on how to fine tune Flux: https://fluxcd.io/flux/installation/configuration/vertical-scaling/#increase-the-number-of-workers-and-limits

[stefanprodan]

We have previously discussed the requeue-dependency option in my team, and there was some concern regarding it; mostly resource and performance concerns.

Did your team run any benchmarks, what’s the base of your concerns?

[jvcdk]

No, sorry, I don't think so. I think they liked the default value out from a reasoning that Flux would choose sane defaults, and as I am fairly new in the world of K8s, I think it is wise for me to listen to them.

Thank you for taking time to reply to me 🙂

[stefanprodan]

I've set these defaults many years ago when Kubernetes API behaved differently. Nowadays, fast requeuing should have minimal impact on CPU/Memory usage since all it does now is reading from the controller-runtime cache.

[matheuscscp]

@jvcdk Your team can trust advice from the person who chose these defaults :)

[jvcdk]

Hi there :)

Sorry for the slow response, but I needed to check out some details before replying (and also got side tracked a bit at work).

Regarding --requeue-dependency=5s (and other tricks):

Turns out that some time ago (without me noticing it) they did set --requeue-dependency=2s and concurrent=20 in our system and have been running with that since. Note the different values.

However; it does not seem to relieve the issue. I played with these (setting it to 5s and 10) without any change. I also tried:

• Disabled in-memory kustomize builds (enabled by default on our system)
• Enabled Helm repositories caching
• --max-retry-delay=5s
• Feature gate DisableFailFastBehavior=true
• Feature gate DisableStatusPollerCache=false

Nothing really seemed to make a difference.

The issue – now in a graphical form :)

So, I am still left with the question: How does it happen that Kyverno-system sees Kyverno-system-conrollers as being not ready, when a flux reconcile ... cli command exited cleanly, stating that it is.

(And also; inspecting in k9s indicates that Kyverno-system-conrollers is in fact done reconciling.)

I have tried to illustrate the situation here:

Or stated in a more short form

• We apply two kustomizations, a and b
• a depends on b
• We reconcile both
• b becomes ready
• a still reports back that it is waiting for b, and stays like this for ~200 seconds.

We see this mostly with kyverno-system, but also sometimes with cilium-system, storage-system, and supply-chain-system.

Do you have any (new) ideas of why this is happening? And what we can do to mitigate it?

BR
Jørn