Optimistic-concurrency conflicts are normal in a controller. These are not spread thin - they come in bursts tied to activity, always from the same few write sites, and every one of them throws away a whole reconcile.
What is in the log
74 conflict errors in 2000 lines of manager log from one lab session, splitting cleanly:
| Count |
Controller |
Write |
| 37 |
image |
update image seeder status |
| 27 |
machine |
agent deployer: clearing registration state for Register |
| 10 |
machine |
plain Operation cannot be fulfilled on machines.kezio.kojuro.date |
One in full:
ERROR Reconciler error {"controller": "image", "controllerKind": "Image",
"Image": {"name":"lab-ubuntu","namespace":"kezio-system"},
"reconcileID": "5c8d5d3a-...",
"error": "update image seeder status: Operation cannot be fulfilled on
images.kezio.kojuro.date \"lab-ubuntu\": the object has been modified;
please apply your changes to the latest version and try again"}
Timestamps cluster rather than trickle - five inside 25 seconds, then a gap:
09:34:37 09:34:47 09:34:55 09:34:55 09:35:00 09:35:00 ... 09:40:00
That shape says several writers touching one object while something is happening to it (machines deploying, a seeder scaling), not two reconciles brushing past each other once.
Why it is worth looking at rather than tolerating
- A conflict discards the reconcile that hit it, including any work it did before the write. With more machines it happens more often, and the retries are not free at the API server either.
- It hides real failures.
Reconciler error at ERROR level, dozens of times, trains a reader to skip that line - which is exactly the line a genuine fault will appear on.
- The same shape has already produced a real bug once: writing spec before a not-yet-persisted status meant
Update refreshed the object and silently discarded status.provisioning and status.poweredOn (fixed in 54be293). Whatever is doing this to Image may be doing it for the same reason.
Worth checking
- Does one reconcile pass write the same object more than once (spec then status, or status twice), reusing a copy that the first write already made stale?
update image seeder status and update seeder deployment both appear. If the Image reconcile writes the Deployment and then writes Image status derived from it, the ordering matters.clearing registration state for Register is the machine controller's equivalent hot spot and probably has the same cause.- Where a conflict is genuinely expected and harmless, it does not belong at ERROR - requeue quietly instead, so the level keeps meaning something.
Found while deploying 10-30 machines against a Proxmox VE lab. Nothing failed because of this; it is a smell with a track record, not an outage.
Optimistic-concurrency conflicts are normal in a controller. These are not spread thin - they come in bursts tied to activity, always from the same few write sites, and every one of them throws away a whole reconcile.
What is in the log
74 conflict errors in 2000 lines of manager log from one lab session, splitting cleanly:
imageupdate image seeder statusmachineagent deployer: clearing registration state for RegistermachineOperation cannot be fulfilled on machines.kezio.kojuro.dateOne in full:
Timestamps cluster rather than trickle - five inside 25 seconds, then a gap:
That shape says several writers touching one object while something is happening to it (machines deploying, a seeder scaling), not two reconciles brushing past each other once.
Why it is worth looking at rather than tolerating
Reconciler errorat ERROR level, dozens of times, trains a reader to skip that line - which is exactly the line a genuine fault will appear on.Updaterefreshed the object and silently discardedstatus.provisioningandstatus.poweredOn(fixed in 54be293). Whatever is doing this toImagemay be doing it for the same reason.Worth checking
update image seeder statusandupdate seeder deploymentboth appear. If the Image reconcile writes the Deployment and then writes Image status derived from it, the ordering matters.clearing registration state for Registeris the machine controller's equivalent hot spot and probably has the same cause.Found while deploying 10-30 machines against a Proxmox VE lab. Nothing failed because of this; it is a smell with a track record, not an outage.