Ceph: Allow flattening image after instance has been created

[DarkressX]

Is there an existing issue for this?

• There is no existing issue for this feature

What are you currently unable to do

Incus in combination with ceph allows for the convenient option to use cloning of images rather than copying. This offers the advantage, that images don't have to be unpacked again for new instances, if a smaller or eual sized image already exists in ceph, making instance deployment very fast.
The downside of this is, that ceph is unable to report the proper size used by the cloned image, as some blocks of the image belong to the parent image it was cloned of. This effect worsens if the cloned image has snapshots.

Unfortunately getting the disk utilization via the agent is not an option for us, as it's potentially unreliable because the agent can be deactivated from within the instance.

What do you think would need to be added

A bool configuration key in the device section of the instance like flatten or postFlatten that causes the link between the cloned image and its parent to be severed after the clone in ceph has been created. In ceph this can be achieved via the rbd flatten command
This would offer the time advantage of instance deployment by utilizing cloning, and at the same time allow ceph to report correct disk utilization of the image.

If this is an acceptable feature to be added, I'd be happy to implement it :)

[stgraber]

@DarkressX doesn't ceph.rbd.clone_copy=false do what you want?

[stgraber]

If not, maybe the flattening logic can be used for clone_copy=false, leading to the same result but in a more optimized way?

[DarkressX]

Unfortunately ceph.rbd.clone_copy=false causes the image to have to be unpacked again which slows down the VM deployment.
If cloning the image and then flattening it after when clone_copy=false is set is an option, that would work perfectly fine as well.

[stgraber]

Yeah, if we end up with the same result but with less work on the Incus side, we should definitely do that.

[DarkressX]

Apologies for the misunderstanding. What I meant was, having incus flatten the image after the instance was created when.
This should be a relatively inexpensive operation if done directly after instance creation, as the instance disk, which contains actual data is very small to that time.
When clone_copy=false is set, incus currently copies the image instead of cloning. Can we switch out this copying operation on incus side with a cloning operation similar to the behaviour when clone_copy=true is set, and send a rbd flatten command directly after?

As an alternative, could we have yet another config key on the storage pool, which requires clone_copy=true to be set, that would then trigger the flatten operation?

[stgraber]

Yeah, just switching the logic should be fine. It gets you the same end result with less overhead.

[DarkressX]

Could I be assigned to this :)

[stgraber]

done!

[stgraber]

@DarkressX any progress on this one?

[DarkressX]

Yes, sorry for not getting back earlier. I've implemented this, but realized that the flattening operation is quite CPU heavy on the node which houses the OSD. Additionally it can take quite long if the VM was created from a large image.
I'd like some more time to test this on better hardware than what I currently have.