Same CephFS mounted in 2 clusters at the same time

[ecooke-macu]

A little background and how we're hoping to solve the problem.
Background:

• We have 2 separate Kubernetes clusters. One in each of our 2 datacenters. We can't do a stretched cluster due to requiring an odd number of control plane nodes during a disaster event, which we practice yearly.
• We have an external Ceph cluster replicated between the 2 datacenters.
• We run active/active workloads in both clusters.
• The same workload in both clusters need access to the same shared persistent storage. They have the same namespace and names of all resources.

2 ideas that we're thinking of

• Create a CephFS volume exposed via NFS and use the NFS provisioner with a specified path format, we've tested this configuration with other NFS servers and Kubernetes, and it works. But it has its limitations, particularly around snapshots.
• Create a CephFS volume and mount that using the Ceph CSI in both clusters. This would be our preferred option.
• Any other ideas?

My questions

• Can we do our preferred option, mounting the same CephFS volume in both clusters. I read through the DR documentation with the mapping options, and it looks like it should be possible. Though the docs are geared towards an active/passive Kubernetes cluster.
• Does it need to be the same UUID on the volume in both clusters or does that even matter?
• Is there any guide on how to do this? I couldn't find anything. If not and this works, I'm perfectly willing to submit a PR with what I did to make it work.

Thanks in advance,
-Ed-

[Madhu-1]

If you are looking for features like snapshot,clone etc, it need to be dynamically provisioned PVC.

If both csi are configured to work against a single ceph cluster and (namespace for omap)

You can create PVC on one cluster (dynamic) and take a copy of PVC and PV from the cluster and recreate it on other cluster, With this you should be able to have active/active application where the same storage is used for both.

Does it need to be the same UUID on the volume in both clusters or does that even matter?

Not sure which UUID you are referring to

Is there any guide on how to do this? I couldn't find anything. If not and this works, I'm perfectly willing to submit a PR with what I did to make it work.

https://rook.io/docs/rook/v1.16/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage/?h=share+cephfs+namespace#consume-the-shared-filesystem-across-namespaces, its written for sharing across namespace same can be used across cluster as well, you need to adjust the names of secrets. csi driver etc.

[ecooke-macu]

The UUID was the UUID on the volume object.

I was wondering about the static PV option.

We can for sure make the one cluster the "main" cluster that would do all of the snapshotting and stuff like that so this way of going sounds like a great solution for our needs.

This information just made my day brighter.

Thank you very much.