[Question] Is it OK to scale the seaweedfs-controller StatefulSet with 2 or more replicas ?

[nicned]

In the current implementation, the seaweedfs-controller's StatefulSet is a single instance (1 replica). This presents a SPOF. When the node serving the seaweedfs-controller unexpectedly goes down, and stays down, Kubernetes by design does not recreate StatefulSets elsewhere, and the seaweedfs-controller becomes unavailable.

In an effort to remove this SPOF, and to make the seaweedfs-controller resilient, we have experimented with 2 replicas.

# Source: seaweedfs-csi-driver/templates/statefulset.yml
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: seaweedfs-controller
spec:
  selector:
    matchLabels:
      app: seaweedfs-controller
  serviceName: "csi-seaweedfs"
  replicas: 2

The seaweedfs-csi-driver seems to still work fine. And if one of seaweedfs-controller pods goes down, the second one keeps the whole system still working.

QUESTION 1 - is this a valid configuration from the point of how the seaweedfs-csi solution is designed and implemented? ...Could two or more replicas of seaweedfs-controller StatefulSet create issues that we don't see on the surface (yet)?

QUESTION 2 - is there perhaps a better way to create redundant seaweedfs-controller, to avoid the single-point-of-failure?

[chrislusf]

1. This is fine.
2. Not aware of any other options.