CSI driver writes (a lot of) files to the default collection and creates huge system logs

[llc1123]

I am using SeaweedFS as the storage backend of a distributed vector database (milvus) on K8S. An S3 bucket and 3 PVC provisioned by SeaweedFS are used.

While I do a simple benchmark, I see file count grows both in the default collection (""). If I do volume.fsck -reallyDeleteFromVolume, the database corrupts. Also, I see huge system logs generated under /topics/.system/log (near 300MB per minute).

Then I tried using longhorn for StorageClass and only use SeaweedFS for S3, the default collection stops growing, and the system log sizes are much smaller (100+kb). But I still see 1 file added to the default collection, which is invisible in the filer.)

SeaweedFS Version(masters, filers, volume servers): SeaweedFS 8000GB 3.97 76452ab593995ab52b75e681de5b221de1e3f006
CSI Driver Version: 0.2.3 from master branch

Here is my seaweedfs-csi-driver installation:

helm repo add seaweedfs-csi-driver https://seaweedfs.github.io/seaweedfs-csi-driver/helm

helm -n seaweedfs-csi-driver upgrade --install --create-namespace \
  --set seaweedfsFiler=192.168.1.2:8888 \
  seaweedfs-csi-driver seaweedfs-csi-driver/seaweedfs-csi-driver

[llc1123]

Reproduction Steps:

1. Install Milvus Operator

helm repo add milvus-operator https://zilliztech.github.io/milvus-operator/
helm -n milvus-operator upgrade --install --create-namespace milvus-operator milvus-operator/milvus-operator

2. Create S3 bucket and credentials for Milvus

3. Apply YAML

apiVersion: v1
kind: Secret
metadata:
  name: milvus-minio-secret
type: Opaque
stringData:
  accesskey: SECRET
  secretkey: SECRET
---
apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
  name: milvus
  labels:
    app: milvus
spec:
  mode: cluster
  components:
    mixCoord:
      replicas: 1
      resources:
        limits:
          cpu: 2
          memory: 8Gi
    streamingNode:
      replicas: 1
      resources:
        limits:
          cpu: 4
          memory: 16Gi
    proxy:
      replicas: 1
      resources:
        limits:
          cpu: 2
          memory: 8Gi
      serviceType: LoadBalancer
      port: 19530
    queryNode:
      replicas: 6
      resources:
        limits:
          cpu: 4
          memory: 16Gi
    dataNode:
      replicas: 4
      resources:
        limits:
          cpu: 4
          memory: 16Gi
  config:
    minio:
      bucketName: milvus-prod
      useSSL: false
  dependencies:
    msgStreamType: woodpecker
    etcd:
      inCluster:
        values:
          autoCompactionMode: revision
          autoCompactionRetention: "1000"
          extraEnvVars:
            - name: ETCD_QUOTA_BACKEND_BYTES
              value: "4294967296"
            - name: ETCD_HEARTBEAT_INTERVAL
              value: "500"
            - name: ETCD_ELECTION_TIMEOUT
              value: "25000"
            - name: ETCD_SNAPSHOT_COUNT
              value: "10000"
            - name: ETCD_ENABLE_PPROF
              value: "true"
          persistence:
            accessMode: ReadWriteOnce
            enabled: true
            size: 20Gi
            storageClass: seaweedfs-storage
          replicaCount: 3
          resources:
            limits:
              cpu: 1
              memory: 4Gi
            requests:
              cpu: 1
              memory: 4Gi
    storage:
      external: true
      type: S3
      endpoint: 192.168.1.2:8333
      secretRef: milvus-minio-secret

4. After Milvus deployed, run the benchmark

• Go to "run test" and do the smallest benchmark for Milvus (Search Performance Test (1M Dataset, 768 Dim))
• Check the volumes in weed shell.