Object stores

[cboettig]

Test minio via kubernetes on thelio

• to zfs or not zfs the 2.5" hdds?
• upgrade hdds to ssds?

[cboettig]

Design doc: multi-node (RWX) home storage via JuiceFS + S3

The plan for moving off node-pinned home dirs lives in the repo:

docs/content/docs/infrastructure/shared-home-storage.md (status: PROPOSED — not yet deployed)

Summary: add a JuiceFS StorageClass (juicefs-sc) giving ReadWriteMany, not node-pinned home dirs so /home/jovyan follows a user to either node (unblocks GPU notebooks on thelio). Purely additive — openebs-zfs (ZFS LocalPV) stays for node-local scratch. JuiceFS backends:

• data → S3 (RustFS long-term; can run against existing MinIO in the interim — JuiceFS treats it as a generic S3 endpoint)
• metadata → dedicated PostgreSQL (the file index is the filesystem; back it up)

Two independent migrations are tracked here:

1. S3 backend: MinIO → RustFS (deferred — lots of live data on MinIO; MinIO currently on hostPath /mnt/nvme2,3, not the zpool).
2. Homes: ZFS LocalPV → JuiceFS RWX (per-user rsync while server stopped; ~20 live 100Gi PVCs in ns jupyter to preserve).

JuiceFS adoption does not require the RustFS swap to happen first.