Benchmarking StorageClasses

[rajsinghtech]

NFS vs CephFS

Local vs CephBlock

Ceph Block NVME 2xreplication vs Ceph Block NVME 3xreplication

Unas SSD smb vs Unas SSD nfs

[LukeHouge]

@rajsinghtech looking at this again and makes sense why Ceph did not work well for Frigate.

NVR is 90% write and all sequential. IOPS and Bandwidth are both better in NFS compared to CephFS and CephBlock, and NFS IOPS even higher than CephNVME, but bandwidth is not as high but not sure we need that high-- all cameras combined is ~30MB/s of data.

For random (like DB) 100% should use Ceph and probably CephNVME.

So I am thinking end state is something like this:

• Ceph NVME (~2TB usable)
  • All DB's (Immich, Jellyfin, etc.)
  • All configs (these could be in either place, NAS might be nice for easier snapshotting and reliability, and can backup nightly back to Ceph)
  • Immich uploads? Enough storage for this long term?
  • Backups/snapshots:
    • Store backups of anything important from NAS (nightly backups from NAS → Ceph for important data in addition to snapshots), especially configs
    • Ceph snapshots nightly
    • Backup to the NAS (use the sync job things?)
• Unifi NAS (~15TB usable)
  • Frigate
  • Media
  • Pics archive (~2TB)
  • Apple Photos library (~2TB)
  • Snapshots nightly
  • Backups/snapshots:
    • Snapshots nightly using Unifi built in snapshots
    • Backup to Ceph using Unifi built in backups
    • Store backups of DB's and configs from Ceph (these will also be snapshotted in Unifi)

What do you think?

[LukeHouge]

@rajsinghtech It's cracked on read, not sure what is going on with writes 😆

[rajsinghtech]

Nice man! I like your plan alot, excited to get the drives

[LukeHouge]

SMB vs. NFS connecting to UNAS Pro

• For IOPS and Bandwidth:
  • SMB sequential reads were slightly slower than NFS sequential reads
  • SMB random reads were significantly slower (about half as fast) than NFS random reads
• For Latency:
  • SMB read and NFS read latency were similar
  • NFS write latency was dramatically higher (10-12ms)