【Need help】How to install the multi-machine and multi-disk mode

[cscowx]

Hello, everyone，

I have four test nodes, and each node has two data directories. The details are as follows:
There is a data directory named "/data/rustfs-1-1" and "/data/rustfs-1-2" on the rustfs-1 node.
There is a data directory named "/data/rustfs-2-1" and "/data/rustfs-2-2" on the rustfs-2 node.
There is a data directory named "/data/rustfs-3-1" and "/data/rustfs-3-2" on the rustfs-3 node.
There is a data directory named "/data/rustfs-4-1" and "/data/rustfs-4-2" on the rustfs-4 node.

The hosts records on each node:
192.168.110.240 rustfs-1
192.168.110.241 rustfs-2
192.168.110.242 rustfs-3
192.168.110.243 rustfs-4

Refer to the 《Multi-machine Multi-disk Installation》 configuration on the RustFS official website. Based on the actual number of nodes and directories in my system, how should I write RustFS_VOLUMES here? Also, how does the RustFS_VOLUMES parameter actually work?

# Multi-machine multi-disk mode

sudo tee /etc/default/rustfs <<EOF
RUSTFS_ACCESS_KEY=rustfsadmin
RUSTFS_SECRET_KEY=rustfsadmin
RUSTFS_VOLUMES="http://node{1...4}:9000/data/rustfs{0...3}"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUST_LOG=error
RUSTFS_OBS_LOG_DIRECTORY="/var/logs/rustfs/"
EOF

sudo tee /etc/systemd/system/rustfs.service <<EOF
[Unit]
Description=RustFS Object Storage Server
Documentation=https://rustfs.com/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main
User=root
Group=root

WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs \$RUSTFS_VOLUMES

LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity

Restart=always
RestartSec=10s

OOMScoreAdjust=-1000
SendSIGKILL=no

TimeoutStartSec=30s
TimeoutStopSec=30s

NoNewPrivileges=true
 
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true

# service log configuration
StandardOutput=append:/var/logs/rustfs/rustfs.log
StandardError=append:/var/logs/rustfs/rustfs-err.log

[Install]
WantedBy=multi-user.target
EOF

Refer to the 《Multi-machine Multi-disk Installation》 configuration on the RustFS official website. Based on the actual number of nodes and directories in my system, how should I write RustFS_VOLUMES here? Also, how does the RustFS_VOLUMES parameter actually work?

Is it configured like this?
RUSTFS_VOLUMES="/data/rustfs-1-1,/data/rustfs-1-2"

Do you still want to configure it this way?
RUSTFS_VOLUMES="http://rustfs-{1...4}:9000/data/rustfs-{1...4}-1,http://rustfs-{1...4}:9000/data/rustfs-{1...4}-2"

I don't know how to configure it here.
Does each host node need to be configured in this way as well?

Now the log reports the following error.
tail -f rustfs-err.log
Error: Custom { kind: Other, error: VolumeNotFound }
Error: Custom { kind: Other, error: VolumeNotFound }
Error: Custom { kind: Other, error: VolumeNotFound }
Error: Custom { kind: Other, error: VolumeNotFound }
Error: Custom { kind: Other, error: VolumeNotFound }
Error: Custom { kind: Other, error: VolumeNotFound }

[cscowx]

@loverustfs
Hi, could you please explain it to me? Thank you!

[loverustfs]

In RustFS, drives are organized into Stripes to ensure data safety and high performance.
Stripe Size Rule: A single stripe consists of 4 to 16 drives.
Multiple Stripes: If the total number of drives in your cluster exceeds 16 (e.g., 32 drives), RustFS automatically divides them into multiple stripe sets (e.g., 2 stripes of 16 drives each).

1. 8+8 Configuration (Max Redundancy): In a full 16-drive stripe, you can set a maximum redundancy of 50%. This means:
2. 8 drives are used for Data.
3. 8 drives are used for Parity.
   Fault Tolerance: This configuration allows up to 8 drives to fail simultaneously without data loss.

[loverustfs]

Assuming you are using 16 drives, with a single drive size of $S$ (e.g., 10TB), and an 8 Data + 8 Parity configuration.The formula is:

• Total Drives: 16
• Drive Size: 10 TB
• Ratio: $8 / (8+8) = 0.5$ (50%)

Usable Capacity= 16 x 10TB x 0.5 = 80TB

Note: You need to calculate the difference between 1000KB and 1024KiB hard drive capacity.

[loverustfs]

Multi-Node Deployment Syntax Analysis
Command :

RUSTFS_VOLUMES="http://node{1...4}:9000/data/rustfs{0...3}"

1. node{1...4} represents 4 server nodes: node1, node2, node3, node4.

[cscowx]

@loverustfs
Thank you for your reply.

My environment is like this.

I have four test nodes, and each node has two data directories. The details are as follows:
There is a data directory named "/data/rustfs-1-1" and "/data/rustfs-1-2" on the rustfs-1 node.
There is a data directory named "/data/rustfs-2-1" and "/data/rustfs-2-2" on the rustfs-2 node.
There is a data directory named "/data/rustfs-3-1" and "/data/rustfs-3-2" on the rustfs-3 node.
There is a data directory named "/data/rustfs-4-1" and "/data/rustfs-4-2" on the rustfs-4 node.

The hosts records on each node:
192.168.110.240 rustfs-1
192.168.110.241 rustfs-2
192.168.110.242 rustfs-3
192.168.110.243 rustfs-4

How should the RUSTFS_VOLUMES parameter be configured?

[loverustfs]

All server disk names and mount points must be identical.

rustfs-1
/data/rustrfs-1

rustfs-2
/data/rustrfs-1

rustfs-3
/data/rustrfs-1

rustfs-4
/data/rustrfs-1

[cscowx]

@loverustfs

All server disk names and mount points must be identical.

rustfs-1
/data/rustrfs-1

rustfs-2
/data/rustrfs-1

rustfs-3
/data/rustrfs-1

rustfs-4
/data/rustrfs-1

Based on the above configuration, in the future I will add another mount point named '/data/rustrfs-2' on each node. How can I increase the storage capacity? Do you support this? Is this RUSTFS_VOLUMES parameter requiring reconfiguration?

rustfs-1
/data/rustrfs-2

rustfs-2
/data/rustrfs-2

rustfs-3
/data/rustrfs-2

rustfs-4
/data/rustrfs-2

[cscowx]

@loverustfs
Should the parameters be configured like this?

#Initial configuration
RUSTFS_VOLUMES = “/data/rustrfs-1”

#The revised configuration
RUSTFS_VOLUMES = “/data/rustrfs-1, /data/rustrfs-2”

[loverustfs]

@loverustfs

All server disk names and mount points must be identical.

rustfs-1 /data/rustrfs-1

rustfs-2 /data/rustrfs-1

rustfs-3 /data/rustrfs-1

rustfs-4 /data/rustrfs-1

Based on the above configuration, in the future I will add another mount point named '/data/rustrfs-2' on each node. How can I increase the storage capacity? Do you support this? Is this RUSTFS_VOLUMES parameter requiring reconfiguration?

rustfs-1 /data/rustrfs-2

rustfs-2 /data/rustrfs-2

rustfs-3 /data/rustrfs-2

rustfs-4 /data/rustrfs-2

Prioritize scaling up nodes. RustFS, like MinIO, does not support vertical scaling.

[loverustfs]

RUSTFS_VOLUMES = “/data/rustrfs-1, /data/rustrfs-2” This is incorrect configuration syntax.
The correct syntax is: /data/rustrfs{1...2}. However, the minimum stripe size for EC is 4.