Cannot automatically mount or evacuate added device after device UUID change #1034
Description
I added a new nvme drive (/dev/nvme1n1) to an existing bcachefs instance with bcachefs device add /bpool /dev/nvme1n1 --label ssd.ssd2 (iirc). This drive was previously GPT partitioned with a fat32 partition and a ZFS partition. When I added the device, it seems its disk UUID changed, and bcachefs can no longer mount it automatically when I try to mount the FS by UUID=....
The superblock (attached in full later) shows the device as not found, with a UUID that does not exist in /dev/disk/by-uuid, which should be /dev/nvme1n1
Device 3: (not found)
Label: ssd.ssd2
UUID: ae50be1f-5f7f-48f7-9f3b-9ceb6ec201ac
# ls /dev/disk/by-uuid -l
total 0
lrwxrwxrwx 1 root root 9 Jan 5 04:08 4b692493-7142-4b17-8846-52ff010f96af -> ../../sda
lrwxrwxrwx 1 root root 15 Jan 5 04:08 61B5-BEE6 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 Jan 5 04:08 c5fa6721-4dd0-44f0-88ab-7793ed11bac0 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 9 Jan 5 13:22 d18c248b-5c1e-4c8a-9d47-d6e64ed34de2 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jan 5 04:08 d53e038d-860a-43ad-9be6-8c9b41096a9c -> ../../sdb
lrwxrwxrwx 1 root root 15 Jan 5 04:08 e9dd359b-6b92-4bd1-9d1d-ec1f57719e22 -> ../../nvme0n1p2The filesystem was initially created with the following command:
bcachefs format --compression=lz4 \
--encrypted \
--replicas=2 \
--label=ssd.ssd1 /dev/nvme0n1p2 \
--label=hdd.hdd1 /dev/sda \
--label=hdd.hdd2 /dev/sdb \
--foreground_target=ssd \
--promote_target=ssd \
--background_target=hdd
Mounting by device string /dev/sda:/dev/sdb:/dev/nvme0n1p2:/dev/nvme1n1 worked, but I wasn't able to find any way to change the uuid. bcachefs device online fails because the device is in use, and bcachefs device offline fails because it apparently contains unreplicated data even though it is configured as a cache drive.
# bcachefs device online /dev/nvme1n1
BCH_IOCTL_DISK_ONLINE_v2 error: Device or resource busy
error reading superblock: EBUSY
error=EBUSY
# bcachefs device offline /dev/nvme1n1
BCH_IOCTL_DISK_OFFLINE_v2 error: Invalid argument
insufficient devices online (0) for replicas entry user: 1/1 [3]
Cannot offline required disk
error=device_state_not_allowedAt this point I tried to remove the drive and re-add it, which tells me there would be data loss:
# bcachefs device remove /dev/nvme1n1
BCH_IOCTL_DISK_REMOVE_v2 error: Invalid argument
cannot drop device without degrading/losing data
u64s 8 type extent 2149420463:8:4294967284 len 8 ver 18348870 : durability: 0
crc64: c_size 8 size 8 offset 0 nonce 0 csum chacha20_poly1305_80 12a3:c3f0a61d3472b5f5 compress incompressible
ptr: nvme1n1 3:15476:392 gen 0
error=remove_would_lose_dataSo I tried to evacuate it which hangs forever (at least 12 hours), with no changes shown in bcachefs fs usage in that time, except the new -1x column and evacuating status of ssd.ssd2.
# bcachefs device evacuate /dev/nvme1n1
Setting /dev/nvme1n1 evacuating
24346154496# bcachefs fs usage /bpool -h
Filesystem: d18c248b-5c1e-4c8a-9d47-d6e64ed34de2
Size: 31.3T
Used: 1.11T
Online reserved: 2.21M
Data by durability desired and amount degraded:
undegraded -1x
1x: 2.54G 34.4M
2x: 1.06T 45.2G
cached: 294G
reserved: 375M
Device label Device State Size Used Use%
hdd.hdd1 (device 1): sdb rw 14.5T 432G 02%
hdd.hdd2 (device 2): sda rw 14.5T 422G 02%
ssd.ssd1 (device 0): nvme0n1p2 rw 3.18T 577G 17%
ssd.ssd2 (device 3): nvme1n1 evacuating 1.81T 30.6G 01%I tried bcachefs data rereplicate which redirected me to bcachefs reconcile wait which shows nothing active.
# bcachefs data rereplicate
rereplicate no longer required or support >= reconcile; use 'bcachefs reconcile wait'
# bcachefs reconcile wait /bpool
Scan pending: 0
data metadata
replicas: 0 0
checksum: 0 0
erasure_code: 0 0
compression: 0 0
target: 0 0
high_priority: 0 0
I'm not sure where to go from here, but if necessary I can build and run patches.
Superblock
# bcachefs show-super /dev/sda
Enter passphrase:
External UUID: d18c248b-5c1e-4c8a-9d47-d6e64ed34de2
Internal UUID: 7d5d673e-6879-4513-9529-bc6f049fb715
Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index: 2
Label: (none)
Version: reconcile (1.33)
Incompatible features allowed: inode_has_child_snapshots (1.13)
Incompatible features in use: (unknown version) (0.0)
Version upgrade complete: reconcile (1.33)
Oldest version on disk: inode_has_child_snapshots (1.13)
Created: Tue Dec 30 03:24:21 2025
Sequence number: 11131
Time of last write: Mon Jan 5 13:22:00 2026
Superblock size: 7.90k/1.00M
Clean: 0
Devices: 4
Sections: members_v1,crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes,extent_type_u64s
Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done
Options:
block_size: 4.00k
btree_node_size: 256k
errors: continue [fix_safe] panic ro
write_error_timeout: 30
metadata_replicas: 2
data_replicas: 2
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0k
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
checksum_err_retry_nr: 3
compression: lz4
background_compression: none
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: ssd
background_target: hdd
promote_target: ssd
erasure_code: 0
casefold: 0
inodes_32bit: 0
shard_inode_numbers_bits: 5
gc_reserve_percent: 8
gc_reserve_bytes: 0
root_reserve_percent: 0
wide_macs: 0
promote_whole_extents: 1
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
degraded: [ask] yes very no
journal_flush_delay: 1000
journal_flush_disabled: 1
journal_reclaim_delay: 1000
writeback_timeout: 0
journal_transaction_names: 1
allocator_stuck_timeout: 30
version_upgrade: [compatible] incompatible none
nocow: 0
reconcile_on_ac_only: 0
errors (size 56):
reconcile_work_incorrectly_set 177629 Wed Dec 31 04:51:42 2025
accounting_key_junk_at_end 172867 Tue Dec 30 22:08:42 2025
logged_op_but_clean 24 Tue Dec 30 14:17:40 2025
ext (size 96):
Recovery passes required: btree_bitmap_gc
Errors to silently fix:
Btrees with missing data:
Device 0: /dev/nvme0n1p2 TEAM TM8FPW004T
Label: ssd.ssd1
UUID: e9dd359b-6b92-4bd1-9d1d-ec1f57719e22
Size: 3.18T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 1668686
Last mount: Mon Jan 5 04:08:30 2026
Last superblock write: 11131
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Rotational: 0
Btree allocated bitmap blocksize: 32.0M
Btree allocated bitmap: 0000000000000000000000000000010000000000000110000000100011000101
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: nvme0n1p2
Last device model: Samsung SSD 980 PRO 2TB
Device 1: /dev/sdb OOS16000G
Label: hdd.hdd1
UUID: d53e038d-860a-43ad-9be6-8c9b41096a9c
Size: 14.5T
read errors: 0
write errors: 0
checksum errors: 2
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 7629824
Last mount: Mon Jan 5 04:08:30 2026
Last superblock write: 11131
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Rotational: 1
Btree allocated bitmap blocksize: 16.0M
Btree allocated bitmap: 0000000000000000000000011000000000000000100000000110000000010010
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sdb
Last device model: OOS16000G
Device 2: /dev/sda OOS16000G
Label: hdd.hdd2
UUID: 4b692493-7142-4b17-8846-52ff010f96af
Size: 14.5T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 7629824
Last mount: Mon Jan 5 04:08:30 2026
Last superblock write: 11131
State: rw
Data allowed: journal,btree,user
Has data: btree,user
Rotational: 1
Btree allocated bitmap blocksize: 16.0M
Btree allocated bitmap: 0000000000000000001000000000100000000000000000000000000000000000
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sda
Last device model: OOS16000G
Device 3: (not found)
Label: ssd.ssd2
UUID: ae50be1f-5f7f-48f7-9f3b-9ceb6ec201ac
Size: 1.81T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 953864
Last mount: Mon Jan 5 04:08:30 2026
Last superblock write: 11131
State: evacuating
Data allowed: journal,btree,user
Has data: btree,user
Rotational: 0
Btree allocated bitmap blocksize: 1.00M
Btree allocated bitmap: 0000000000001000000000000000000000000000000000010000000000000010
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: nvme1n1
Last device model: Samsung SSD 980 PRO 2TBSystem:
NixOS 26.05 / unstable
Kernel 6.17.13
Bcachefs 1.33.3