bch2_btree_write_buffer_flush_work, unable to handle page fault for address: ffffc9ff64e01000 #1009
Description
This problem occurred during copy large amount of data using rsync from one bcachefs to other bcachefs (two separated pools of disks).
[132984.991980] BUG: unable to handle page fault for address: ffffc9ff64e01000
[132984.992047] #PF: supervisor read access in kernel mode
[132984.992078] #PF: error_code(0x0000) - not-present page
[132984.992109] PGD 100000067 P4D 100000067 PUD 101abe067 PMD 266633067 PTE 0
[132984.992163] Oops: Oops: 0000 [#1] SMP NOPTI
[132984.992222] CPU: 2 UID: 0 PID: 204849 Comm: kworker/u32:6 Tainted: G O 6.18.0 #1-NixOS PREEMPT(voluntary)
[132984.992284] Tainted: [O]=OOT_MODULE
[132984.992307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 4.2025.02-4 01/01/1980
[132984.992354] Workqueue: events_unbound bch2_btree_write_buffer_flush_work [bcachefs]
[132984.993323] RIP: 0010:bch2_btree_write_buffer_flush_locked+0x151/0x1380 [bcachefs]
[132984.993570] Code: 76 48 83 c6 01 48 89 b3 48 96 00 00 48 8d 0c ca 48 89 c2 48 29 fa 48 c1 fa 03 66 89 11 48 c1 ea 10 88 51 02 0f b6 10 88 51 17 <48> 8b 78 1c 48 8d 51 03 48 89 79 03 48 8b 48 24 48 89 4a 08 8b 48
[132984.994619] RSP: 0018:ffffc9ff416d7c50 EFLAGS: 00010217
[132984.995507] RAX: ffffc9ff64e00fe0 RBX: ffff88c6af180000 RCX: ffffc9ff66b5d490
[132984.996209] RDX: 000000000000000d RSI: 0000000000023d87 RDI: ffffc9ff64601000
[132984.996602] RBP: ffffc9ff416d7de0 R08: 0000000000000001 R09: 0000000000000001
[132984.996956] R10: 0000000000000000 R11: 00b8222a58447da9 R12: ffff88c594360000
[132984.997267] R13: 00000000000003fe R14: ffff88c6af189698 R15: ffff88c6af189648
[132984.997589] FS: 0000000000000000(0000) GS:ffff88c81e22f000(0000) knlGS:0000000000000000
[132984.997914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[132984.998311] CR2: ffffc9ff64e01000 CR3: 000000017246e000 CR4: 00000000000006f0
[132984.998896] Call Trace:
[132984.999505] <TASK>
[132984.999972] ? bch2_btree_write_buffer_flush_work+0x5d/0xe0 [bcachefs]
[132985.000691] bch2_btree_write_buffer_flush_work+0x5d/0xe0 [bcachefs]
[132985.001301] process_one_work+0x18d/0x340
[132985.001823] worker_thread+0x225/0x360
[132985.002166] ? __pfx_worker_thread+0x10/0x10
[132985.002500] kthread+0xfb/0x250
[132985.002817] ? __pfx_kthread+0x10/0x10
[132985.003115] ? _raw_spin_unlock+0xe/0x30
[132985.003426] ? finish_task_switch.isra.0+0x99/0x2e0
[132985.003747] ? __pfx_kthread+0x10/0x10
[132985.004063] ? __pfx_kthread+0x10/0x10
[132985.004510] ret_from_fork+0x1ce/0x200
[132985.005156] ? __pfx_kthread+0x10/0x10
[132985.005773] ret_from_fork_asm+0x1a/0x30
[132985.006412] </TASK>
[132985.007090] Modules linked in: xfs sha1 rpcsec_gss_krb5 dm_crypt encrypted_keys trusted asn1_encoder tee af_packet cfg80211 rfkill 8021q nf_log_syslog nft_log nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nf_tables nls_iso8859_1 nls_cp437 vfat fat edac_core psmouse aesni_intel ses enclosure floppy rtc_cmos joydev tiny_power_button evdev mousedev i2c_piix4 ata_generic intel_agp button intel_gtt pata_acpi vmgenid bochs i2c_smbus input_leds led_class mac_hid serio_raw sch_fq_codel nfsd nfs_acl lockd auth_rpcgss grace nfs_localio fuse loop sunrpc configfs efi_pstore nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock vmw_vmci dmi_sysfs qemu_fw_cfg bcachefs(O) lz4_compress lz4hc_compress libchacha libpoly1305 crc32c_cryptoapi hid_generic usbhid hid sd_mod atkbd libps2 virtio_rng vivaldi_fmap mpt3sas virtio_gpu nvme ata_piix virtio_dma_buf raid_class dm_mod libata scsi_transport_sas nvme_core tpm_crb virtio_scsi dax i8042 nvme_keyring
[132985.007260] virtio_net serio scsi_mod uhci_hcd net_failover nvme_auth tpm_tis virtio_balloon virtio_console failover ehci_hcd tpm_tis_core scsi_common hkdf btrfs blake2b_generic xor raid6_pq efivarfs tpm rng_core virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring autofs4
[132985.013890] CR2: ffffc9ff64e01000
[132985.014327] ---[ end trace 0000000000000000 ]---
[132986.996343] RIP: 0010:bch2_btree_write_buffer_flush_locked+0x151/0x1380 [bcachefs]
[132986.999810] Code: 76 48 83 c6 01 48 89 b3 48 96 00 00 48 8d 0c ca 48 89 c2 48 29 fa 48 c1 fa 03 66 89 11 48 c1 ea 10 88 51 02 0f b6 10 88 51 17 <48> 8b 78 1c 48 8d 51 03 48 89 79 03 48 8b 48 24 48 89 4a 08 8b 48
[132987.000762] RSP: 0018:ffffc9ff416d7c50 EFLAGS: 00010217
[132987.001128] RAX: ffffc9ff64e00fe0 RBX: ffff88c6af180000 RCX: ffffc9ff66b5d490
[132987.001476] RDX: 000000000000000d RSI: 0000000000023d87 RDI: ffffc9ff64601000
[132987.001840] RBP: ffffc9ff416d7de0 R08: 0000000000000001 R09: 0000000000000001
[132987.002246] R10: 0000000000000000 R11: 00b8222a58447da9 R12: ffff88c594360000
[132987.002754] R13: 00000000000003fe R14: ffff88c6af189698 R15: ffff88c6af189648
[132987.003255] FS: 0000000000000000(0000) GS:ffff88c81e22f000(0000) knlGS:0000000000000000
[132987.003610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[132987.003934] CR2: ffffc9ff64e01000 CR3: 00000002dc024000 CR4: 00000000000006f0
[132987.004275] note: kworker/u32:6[204849] exited with irqs disabled
System didn't crash but copying process freeze and systemd can't kill rclone.
Dec 14 17:43:19 nixos-nas sync-and-snapshot[204345]: @backup/restic-backup-server/my/backup/data/b6/b669e6d1be3038947d41ca0693e40b4736221315f9ffa3ec8820981bedb6aafd
Dec 14 17:43:24 nixos-nas sync-and-snapshot[204345]: [307B blob data]
Dec 14 17:43:24 nixos-nas sync-and-snapshot[204345]: @backup/restic-backup-server/my/backup/data/b6/b669edda846ee9637d6c03555f1c6e4ce0a46665f84c39dae7c699ed52518114
Dec 14 20:39:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Main process exited, code=killed, status=15/TERM
Dec 14 20:39:37 nixos-nas sync-and-snapshot[204345]: [135B blob data]
Dec 14 20:39:38 nixos-nas sync-and-snapshot[204346]: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(716) [generator=3.4.1]
Dec 14 20:41:07 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: State 'final-sigterm' timed out. Killing.
Dec 14 20:41:07 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Killing process 204347 (rsync) with signal SIGKILL.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Processes still around after final SIGKILL. Entering failed mode.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Failed with result 'signal'.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Unit process 204347 (rsync) remains running after unit stopped.
Dec 14 20:42:37 nixos-nas systemd[1]: Stopped Sync data and manage bcachefs snapshots.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Consumed 1h 51min 47.564s CPU time, 8.8G memory peak, 4.3M memory swap peak, 1.1T read from disk, 4.7M written to disk.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: Found left-over process 204347 (rsync) in control group while starting unit. Ignoring.
Dec 14 20:42:37 nixos-nas systemd[1]: bcachefs-sync-to-secondary.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 14 20:42:37 nixos-nas systemd[1]: Starting Sync data and manage bcachefs snapshots...
Dec 14 20:42:37 nixos-nas sync-and-snapshot[208295]: sending incremental file list
uname -a
Linux nixos-nas 6.18.0 #1-NixOS SMP PREEMPT_DYNAMIC Sun Nov 30 22:42:10 UTC 2025 x86_64 GNU/Linux
Target FS is fresh, created a few hours before.
dmesg:
bcachefs show-super /dev/sdg1
External UUID: 045a5c97-d425-4c14-b2d5-73364c216d1f
Internal UUID: 4147d6c3-a465-452b-8947-db87309113e8
Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index: 0
Label: (none)
Version: reconcile (1.33)
Incompatible features allowed: reconcile (1.33)
Incompatible features in use: reconcile (1.33)
Version upgrade complete: reconcile (1.33)
Oldest version on disk: reconcile (1.33)
Created: Sun Dec 14 05:46:25 2025
Sequence number: 172
Time of last write: Sun Dec 14 17:41:56 2025
Superblock size: 7.32k/1.00M
Clean: 0
Devices: 4
Sections: crypt,replicas_v0,clean,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes,extent_type_u64s
Features: zstd,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done,no_stale_ptrs
Options:
block_size: 4.00k
btree_node_size: 256k
errors: continue [fix_safe] panic ro
write_error_timeout: 30
metadata_replicas: 3
data_replicas: 2
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0k
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
checksum_err_retry_nr: 3
compression: none
background_compression: zstd
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
casefold: 0
inodes_32bit: 0
shard_inode_numbers_bits: 3
gc_reserve_percent: 8
gc_reserve_bytes: 0
root_reserve_percent: 0
wide_macs: 0
promote_whole_extents: 1
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
degraded: [ask] yes very no
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
writeback_timeout: 0
journal_transaction_names: 1
allocator_stuck_timeout: 30
version_upgrade: [compatible] incompatible none
nocow: 0
reconcile_on_ac_only: 0
errors (size 8):
ext (size 96):
Recovery passes required:
Errors to silently fix:
Btrees with missing data:
Device 0: (not found)
Label: (none)
UUID: ea3c72ef-32ed-409a-bfae-28193411fac0
Size: 1.81T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 953855
Last mount: Sun Dec 14 05:53:31 2025
Last superblock write: 172
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Rotational: 1
Btree allocated bitmap blocksize: 32.0M
Btree allocated bitmap: 0000000000000000000001000000000000000000000011000000001000100111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sdg1
Last device model:
Device 1: (not found)
Label: (none)
UUID: 1f99c5df-1d22-41c8-be5f-f16956e3e022
Size: 3.63T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 1907710
Last mount: Sun Dec 14 05:53:31 2025
Last superblock write: 172
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Rotational: 1
Btree allocated bitmap blocksize: 64.0M
Btree allocated bitmap: 0000000000000000000000000000010000000000000000101000100110001011
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sdf1
Last device model:
Device 2: (not found)
Label: (none)
UUID: e18712cb-cd75-4f98-a7e8-ab3df881e9fc
Size: 1.81T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 953855
Last mount: Sun Dec 14 05:53:31 2025
Last superblock write: 172
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Rotational: 1
Btree allocated bitmap blocksize: 32.0M
Btree allocated bitmap: 0000000000000000001000000000000000000000000100000000001000100111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sdh1
Last device model:
Device 3: /dev/sdd1 ST3000NM0023
Label: (none)
UUID: bf62ba6b-36cb-4f2d-bfb3-71f03bd86b9e
Size: 2.72T
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00M
First bucket: 0
Buckets: 1430780
Last mount: Sun Dec 14 05:53:31 2025
Last superblock write: 172
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Rotational: 1
Btree allocated bitmap blocksize: 64.0M
Btree allocated bitmap: 0000000000000000000000000000100000000011001000000000001001000111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: sdd1
Last device model:
bcachefs-tools: 1.33.0