ZFS 2.3.3 crash: kernel panic in scrub path (zio_*)

[xaimepardal]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	24.04 LTS
Kernel Version	6.8.0-64-generic
Architecture	x86_64
OpenZFS Version	2.3.3-1

---

Describe the problem you're observing

During a zpool scrub on a raidz1 pool with 8 SATA HDDs, the system experiences a kernel crash.
The crash consistently occurs early in the scrub process.
This leads to the txg_sync thread becoming blocked indefinitely. The system becomes partially unresponsive and requires a hard reboot to recover.

---

Describe how to reproduce the problem

1. Boot into a system with ZFS 2.3.3 and Linux 6.8.0.
2. Have a pool configured with raidz1 using 8 physical drives (WWN-based paths).
3. Start a zpool scrub on the pool:

   zpool scrub storage

4. Monitor with:

   watch zpool status -v

5. After some GB scanned, observe the system lock and crash in kernel.

---

Include any warning/errors/backtraces from the system logs

🔧 Live kernel messages captured:

jul 21 13:07:03 gresint-server kernel: BUG: unable to handle page fault for address: 00007970a8dcc605
jul 21 13:07:03 gresint-server kernel: RIP: 0010:zio_vdev_io_done+0x6e/0x240 [zfs]
jul 21 13:07:03 gresint-server kernel:  ? zio_vdev_io_done+0x6e/0x240 [zfs]
jul 21 13:07:03 gresint-server kernel:  ? zio_vdev_io_done+0x4e/0x240 [zfs]
jul 21 13:07:03 gresint-server kernel:  zio_execute+0x94/0x170 [zfs]
jul 21 13:07:03 gresint-server kernel:  ? __pfx_zio_execute+0x10/0x10 [zfs]
jul 21 13:07:03 gresint-server kernel: RIP: 0010:zio_vdev_io_done+0x6e/0x240 [zfs]

---

Additional Information

zpool status -v

  pool: storage
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub in progress since Sun Jul 20 23:49:14 2025
	7.19T / 125T scanned at 1.13G/s, 2.96T / 125T issued at 474M/s
	884K repaired, 2.37% done, 3 days 02:55:32 to go
config:

	NAME                        STATE     READ WRITE CKSUM
	storage                     ONLINE       0     0     0
	  raidz1-0                  ONLINE       0     0     0
	    wwn-0x5000c500e8a8c504  ONLINE       0     0     9  (repairing)
	    wwn-0x5000c500e8a8a1f6  ONLINE       0     0     4  (repairing)
	    wwn-0x5000c500f6e5f0ab  ONLINE       0     0     7  (repairing)
	    wwn-0x5000c500e8496d51  ONLINE       0     0     2  (repairing)
	    wwn-0x5000c500f6ee1532  ONLINE       0     0     3  (repairing)
	    wwn-0x5000c500e8b3a6f9  ONLINE       0     0     7  (repairing)
	    wwn-0x5000c500e88ed746  ONLINE       0     0     8  (repairing)
	    wwn-0x5000c500e8a8a0aa  ONLINE       0     0     6  (repairing)
	logs	
	  ubuntu-vg/slog-lv         ONLINE       0     0     0
	cache
	  ubuntu--vg-l2arc--lv      ONLINE       0     0     0

errors: No known data errors

zpool get all storage

NAME     PROPERTY                       VALUE                          SOURCE
storage  size                           146T                           -
storage  capacity                       85%                            -
storage  altroot                        -                              default
storage  health                         ONLINE                         -
storage  guid                           13753470766290521828           -
storage  version                        -                              default
storage  bootfs                         -                              default
storage  delegation                     on                             default
storage  autoreplace                    off                            default
storage  cachefile                      -                              default
storage  failmode                       wait                           default
storage  listsnapshots                  off                            default
storage  autoexpand                     on                             local
storage  dedupratio                     1.00x                          -
storage  free                           20.7T                          -
storage  allocated                      125T                           -
storage  readonly                       off                            -
storage  ashift                         12                             local
storage  comment                        -                              default
storage  expandsize                     -                              -
storage  freeing                        0                              -
storage  fragmentation                  36%                            -
storage  leaked                         0                              -
storage  multihost                      off                            default
storage  checkpoint                     -                              -
storage  load_guid                      4324850870775088562            -
storage  autotrim                       off                            default
storage  compatibility                  off                            default
storage  bcloneused                     0                              -
storage  bclonesaved                    0                              -
storage  bcloneratio                    1.00x                          -
storage  dedup_table_size               0                              -
storage  dedup_table_quota              auto                           default
storage  last_scrubbed_txg              0                              -
storage  feature@async_destroy          enabled                        local
storage  feature@empty_bpobj            enabled                        local
storage  feature@lz4_compress           active                         local
storage  feature@multi_vdev_crash_dump  enabled                        local
storage  feature@spacemap_histogram     active                         local
storage  feature@enabled_txg            active                         local
storage  feature@hole_birth             active                         local
storage  feature@extensible_dataset     active                         local
storage  feature@embedded_data          active                         local
storage  feature@bookmarks              enabled                        local
storage  feature@filesystem_limits      enabled                        local
storage  feature@large_blocks           enabled                        local
storage  feature@large_dnode            enabled                        local
storage  feature@sha512                 enabled                        local
storage  feature@skein                  enabled                        local
storage  feature@edonr                  enabled                        local
storage  feature@userobj_accounting     active                         local
storage  feature@encryption             enabled                        local
storage  feature@project_quota          active                         local
storage  feature@device_removal         enabled                        local
storage  feature@obsolete_counts        enabled                        local
storage  feature@zpool_checkpoint       enabled                        local
storage  feature@spacemap_v2            active                         local
storage  feature@allocation_classes     enabled                        local
storage  feature@resilver_defer         enabled                        local
storage  feature@bookmark_v2            enabled                        local
storage  feature@redaction_bookmarks    enabled                        local
storage  feature@redacted_datasets      enabled                        local
storage  feature@bookmark_written       enabled                        local
storage  feature@log_spacemap           active                         local
storage  feature@livelist               enabled                        local
storage  feature@device_rebuild         enabled                        local
storage  feature@zstd_compress          enabled                        local
storage  feature@draid                  enabled                        local
storage  feature@zilsaxattr             enabled                        local
storage  feature@head_errlog            active                         local
storage  feature@blake3                 enabled                        local
storage  feature@block_cloning          enabled                        local
storage  feature@vdev_zaps_v2           active                         local
storage  feature@redaction_list_spill   enabled                        local
storage  feature@raidz_expansion        enabled                        local
storage  feature@fast_dedup             enabled                        local
storage  feature@longname               enabled                        local
storage  feature@large_microzap         enabled                        local

SMART data (excerpt)

Obteniendo todos los discos del pool ZFS...
Se encontraron los siguientes discos:
  - wwn-0x5000c500e8a8c504
  - wwn-0x5000c500e8a8a1f6
  - wwn-0x5000c500f6e5f0ab
  - wwn-0x5000c500e8496d51
  - wwn-0x5000c500f6ee1532
  - wwn-0x5000c500e8b3a6f9
  - wwn-0x5000c500e88ed746
  - wwn-0x5000c500e8a8a0aa

---------------------------------------------------------
SMART info para wwn-0x5000c500e8a8c504 (/dev/sda):
---------------------------------------------------------
Device Model:     ST20000NM007D-3DJ103
Serial Number:    
SMART overall-health self-assessment test result: PASSED
Reallocated_Sector_Ct: 0
Current_Pending_Sector: 0
Offline_Uncorrectable: 0
Temperature: 36°C

---------------------------------------------------------
SMART info para wwn-0x5000c500e8a8a1f6 (/dev/sdb):
---------------------------------------------------------
Device Model:     ST20000NM007D-3DJ103
Serial Number:    
SMART overall-health self-assessment test result: PASSED
Reallocated_Sector_Ct: 0
Current_Pending_Sector: 0
Offline_Uncorrectable: 0
Temperature: 36°C

---------------------------------------------------------
SMART info para wwn-0x5000c500f6e5f0ab (/dev/sdc):
---------------------------------------------------------
Device Model:     ST20000NM007D-3DJ103
Serial Number:    
SMART overall-health self-assessment test result: PASSED
Reallocated_Sector_Ct: 2
Current_Pending_Sector: 0
Offline_Uncorrectable: 0
Temperature: 36°C
⚠️  Reallocated sectors detectados

-----------------------------------
... (truncated)

Troubleshooting steps attempted:

• Tested with zfs_deadman_failmode set to panic, wait, and continue
• Adjusted zfs_vdev_scrub_max_active and min_active
• Monitored journalctl -k live during scrub
• Verified all disks are SMART clean

---

Final Notes

This appears to be a kernel-space memory access bug triggered by ZIO completion under scrub load.
I'm available to test debug builds or apply custom patches if required.

[robn]

Could you please post the full output from dmesg/journalctl -k around that time, include the full register dump and stack trace and any other errors reported on the devices in the pool, including the ones that start with zio.

[xaimepardal]

Sorry for the wait. I've attached the full kernel logs.

dmesg_output.txt
journalctl_kernel_output.txt

[filip-paczynski]

There were a number of prior crashes:

[  241.719312] traps: mate-power-mana[9002] trap int3 ip:7e2f522a70ef sp:7ffc4060b250 error:0 in libglib-2.0.so.0.8000.0[7e2f52263000+a0000]
[  362.595987] traps: deja-dup[10838] trap int3 ip:758f513517ea sp:7ffda4114b30 error:0 in libglib-2.0.so.0.8000.0[758f51305000+a0000]
[  362.796001] traps: deja-dup[10862] trap int3 ip:7ccf9baee7ea sp:7ffc13da6800 error:0 in libglib-2.0.so.0.8000.0[7ccf9baa2000+a0000]
[ 6783.639659] python[5247]: segfault at 0 ip 00007e7d005d56e0 sp 00007e7caa677c50 error 4 in libpython3.12.so.1.0[7e7d005bb000+222000] likely on CPU 4 (core 8, socket 0)
[ 6783.639675] Code: 4c 89 ad 00 ff ff ff 4c 89 ad c8 fe ff ff e9 75 a4 ff ff 4c 8b 2d 18 d6 3c 00 48 8b 3f 4c 39 ee 0f 85 f8 db ff ff 4c 8b 77 08 <41> 8b 06 83 c0 01 0f 85 e6 0b 00 00 48 8b 07 85 c0 0f 89 83 0c 00
[ 8117.822317] python[5252]: segfault at 7abafff97968 ip 00007abafbba6a21 sp 00007abaa5b727e0 error 4 in libpython3.12.so.1.0[7abafbab7000+222000] likely on CPU 0 (core 0, socket 0)
[ 8117.822337] Code: ec 28 64 48 8b 04 25 28 00 00 00 48 89 45 e8 48 8b 47 08 f6 80 ab 00 00 00 20 0f 84 a8 a5 f1 ff 48 8b 05 ca 81 2f 00 48 89 fb <48> 39 46 08 75 31 48 8b 56 18 48 83 fa ff 74 27 48 8d 4d e0 e8 e6
[ 9074.228403] BUG: kernel NULL pointer dereference, address: 000000000000002c

I guess, it might be hardware related, possibly intel 14th gen self-destruct feature.

[snajpa]

I'd suggest trying:

#17418

and if that doesn't help, then:

vpsfreecz@dbff86d

and if that doesn't help, then:

vpsfreecz@0e22b21

[claunia]

Having a similar one only on scrub:

[384945.497737]  zio_destroy.cold+0xf/0x14 [zfs 825ca6bfcb909508b67069f3accfbce9837c819a]
[384945.498043]  zio_done+0xa5d/0x1480 [zfs 825ca6bfcb909508b67069f3accfbce9837c819a]
[384945.498330]  zio_execute+0x92/0x1a0 [zfs 825ca6bfcb909508b67069f3accfbce9837c819a]
[384945.498615]  taskq_thread+0x380/0x7b0 [spl fa9fd459efb467d6f915c80ae26b21dca19ebfb9]
[384945.498635]  ? __pfx_default_wake_function+0x10/0x10
[384945.498641]  ? __pfx_zio_execute+0x10/0x10 [zfs 825ca6bfcb909508b67069f3accfbce9837c819a]
[384945.498939]  ? __pfx_taskq_thread+0x10/0x10 [spl fa9fd459efb467d6f915c80ae26b21dca19ebfb9]
[384945.498960]  kthread+0x11b/0x250
[384945.498964]  ? __pfx_kthread+0x10/0x10
[384945.498968]  ret_from_fork+0x34/0x50
[384945.498971]  ? __pfx_kthread+0x10/0x10
[384945.498974]  ret_from_fork_asm+0x1a/0x30

whole dmesg just repeats that. it all started when I enabled block cloning.

[snajpa]

Oh, block cloning... That and direct IO is why I consider the current dbuf design pretty much insane, it's IMO outdated. Accessing db_buf while praying some ancient guarantees hold, instead of locking it properly... block cloning and direct IO brought the unexpected and IMO anti-patternish db_state mangling to DB_NOFILL and back and I don't think the codebase is fully fixed up... in my opinion, block cloning and direct IO are going to keep being buggy for a long time. I'm not enabling either of them, leaving that for more adventurous users. IMO the db_mtx and db_rwlock situation vs. cv_wait on db_mtx and not being able to call that with a rwlock... calls for straightening out by deleting db_mtx completely and religiously taking db_rwlock any time the db_buf pointer is deref'd. But first we'd need cv_wait equivalent for a rwlock, so...

[snajpa]

Oh I forgot! It will stay being buggy even when the mess around the ambiguity of db_mtx vs db_rwlock is straigthened out, because as direct IO and block cloning are changing the db_buf pointer even when that dbuf is held...

The will_dirty, will_fill, etc. family of functions will keep being racey also - will_dirty, okay? then that function again drops db_mtx and what do you know... guarantees guarantees, yet I can prove it's at the very least racing against dbuf resize :D naaah, that design is not okay, even though @amotin tried to assure me of that :D

[snajpa]

@amotin is there any chance that a PR redoing dbuf locking could be merged? IMO any case of db_buf being accessed without a lock should not exist - it was OK before block cloning and direct IO, but these two can change db_buf even when it's held... it's a huge source of anxiety for me, I can't trust the code when I know there are bugs in it such as this one vpsfreecz@2360de3

I'd prefer using db_rwlock for the whole dbuf, I think the R/W distinction is very useful, IMO this ambiguity of using one or the other is going to keep haunting us...

I would also change all the dirtying functions, such as will_dirty, will_fill, etc. to take the db_rwlock (for writing) and keep it for the caller to unlock when it's done with the change, so that we can guarantee no other function, like dbuf_new_size (and others) jumps into middle of other stuff.

[amotin]

is there any chance that a PR redoing dbuf locking could be merged?

@snajpa Last time I looked on it, my personal verdict was "not under my signature". I may change my opinion some day, or you may look for other reviewers.

[xaimepardal]

I’ve migrated the system to Debian 12 for improved stability, currently running the 6.12.33+deb12-amd64 kernel from backports. Since the change, the kernel errors previously observed no longer occur.

As I needed to restore system functionality, I won’t be able to provide further diagnostics or reproduce the issue. Maintainers may decide whether to close this or continue investigating the behavior on Ubuntu Server with kernel 6.8.

Thanks to all involved.