L2ARC: inexplicable disappearance, without removal, of cache device #12519

grahamperrin · 2021-08-29T04:39:37Z

grahamperrin
Aug 29, 2021

This morning, I was surprised by zpool status not showing the cache for one of my two pools.

As if removed, however I had not removed it – there was no history of removal for the device.

Should I report a bug? I don't know whether it can be reproducible.

root@mowa219-gjp4-8570p-freebsd:~ # zpool history Transcend | tail -n 15
2021-08-27.01:15:13 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-27.00:19:22 zpool scrub Transcend
2021-08-27.06:04:59 zfs load-key Transcend/VirtualBox
2021-08-27.07:29:12 zpool online Transcend gpt/cache-transcend
2021-08-27.12:20:00 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-27.13:29:57 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-27.12:34:07 zfs load-key Transcend/VirtualBox
2021-08-28.13:53:07 zfs load-key Transcend/VirtualBox
2021-08-28.21:36:05 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-28.21:43:33 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-29.05:31:43 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-29.05:54:00 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-29.06:11:02 zpool import -c /etc/zfs/zpool.cache -a -N
2021-08-29.06:14:30 zpool import -c /etc/zfs/zpool.cache -a -N

root@mowa219-gjp4-8570p-freebsd:~ # zpool status Transcend
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:32:24 with 0 errors on Fri Aug 27 03:51:45 2021
config:

        NAME        STATE     READ WRITE CKSUM
        Transcend   ONLINE       0     0     0
          da5p1     ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p-freebsd:~ # zpool online Transcend gpt/cache-transcend
cannot online gpt/cache-transcend: no such device in pool
root@mowa219-gjp4-8570p-freebsd:~ # /usr/home/grahamperrin/dev/lsblk/lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:121 932G GPT                                               - -
  ada0p1         0:123 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:125  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:66   16G freebsd-swap                                      - SWAP
  ada0p3         0:127 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:135 915G zfs                                               - -
  <FREE>         -:-   708K -                                                 - -
da0              0:196 466G GPT                                               - -
  da0p1          0:201 200M efi                                  msdosfs/EFISYS -
  da0p2          0:204 512K freebsd-boot                           gpt/gptboot0 -
  da0p3          0:205  16G freebsd-swap                     gpt/FreeBSD%20swap SWAP
  da0p4          0:206 450G freebsd-zfs                                gpt/zfs0 <ZFS>
  da0p4.eli      0:211 450G zfs                                               - -
  <FREE>         -:-   4.0K -                                                 - -
da5              0:230 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da5p1          0:231 466G freebsd-zfs                           gpt/Transcend <ZFS>
da6              0:238  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da6p1          0:239  14G freebsd-zfs                     gpt/cache-transcend <ZFS>
da7              0:247  29G GPT                                               - -
  da7p1          0:248  29G freebsd-zfs                        gpt/cache-august <ZFS>
root@mowa219-gjp4-8570p-freebsd:~ # date ; uptime
Sun Aug 29 05:22:56 BST 2021
 5:22AM  up 9 mins, 5 users, load averages: 0.23, 0.36, 0.20
root@mowa219-gjp4-8570p-freebsd:~ # uname -KU
1400030 1400032
root@mowa219-gjp4-8570p-freebsd:~ # uname -a
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #105 main-n248685-c9f833abf1d: Fri Aug 13 20:24:43 BST 2021     root@mowa219-gjp4-zbook-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG  amd64
root@mowa219-gjp4-8570p-freebsd:~ # zfs --version
zfs-2.1.99-FreeBSD_g8ae86e2ed
zfs-kmod-2.1.99-FreeBSD_gf3678d70f
root@mowa219-gjp4-8570p-freebsd:~ # kldstat | grep zfs
 5    1 0xffffffff8268d000   5bb9b8 zfs.ko
root@mowa219-gjp4-8570p-freebsd:~ # zpool history Transcend | grep -e add -e remove
2021-04-19.06:15:22 zpool add Transcend cache da2
2021-07-24.18:11:47 zpool remove Transcend da0
2021-07-25.13:03:00 zpool add Transcend cache gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool add Transcend cache gpt/cache-transcend
root@mowa219-gjp4-8570p-freebsd:~ # zpool status
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:32:24 with 0 errors on Fri Aug 27 03:51:45 2021
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          da5p1                ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors

  pool: august
 state: ONLINE
  scan: scrub repaired 0B in 00:53:41 with 0 errors on Fri Aug 27 01:13:10 2021
config:

        NAME                STATE     READ WRITE CKSUM
        august              ONLINE       0     0     0
          ada0p3.eli        ONLINE       0     0     0
        cache
          gpt/cache-august  ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p-freebsd:~ #

grahamperrin · 2021-08-29T04:56:03Z

grahamperrin
Aug 29, 2021
Author

Worth mentioning, although I don't imagine this having an impact, a few hours ago I enabled:

hw.usb.no_boot_wait

% grep hw.usb /boot/loader.conf
hw.usb.usbhid.enable=1
hw.usb.no_boot_wait=1
% sysctl hw.usb. | grep wait
hw.usb.no_shutdown_wait: 0
hw.usb.no_suspend_wait: 0
hw.usb.no_boot_wait: 1
% geom disk list da5 da6 da7
Geom name: da5
Providers:
1. Name: da5
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: StoreJet Transcend
   lunid: 5000000000000001
   ident: X3E1SAKRS
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da6
Providers:
1. Name: da6
   Mediasize: 15518924800 (14G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   lunname: KingstonDataTraveler 3.0
   lunid: KingstonDataTraveler 3.0
   ident: 60A44C3FACC93110B9970045
   rotationrate: 0
   fwsectors: 63
   fwheads: 255

Geom name: da7
Providers:
1. Name: da7
   Mediasize: 30943995904 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   ident: E0D55EA1C84FF390A9500FDA
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

% gpart recover da5
da5 recovering is not needed
% gpart recover da6
da6 recovering is not needed
%

USB controller: - allow disabling "root_mount_hold()" by setting "hw.usb.no_boot_wait" sysctl (2009-07-30)

1 reply

grahamperrin Aug 30, 2021
Author

hw.usb.no_boot_wait

This is a probable cause of UNAVAIL in the example below.

I can not imagine hw.usb.no_boot_wait causing disappearance of a device from a pool.

% su -
Password:
root@mowa219-gjp4-8570p-freebsd:~ # zpool iostat -v
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               344G   120G      0      0  21.4K  12.0K
  da6p1                 344G   120G      0      0  21.4K  12.0K
cache                      -      -      -      -      -      -
  gpt/cache-transcend  9.45G  5.00G      0      0  7.26K  3.11K
---------------------  -----  -----  -----  -----  -----  -----
august                  299G   613G     82     22  1.63M   893K
  ada0p3.eli            299G   613G     82     22  1.63M   893K
cache                      -      -      -      -      -      -
  gpt/cache-august         -      -      0      0      0      0
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # gpart show -l | grep august
        34  60437425    1  cache-august  (29G)
root@mowa219-gjp4-8570p-freebsd:~ # /usr/home/grahamperrin/dev/lsblk/lsblk 
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:121 932G GPT                                               - -
  ada0p1         0:123 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:125  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:71   16G freebsd-swap                                      - SWAP
  ada0p3         0:127 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:135 915G zfs                                               - -
  <FREE>         -:-   708K -                                                 - -
da0              0:183 1.9G freebsd-swap                                      - SWAP
da1              0:194 1.9G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:197 1.9G !8c8f8eff-ac95-4770-814a-21994f2dbc8f                                - -
da6              0:234 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da6p1          0:235 466G freebsd-zfs                           gpt/Transcend <ZFS>
da7              0:242  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da7p1          0:243  14G freebsd-zfs                     gpt/cache-transcend <ZFS>
da8              0:252  29G GPT                                               - -
  da8p1          0:253  29G freebsd-zfs                        gpt/cache-august <ZFS>
root@mowa219-gjp4-8570p-freebsd:~ # zpool status
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:32:24 with 0 errors on Fri Aug 27 03:51:45 2021
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          da6p1                ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors

  pool: august
 state: ONLINE
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
  scan: scrub repaired 0B in 00:53:41 with 0 errors on Fri Aug 27 01:13:10 2021
config:

        NAME                STATE     READ WRITE CKSUM
        august              ONLINE       0     0     0
          ada0p3.eli        ONLINE       0     0     0
        cache
          gpt/cache-august  UNAVAIL      0     0     0  cannot open

errors: No known data errors
root@mowa219-gjp4-8570p-freebsd:~ # zpool online august gpt/cache-august
root@mowa219-gjp4-8570p-freebsd:~ # zpool status
  pool: Transcend
 state: ONLINE
  scan: scrub repaired 0B in 03:32:24 with 0 errors on Fri Aug 27 03:51:45 2021
config:

        NAME                   STATE     READ WRITE CKSUM
        Transcend              ONLINE       0     0     0
          da6p1                ONLINE       0     0     0
        cache
          gpt/cache-transcend  ONLINE       0     0     0

errors: No known data errors

  pool: august
 state: ONLINE
  scan: scrub repaired 0B in 00:53:41 with 0 errors on Fri Aug 27 01:13:10 2021
config:

        NAME                STATE     READ WRITE CKSUM
        august              ONLINE       0     0     0
          ada0p3.eli        ONLINE       0     0     0
        cache
          gpt/cache-august  ONLINE       0     0     0

errors: No known data errors
root@mowa219-gjp4-8570p-freebsd:~ # zpool iostat -v
                         capacity     operations     bandwidth 
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               344G   120G      0      0  17.5K  9.84K
  da6p1                 344G   120G      0      0  17.5K  9.84K
cache                      -      -      -      -      -      -
  gpt/cache-transcend  9.45G  5.00G      0      0  5.92K  2.54K
---------------------  -----  -----  -----  -----  -----  -----
august                  299G   613G     67     19  1.34M   745K
  ada0p3.eli            299G   613G     67     19  1.34M   745K
cache                      -      -      -      -      -      -
  gpt/cache-august     8.20G  20.6G      0      0  12.9K  10.6K
---------------------  -----  -----  -----  -----  -----  -----
root@mowa219-gjp4-8570p-freebsd:~ # zpool history august | grep -e online -e offline
2021-08-29.05:19:04 zpool online august gpt/cache-august
2021-08-30.04:09:15 zpool online august gpt/cache-august
root@mowa219-gjp4-8570p-freebsd:~ # nano /boot/loader.conf
root@mowa219-gjp4-8570p-freebsd:~ # exit
logout
% sysctl hw.usb.no_boot_wait
hw.usb.no_boot_wait: 1
% grep -B 1 hw.usb.no_boot_wait /boot/loader.conf
# <https://github.com/openzfs/zfs/discussions/12519>
hw.usb.no_boot_wait=0
%

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

L2ARC: inexplicable disappearance, without removal, of cache device #12519

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

L2ARC: inexplicable disappearance, without removal, of cache device #12519

Uh oh!

Uh oh!

grahamperrin Aug 29, 2021

Replies: 1 comment · 1 reply

Uh oh!

grahamperrin Aug 29, 2021 Author

Uh oh!

grahamperrin Aug 30, 2021 Author

grahamperrin
Aug 29, 2021

Replies: 1 comment 1 reply

grahamperrin
Aug 29, 2021
Author

grahamperrin Aug 30, 2021
Author