image script can hang; temporary zpool sadness ensues #19
Description
I've seen this on two machines. When running image.sh, the script will sometimes hang here:
Jul 14 16:47:18.216 INFO STEP 1: create_dataset, from: /home/jordan/src/helios-engvm/image/templates/helios/generic-ttya-base.json
Jul 14 16:47:18.216 INFO CREATE DATASET: TEMPORARY-helios-generic-ttya-base/home
Jul 14 16:47:18.225 INFO SET DATASET PROPERTY ON TEMPORARY-helios-generic-ttya-base/home: mountpoint = /home
Jul 14 16:47:18.268 INFO STEP 1 (create_dataset) COMPLETE
Jul 14 16:47:18.268 INFO STEP 2: unpack_tar, from: /home/jordan/src/helios-engvm/image/templates/helios/generic-ttya-base.json
Jul 14 16:47:18.282 INFO exec: ["/usr/sbin/tar", "xzeEp@/f", "/rpool/images/output/helios-dev-base.tar.gz", "-C", "/rpool/images/work/helios/generic-ttya-base/a"]
The temporary zpool is in a sad state:
pool: TEMPORARY-helios-generic-ttya-base
state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://illumos.org/msg/ZFS-8000-HC
scan: none requested
config:
NAME STATE READ WRITE CKSUM
TEMPORARY-helios-generic-ttya-base UNAVAIL 0 0 0 insufficient replicas
c6t1d0 UNAVAIL 0 0 0 cannot open
errors: 250 data errors, use '-v' for a list
I was able to work around this as follows. First with a zpool export -f <temp pool name> (which will hang) and a zpool clear of the pool until the export finishes, then deleting the device in lofiadm.
$ pfexec zpool export -f TEMPORARY-helios-generic-ttya-base
# separate terminal
$ pfexec zpool clear TEMPORARY-helios-generic-ttya-base
jordan@dunkin ~/propolis $ lofiadm
Block Device File Options
$ pfexec lofiadm -d /rpool/images/work/helios/generic-ttya-base/lofi.raw
After that, I was able to run the script again and successfully create an image.