Inconsistent behavior and eventual BSOD after playing with snapshots, clones and datasets

[Elfy]

System information

Type	Version/Name
Distribution Name	Windows 10
Distribution Version	10.0.19041.1
Architecture	x64
OpenZFS Version	zfswin-2.3.1rc14

Describe the problem you're observing

Some datasets' mounts become inaccessible after doing a lot of creation/destruction of datasets.
In attempt to fix it I tried unmounting/remounting them, which worked for some but others got me error "not currently mounted", despite mount point existing. Trying to mount such dataset gave error "Unknown error". Some of successfully remounted datasets became inaccessible again at some point. Finally I tried to fix unmounted but impossible to mount dataset with rename. Rename was successful, but then, alas, my memory gets foggy, and BSOD ate some of the latest commands. The gist of it is that mounting created yet another inaccessible mountpoint with new name (old name is still present btw, despite associated dataset being renamed and apparently nothing being mounted under it!) and then either on trying to unmount this new mountpoint or remount it system crashed.

Describe how to reproduce the problem

I can't give perfect recreation of everything done, but roughly I created a pool (single hdd, entirely free of partitions), copied a few folders, made snapshot of the result, made 2 clones of that snapshot, modified a few bytes in a file in both of the clones. Created 2 more datasets that I just copied some test data in and didn't do anything weird with them (one of said datasets became eventually inaccessible, but could be remounted successfully). Then I went in a loop for a few hours: create new dataset with different recordsize and compression, copy test data, get stats, destroy dataset. After that I noticed inaccessibility of some datasets. Reboot didn't help with all of them. And then rename -> dismount/mount led to bsod. Right now the only inaccessible is second clone, with two mountpoints shown, one for old name and one for new.

Include any warning/errors/backtraces from the system logs

info.txt
stack.txt
cbuf.txt

I'm going to leave this pool as is for a bit, in case something useful for debugging can be extracted out of it.

Somewhat unrelated: the driver does seem to leak, the memory usage quickly stabilized after a I started doing create/copy/destroy loop, but then ever so slowly kept inching up, reaching almost 2 extra GB after ~ 2h. Also, default limits seem unreasonable on windows, with default config the system starts lagging before driver starts giving back memory (if it ever does). The above was done with zfs_arc_max=4GB, with memory usage initially stabilizing at 8GB, then reaching 10GB/32GB.

[lundman]

Arg1: 0000000000000004, Invalid enumeration of deleted PDO
	An enumerator has returned a PDO which it has previously deleted
	using IoDeleteDevice.
Arg2: ffff8d87f2326080, PDO with DOE_DELETE_PENDING set.

Ah interesting - I need to be careful not to include any device slated for delete.

On memory leak, we are working on that over in the other issue, feel free to lend a hand if you can run the special build as well

[Elfy]

Any idea what's up with shroedinger's mount? I did manage to get access to the data in the dataset that couldn't be mounted by making a snapshot of it, cloning and mounting the clone (can't mount snapshot directly because "parent dataset 'tank/bclonedfs3' must be mounted"). And the data seem to be fine at least.

[Elfy]

UPD: after installing memleak version from another issue and importing my pool, the original problematic folder remained, can not be deleted or anything. After running zfs mount, everything seems to be mounted, but now the latest clone became inaccessible, while previous problematic one became accessible under its new name.
I wish I knew more about zfs, this issue is so weird I want to dig into internals :D

[lundman]

There might be a race condition with mount sometimes returning error, generally I would suggest doing "zfs unmount dataset ; zfs mount dataset" to get it to mount better. mount/unmount is probably the weakest link, since Windows do not have it as a concept. So this includes clones, snapshots mounting etc - so that is some heavy testing of that code, which is good, but not entirely surprising if you find quirks.

But even if you get mounts that stall, a reboot should clear it out. Your data should still be there, just the attachment to Windows did not take.

[Elfy]

UPD: After another reboot I have now 5 inaccessible datasets (and one mount point leftover from renamed dataset).

Some of the inaccessible datasets have nothing to do with contrived sequence of action from original report -- just an empty new dataset.

Some of them can be unmounted/mounted and become accessible.

Some other datasets become inaccessible after that.

And after unmounting/mounting a different inaccessible dataset the ones above become accessible again.

I don't think race condition covers it. xD Something is very broken.

Also I noticed that mounting varies widely in how much time it takes, sometimes it's instantaneous, sometimes there's a few seconds of delay.

Also zdb always fails with "zdb: can't open 'tank': No such file or directory" with any parameters requiring pool name.

Is there anything I can do to make sense out of it?

Also this is the mysterious leftover object as reported by junction:

D:\bclonedfs212: MOUNT POINT
   Print Name     : (null)\
   Substitute Name: (null)\

Even junction can't delete it though. "Error deleting bclonedfs212: Access is denied." mountvol can't either

mountvol d:\bclonedfs212\ /D
The handle is invalid.

Sorry for the wall of text, I'm just trying everything I can think of in hope it helps someone to make sense out of it.

[Elfy]

Empty mount point fixed itself (became normal empty directory that I could delete) after I tested zfs send (which is awesome btw).

shrug I don't know anymore. I'm in love with zfs features. But windows driver... well, the warning about never ever using it on live system was very fair. Although, despite very rough ride, no data was actually lost throughout torture session I put it through. So that's something :D

[lundman]

The core of ZFS is untouched, so I am confident about its ability to retain your data. The hard bits are in the glue that interfaces with Windows, which need to be made silky smooth. That's just how Windows talks to ZFS, so not about data integrity..

But I can see if a mount fails, which to me is trivial - just mount it again, to a new users it would seem everything was lost - so I do need to figure out what goes wrong, and handle it better.

[Elfy]

I managed to reproduce weird mountpoint on entirely fresh pool:

D:\>fsutil reparsepoint query temp
Reparse Tag Value : 0xa0000003
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Mount Point
Substitue Name offset: 0
Substitue Name length: 14
Print Name offset:     16
Print Name Length:     14
Substitute Name:       (null)\
Print Name:            (null)\

D:\>zfs unmount pool1/temp
cannot unmount 'pool1/temp': not currently mounted
D:\>fsutil reparsepoint delete temp
Error:  The filename, directory name, or volume label syntax is incorrect.
A local NTFS volume is required for this operation.
D:\>zfs mount pool1/temp
sending mountpoint: '\??\D:/temp'
cannot mount 'pool1/temp': Unknown error

The end. It was mounted. The only thing I did with this pool is create 6 datasets, remove one (5th), and create new one (same name). And there were a bunch of files copying in background.

[Elfy]

Should we be allowed to just rmdir active mountpoint? Because it works on nonglitched ones.

[lundman]

Oh that is quite interesting.. especially the "(null)" targets. I wonder if we fail to remove a reparsepoints sometimes, then future mounts will fail. (You have to remove the reparsepoint attribute first, then you can rmdir it)