Trouble reading files between 8KB and 512GB

[mennozon]

I'm having trouble reading data from a certain vmfs5 datastore, I've successfully migrated many machines all varying sizes from another datastore, also vmfs5.

The problem seems to be any file between 8KB and 512GB (though the 512GB is a guess, 500GB doesn't work but 718GB and larger images work)

My testcase is a small text file that I appended until it became unreadable through vmfs-fuse.

Test case: a 2784 byte file is readable with vmfs-fuse, on a Vmware node I copy the file to a new file, still works, now I append the original file to the copy, data is still correct, I once more append the data and I have a 8352 byte file that has 'garbage' content but stays the same md5sum if I re-read the file.

Editing the file and reducing the size does not make it readable again.

With vmfs-fuse in debug mode nothing obvious (for me) changes, still same node id.

I've tried applying pull request #14 (which I needed anyway for 256GB+) but that does not help for the issue of the 8KB file.

Any suggestions on how to further debug this issue would be greatly appreciated.

This is the debug log of the successful and failed reading of a file.

Start vmfs-fuse -d

FUSE library version: 2.9.7
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
INIT: 7.19
flags=0x00000001
max_readahead=0x00020000
max_write=0x00020000
max_background=0
congestion_threshold=0
unique: 1, success, outsize: 40

cd /to/dir

unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 3917
unique: 2, success, outsize: 120
unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 50, pid: 3917
unique: 3, success, outsize: 144

Copy test file on the Vmware node to a new file and then read it through vmfs-fuse.

unique: 4, opcode: GETATTR (3), nodeid: 239090948, insize: 56, pid: 4727
unique: 4, success, outsize: 120
unique: 5, opcode: LOOKUP (1), nodeid: 239090948, insize: 46, pid: 4727
unique: 5, success, outsize: 144
unique: 6, opcode: OPEN (14), nodeid: 318806468, insize: 48, pid: 4727
unique: 6, success, outsize: 32
unique: 7, opcode: READ (15), nodeid: 318806468, insize: 80, pid: 4727
unique: 7, success, outsize: 2800
unique: 8, opcode: FLUSH (25), nodeid: 318806468, insize: 64, pid: 4727
unique: 8, error: -38 (Function not implemented), outsize: 16
unique: 9, opcode: RELEASE (18), nodeid: 318806468, insize: 64, pid: 0
unique: 9, success, outsize: 16

Append the data on the Vmware node once more and read through vmfs-fuse.

unique: 10, opcode: GETATTR (3), nodeid: 239090948, insize: 56, pid: 4734
unique: 10, success, outsize: 120
unique: 11, opcode: LOOKUP (1), nodeid: 239090948, insize: 46, pid: 4734
unique: 11, success, outsize: 144
unique: 12, opcode: OPEN (14), nodeid: 318806468, insize: 48, pid: 4734
unique: 12, success, outsize: 32
unique: 13, opcode: READ (15), nodeid: 318806468, insize: 80, pid: 4734
unique: 13, success, outsize: 5584
unique: 14, opcode: RELEASE (18), nodeid: 318806468, insize: 64, pid: 0
unique: 14, success, outsize: 16

Append the data on the Vmware node once more and read through vmfs-fuse.

unique: 15, opcode: GETATTR (3), nodeid: 239090948, insize: 56, pid: 4735
unique: 15, success, outsize: 120
unique: 16, opcode: LOOKUP (1), nodeid: 239090948, insize: 46, pid: 4735
unique: 16, success, outsize: 144
unique: 17, opcode: OPEN (14), nodeid: 318806468, insize: 48, pid: 4735
unique: 17, success, outsize: 32
unique: 18, opcode: READ (15), nodeid: 318806468, insize: 80, pid: 4735
unique: 18, success, outsize: 8368
unique: 19, opcode: RELEASE (18), nodeid: 318806468, insize: 64, pid: 0
unique: 19, success, outsize: 16