Handle FIlesystems that don't support locking gracefully

[cmhulbert]

@mkitti mentioned on slack that when attempting to lock for reading a file over an SMB mount on MacOS, you get an OperationNotSupported Exception. (https://bugs.openjdk.org/browse/JDK-8167023). The JDK issues is from 2016, but I did test and this is still relevant, even with our current VolatileReadData #203 approach.

I tested this against the current 3.5.1 release and we do run into this, but we just let the user read anyway (this silently returns as if the lock succeeded on any IOException)

For the VolatileReadData, I'm thinking we can be a bit better about it. I'm proposing when we catch an IOException, we try to read without locking, but immediately materialize the full file, and then back the VolatileReadData with that. that would ensure future read are backed by the single materialize call, so even if the underlying filesystem changed, it should be consistent. Since we can't get the read lock, there isn't a strong guarantee that the data isn't changed during the materialize call itself, but that is a limitation of the filesystem, not our API.

[mkitti]

The issue is not that it errors, but rather we do not have any file locking.

TensorStore creates .lock files manually to do this, which I think may interface with their atomic reads and writes system.

[tpietzsch]

TensorStore creates .lock files manually to do this, which I think may interface with their atomic reads and writes system.

How does that work exactly?
I looked at the javadoc of File.createNewFile which is how one would try to do this presumably. It explicitly says:

"Note: this method should not be used for file-locking, as the resulting protocol cannot be made to work reliably. The FileLock facility should be used instead."

No further explanation though, and google / chatgpt wasn't very helpful either.

[mkitti]

From Gemini:

When TensorStore handles sharded data, .lock files manage concurrency at the shard level rather than the individual chunk level. Since a single shard file contains multiple data chunks, locking the entire shard ensures that concurrent updates from different processes don't corrupt the shared file structure.
Here is how the mechanism works with shards:

• Shard-Level Granularity:
  For formats like Zarr V3 shards or Neuroglancer precomputed sharded volumes, the shard is the smallest unit of persistent storage. TensorStore places a .lock file on the specific shard being modified to prevent other processes from writing to it simultaneously.
• Write Coalescing: To avoid excessive locking and unlocking, TensorStore often uses transactions or internal write contexts. This allows it to buffer multiple chunk updates in memory and then perform a single, atomic write to the shard file under one lock operation.
• Optimistic Concurrency: On modern filesystems, TensorStore defaults to "optimistic" locking. It creates a temporary file (often with a .unique_id.lock or similar suffix), writes the updated shard data there, and then uses an atomic rename operation to replace the original shard.
  This ensures that readers never see a partially written shard.

[mkitti]

Also see https://google.github.io/tensorstore/kvstore/file/index.html#json-Context.file_io_locking

[cmhulbert]

The issue is not that it errors, but rather we do not have any file locking.

TensorStore creates .lock files manually to do this, which I think may interface with their atomic reads and writes system.

I'm not sure I agree that the issue is not the graceful error handling. Currently we rely on the various backends to support atomic read/writes. The only way to make a claim about gaurranteed locking at the N5 API level is something like you are suggesting tensor store does, where we introduce our own lock file or some equivalent. I'm not sure that's something we want to add at this point. Historically we have relied on the backend's atomic gaurantees (or not), and it would fallback to not locking.

I think the idea to gaurantee locking at the API level is nice, but I think it would be a large effort, and I'm not sure that's the scope of this.

[cmhulbert]

related though, I did verify this method works successfully on a mac, interacting with files over a SMB mount:

Files.move(pathObj, destPath, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);

Though I'm not sure we actually want to do this to try and guarantee atomic writes, since this introduces it's own complexities (where to write the tmp file initially; local tmp file store probably isn't on the same mount; could place it next to the target file, but then errors could lead to tmp files cluttering the dataset)

[cmhulbert]

Closing this for now. The behavior in the most current alpha release is consistent with previous behavior. on SMB filesystems with MacOs, it will attempt to lock, but failing to do so, will attempt to read/write anyway. If that succeeds, no exception will throw.

This can be controlled by setting a jvm property of

n5.ioPolicy
"strict" -> Fail if unable to lock
"unsafe" -> don't attempt to lock
"permissive" -> attempt to lock, if unable, attempt without locking 

permissive is the default, as it matches the previous behavior

names are subject to change, but that's what it is currently. Also it is currently a static property, not configurable per KVA