Change `FSDirectory.Open()` defaults to match .NET-specific platform and performance concerns

[NightOwl888]

Is there an existing issue for this?

• I have searched the existing issues

Task description

The FSDirectory.Open() method contains some business logic to select the best FSDirectory implementation for the current platform.

lucenenet/src/Lucene.Net/Store/FSDirectory.cs

Lines 184 to 203 in 5976abd

	/// <summary>
	/// Just like <see cref="Open(DirectoryInfo)"/>, but allows you to
	/// also specify a custom <see cref="LockFactory"/>.
	/// </summary>
	public static FSDirectory Open(DirectoryInfo path, LockFactory lockFactory)
	{
	if ((Constants.WINDOWS || Constants.SUN_OS || Constants.LINUX) && Constants.RUNTIME_IS_64BIT /*&&
	MMapDirectory.UNMAP_SUPPORTED*/) // LUCENENET specific - unmap hack not needed
	{
	return new MMapDirectory(path, lockFactory);
	}
	else if (Constants.WINDOWS)
	{
	return new SimpleFSDirectory(path, lockFactory);
	}
	else
	{
	return new NIOFSDirectory(path, lockFactory);
	}
	}

However, this business logic is based on what is the best option in Java. The 3 directory types may have very different performance characteristics in .NET due to a fundamentally different approach to providing random access and memory-mapped access to files.

We need to do some investigation based on performance profiles which directory option correctly maps to:

• Specific operating systems
• Runtime architecture (x64/x86/ARM64, etc.)
• Target Framework

Note that this is currently blocked by #1267. We should do this investigation based on the new implementation.

AOT Limitations

After researching AOT limitations (see #1278), we need to ensure that the MMapDirectory is never selected as the default option where dynamic code is not supported. The canonical way to do that is to check:

if (!System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported)
{
    // AOT is likely in use and dynamic features are disabled.
    // Fallback to standard FileStream IO or handle gracefully.
}

Although, we should add that check to the Constants class so it can be shared project-wide. We could fall back to NIOFSDirectory instead of SimpleFSDirectory if it performs better, but it will take some experimentation to make that determination.

Note that the API is not supported on .NET Framework or .NET Standard 2.0, but we can build an internal conditionally compiled stub that is hard coded to true for that case as was already done in J2N.

.NET Framework/.NET Standard 2.0 Limitations

The System.IO.RandomAccess class did not exist before .NET 6. As a result the NIOFSDirectory is not currently a good candidate for being the default on anything other than .NET Core.

There has been some discussion about possibly making a port of System.IO.RandomAccess (or even a cut down version to patch our needs), but until it is actually done we should not select this directory as the default on older TFMs than .NET Core. Note that NIOFSDirecotry only uses the Read(SafeFileHandle, Span<byte>, long) overload.

macOS

In Java, the NIOFSDirectory was chosen over MMapDirectory on macOS. I suspect that once #1278 is completed MMapDirectory will probably be the best option instead, but it will require testing to confirm.

---

I suspect there will be other concerns down the road on which directory is best on a given platform, but we can deal with those concerns as they become known. It might not hurt to have AI audit our configuration and make some suggestions, though.

[paulirwin]

We could fall back to NIOFSDirectory instead of SimpleFSDirectory if it performs better, but it will take some experimentation to make that determination.

There are various benchmarks I ran on different OSes and architectures in the issue and comments here: #1151 – and they showed that NIOFS is consistently slower than SimpleFS in Lucene.NET under concurrent requests, when it is faster in Java Lucene. This lines up with the notes in our NIOFS class docs. Since much real-world use is concurrent, I think that's pretty notable to consider.

IMO, fixing the performance of our NIOFS to be better than SimpleFS is low priority since SimpleFS performs so well. But, I would welcome any contributions that improve that perf (and I might get to it myself later if no one beats me to it, once done with everything else).

[NightOwl888]

@paulirwin

Thanks, I wasn't aware of those benchmarks. However, I don't see any benchmarks that have taken into account the switch to RandomAccess, which was made in #1195. Those benchmarks predate that change, and I suspect we will see much better performance on .NET Core now, but whether it is better or worse than the other 2 options is still to be determined.

.NET Standard/.NET Framework are still using the old slow approach of using a lock to synchronize along with the Seek() method of Stream to fake random access, and it is really no surprise that it performs poorly. But I am curious to know how well it does now that we are calling the RandomAccess.Read() method on .NET Core.

I agree that fixing NIOFSDirectory for .NET Framework is low priority due to there being 2 other FSDirectory options available. The current settings ensure that NIOFSDirectory is not returned on Windows (which probably covers 99% of .NET Framework users), but it is still possible for this to be returned as a default setting on .NET Standard 2.0 and we should close that gap if we aren't going to fix it. We may even consider removing it from the compilation on those platforms. But leaving it in may provide more motivation for contributions to fix it.

[paulirwin]

See my comment on #1013, w/r/t the proposed fixes in #1361.

Unless we can solve the non-determinism problem with MMapDirectory cleanup in #1361, my vote is that we should never return MMapDirectory by default on any platform from FSDirectory.Open. We should benchmark between NIOFS and SimpleFS, and choose one of those (possibly different per-platform, or maybe one for all platforms.) I spent many days going around in circles on trying to solve this, and I've run out of energy and ideas to try. If someone wants to try to solve it, my PR is open to collaboration. Even the Lucene team themselves could not solve this without the FFM API being available in Java 22+.

Users can always opt-in to using MMapDirectory when it makes sense for them, and after they test it to ensure it doesn't have problems, if the extra throughput is worth the tradeoff of possibly having file handles not cleaned up until GC time.

Or, perhaps the community feels like this constraint is totally a non-issue, and the performance is worth it. If so, I'm open to MMapDirectory being a default. But, in lieu of that, I'm going to vote for the conservative and safer approach that still has good general performance.