Allow fully bypassing file descriptor cache

[marvin-j97]

Basically like RocksDB's open file limit = -1.
In that case, any table file is just opened and the read path can fully skip the FD cache.

let tree = C.descriptor_table(None).open()?;
// ...

[kunalmohan]

Hi @marvin-j97, I'm interested in contributing to fjall and would like to work on this issue to start with.

IIUC, with RocksDB max_open_file=-1, the DB keeps fds for all SSTables opened at all times (from initialization phase itself). Do you have a similar implementation in mind? Or maybe an on-demand open, but never closing it?

[marvin-j97]

Or maybe an on-demand open, but never closing it

Opening on demand would require multiple threads to synchronize, at which point you are basically using the FD cache again.

So I think the Table should just get an Option<File> e.g. pinned_file_descriptor which is the one opened on Table::recover.