Problem and value
index.IndexFile is an interface, but the directory searcher always constructs
it through index.NewIndexFile, which currently means mmap. In practice the
interface cannot be used to change the webserver's shard read strategy: a caller
must maintain a patch to search/shards.go.
That matters on network filesystems where another client controls shard
publication and backing-object lifetime. A cold access through a mapping whose
backing object became unavailable is a process-level memory fault; normal Go
error handling cannot contain it. An owned-buffer ReadAt implementation can
instead surface storage failures as ordinary read errors and can apply a bounded
page cache, but there is currently no supported way to use one in a directory
searcher.
This is one way to address the read side of the failure class in #1109 without
requiring every Zoekt deployment to stop using mmap.
Proposed API
Add an optional index-file opener to directory-searcher construction:
type IndexFileOpener func(*os.File) (index.IndexFile, error)
type DirectorySearcherOptions struct {
IndexFileOpener IndexFileOpener
}
The existing NewDirectorySearcher and NewDirectorySearcherFast functions
would preserve current behavior by using index.NewIndexFile. New
options-bearing constructors would pass the selected opener through every
initial and watcher-triggered shard load.
The opener receives ownership of the *os.File, matching the existing
index.NewIndexFile contract.
Scope
The first change should contain only the backward-compatible injection point
and tests:
- mmap remains the default;
- existing constructors and callers do not change;
- the selected opener applies to initial loads and watcher reloads;
- ownership and close behavior remain the existing
IndexFile contract.
A buffered or cached reader is a separate change after the extension point is
accepted and validated.
Problem and value
index.IndexFileis an interface, but the directory searcher always constructsit through
index.NewIndexFile, which currently means mmap. In practice theinterface cannot be used to change the webserver's shard read strategy: a caller
must maintain a patch to
search/shards.go.That matters on network filesystems where another client controls shard
publication and backing-object lifetime. A cold access through a mapping whose
backing object became unavailable is a process-level memory fault; normal Go
error handling cannot contain it. An owned-buffer
ReadAtimplementation caninstead surface storage failures as ordinary read errors and can apply a bounded
page cache, but there is currently no supported way to use one in a directory
searcher.
This is one way to address the read side of the failure class in #1109 without
requiring every Zoekt deployment to stop using mmap.
Proposed API
Add an optional index-file opener to directory-searcher construction:
The existing
NewDirectorySearcherandNewDirectorySearcherFastfunctionswould preserve current behavior by using
index.NewIndexFile. Newoptions-bearing constructors would pass the selected opener through every
initial and watcher-triggered shard load.
The opener receives ownership of the
*os.File, matching the existingindex.NewIndexFilecontract.Scope
The first change should contain only the backward-compatible injection point
and tests:
IndexFilecontract.A buffered or cached reader is a separate change after the extension point is
accepted and validated.