Skip to content

Allow directory searchers to use a custom IndexFile opener #1137

Description

@aaaaaandrew

Problem and value

index.IndexFile is an interface, but the directory searcher always constructs
it through index.NewIndexFile, which currently means mmap. In practice the
interface cannot be used to change the webserver's shard read strategy: a caller
must maintain a patch to search/shards.go.

That matters on network filesystems where another client controls shard
publication and backing-object lifetime. A cold access through a mapping whose
backing object became unavailable is a process-level memory fault; normal Go
error handling cannot contain it. An owned-buffer ReadAt implementation can
instead surface storage failures as ordinary read errors and can apply a bounded
page cache, but there is currently no supported way to use one in a directory
searcher.

This is one way to address the read side of the failure class in #1109 without
requiring every Zoekt deployment to stop using mmap.

Proposed API

Add an optional index-file opener to directory-searcher construction:

type IndexFileOpener func(*os.File) (index.IndexFile, error)

type DirectorySearcherOptions struct {
    IndexFileOpener IndexFileOpener
}

The existing NewDirectorySearcher and NewDirectorySearcherFast functions
would preserve current behavior by using index.NewIndexFile. New
options-bearing constructors would pass the selected opener through every
initial and watcher-triggered shard load.

The opener receives ownership of the *os.File, matching the existing
index.NewIndexFile contract.

Scope

The first change should contain only the backward-compatible injection point
and tests:

  • mmap remains the default;
  • existing constructors and callers do not change;
  • the selected opener applies to initial loads and watcher reloads;
  • ownership and close behavior remain the existing IndexFile contract.

A buffered or cached reader is a separate change after the extension point is
accepted and validated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions