Externally-written commands

[efritz]

Zoekt's repo is currently broken up into a public cmd and an internal package for implementation. Sourcegraph has found it necessary to create a specific implementation of zoekt-sourcegraph-indexserver to meet their needs interfacing with a particular API (particularly to access gitserver data and get a list of relevant repositories for a particular indexed-searcher shard).

Similarly, Atlassian will require both a modified version of the webserver and indexserver to interface with Bitbucket Cloud APIs. My preference would be to keep our fork of Zoekt relatively clean of any Atlassian-specific indexing assumptions so that we can easily upstream changes we find beneficial to the core of Zoekt, without having to cherry-pick around code that is specific to our internal infrastructure (or simply not relevant to Zoekt as an index server).

Unfortunately, this isn't possible with the use of the internal package. How would you feel about a change that opens the internal package up for use as a library? There are already a few packages at the root of the repository, outside of the internal package, but I don't see an immediate reason for the split.

Could we move the packges in internal up one level for external import?

[efritz]

If moving packages in internal up one level is acceptable, here's a PR: #928.

If we'd like to be more selective or if there's an different solution that can make downstream changes more ergonomic, happy to discuss alternatives.

[jtibshirani]

hello @efritz 👋 I actually recently moved several root-level packages to internal to clean up our API surface: #901. The intention is for all the key entrypoints for searching and building indexes to be exported. However, I accidentally ended up moving some things that external users (like @dgruzd from GitLab) were depending on -- see #901 (comment).

I would like to keep as much logic in internal as possible, to reduce the risk of breaking external users when we make lower level changes. But I'm totally open to moving things back, for example trace is an obvious candidate to export.

Maybe you could let us know the specific packages you need, and we can move them back to root (or refactor to further clarify the API surface :)) ?

[efritz]

I'm planning to reimplement the indexserver and webserver, which seems like it depends on everything. I can see exactly what surface area is necessary, but it seems like it will require access to most of the behavior that writes and reads index files.

Alternatively, I can maintain a separate branch in my fork that just mirrors the imports I need, which wouldn't require upstream changes. I was hoping to have our fork of zoekt match the public repository as close as possible to have minimal deltas when contributing back upstream, hence the separate repo for Atlassian-specific code.

[jtibshirani]

I looked into the dependencies of zoekt-sourcegraph-indexserver and zoekt-webserver. I only found one package that definitely needs to be exported, and opened a PR to move it to the root-level: #930

As for the other internal dependencies, they are roughly...

	"github.com/sourcegraph/zoekt/internal/debugserver"
	"github.com/sourcegraph/zoekt/internal/profiler"
	"github.com/sourcegraph/zoekt/internal/tenant"
	"github.com/sourcegraph/zoekt/internal/trace"
	"github.com/sourcegraph/zoekt/internal/tracer"

Let me know if you end up needing any of these too, and we can move them.