Skip to content
This repository was archived by the owner on Jun 4, 2026. It is now read-only.

rfc-619-code-intelligence-data-platform.md

Latest commit

 

History

History
466 lines (390 loc) · 27.1 KB

rfc-619-code-intelligence-data-platform.md

File metadata and controls

466 lines (390 loc) · 27.1 KB
Raw

RFC 619 IMPLEMENTED: Code Intelligence Data Platform

Date: 2022-03-21
Status: APPROVED
Recommender: Eric Fritz
Decider: Eric Fritz
Input providers: Code intelligence team
Approvers (please review by EOD 2022-04-04):

  • Please approve platform description: Varun Gandhi, Thorsten Ball, Noah Santschi-Cooney
  • Please approve milestone definitions: TJ Devries Chris Wendt

Approvals:

  • …for platform description (and API drafts): Varun Gandhi, Thorsten Ball, Noah Santschi-Cooney
  • …for milestone definitions: Chris Wendt

Execution status

Follow progress of execution with the tracking issue.

Background

Several recent efforts have signified an increasing need to unify the code intelligence data platform, including:

  • RFC 519 APPROVED: LSIF typed - a Protobuf-based LSIF schema for improved maintainability and faster performance
    • The LSIF-typed protocol defines a shared vocabulary. Most important to note is the idea of document and symbol-based (not location-based) elements, as well as a language agnostic yet structured symbol identifier which can encode information such as definition location and symbol hierarchies with nothing but a name constructed by convention. We should continue standardizing on these types of ideas and move forward on a standard interchange between backend services. See the LSIF-typed platform roadmap document for additional technical details.
  • RFC 593: Unlocking use cases based on the dependency graph of repositories
    • Exposing dependencies via search filters is a great example of cross-team synergy that can be enabled by a properly designed data platform. Historically team-external use-cases and our APIs have not aligned closely enough for the effort to be prioritized. We seem to be entering an era where new and exciting product use cases will come about from the obvious composition of such focused APIs.
  • RFC 542: Rockskip indexed search and RFC 616: Squirrel: tree-sitter code intel
    • Rockskip and Squirrel are both good examples of filling in an unaddressed area of our design space. However, the lack of a concrete definition of the code intelligence data platform caused some confusion about in which level of abstraction these concepts are being applied. While these efforts are orthogonal to many of the code intelligence team’s in-flight concerns, it was not immediately obvious whether or not there is an ordering requirement between them.
  • Build a better references panel (#30227)
    • The effort to improve our code navigation UX has left us with a direct line of sight to a lot of historic tech debt - most of which stems from our APIs being a direct clone of the LSP protocol. We would like to move into a direction of bulk-loading as much data as possible (i.e., being document-centric, à la LSIF-Typed) rather than responding solely to user input (i.e., being locations-centric, à la LSP). The former is much more amenable to concerns related to caching as well as incrementality. However, this requires a significant change to the API surface of the code intelligence data platform, including the GraphQL API and, down the line, the extension host API.

In brief, these efforts show an immediate need to be able to correlate and aggregate data from differents sources at a much lower level (within an API boundary instead of re-implemented on several clients), set precedent for where code goes in relation to code intelligence domains (tackled uniformly by defining services in this proposal), and a way to set ownership boundaries between product teams.

High-level platform description

The following diagram (source) will be referenced frequently in this document and serves as a partial visualization of how I’ve been thinking about the organization and ownership of code intelligence data lately. Intuition for choices will be detailed below.

architecture

This diagram (and by extension the entire document) does not represent reality in its current form, nor is it a prescriptive fixed proposal of what we should build as a goal. This is nothing more than a partial snapshot into one possible technical vision of such a data platform. The data we ingest and the data we’ll serve will evolve over time, and it is important that the data platform API surface remains flexible to enable rapid responses to changes in business requirements.

Diagram legend

The red-shaded boxes in the diagram above represent compute nodes and are present to show in what portion of a deployment a piece of code runs. The frontend compute node serves HTTP and GraphQL traffic. The worker compute node handles the scheduling of background tasks that can be run periodically or in response to a queued event. The executor and indexer compute nodes run on external hardware (the latter running in the executor or in CI independently from Sourcegraph). The dotted lines originating from compute nodes indicate that the service is running (fully or partially) on that node.

The unshaded boxes represent a portion of the data platform API. Here, we call them services, but do not correspond one-to-one to a compute node. The important thing to note about a service is that it forms a well-defined contract for users of the data platform (including team-external users). Whether the service is invoked via Go function call or via a network request is irrelevant.

The remaining (solid black) arrows show control or data relationships between APIs. Green arrows indicate that a response is encoded as LSIF-typed (a standardized format that services within our shared platform can use to communicate). Orange arrows indicate that generated LSIF-typed payloads can also be persisted, either as a write-through cache or as an off-line indexing step. Note that all the green and orange arrows (ultimately) target the LSIF-Typed Layer, which indexes, categorizes, and mixes data from different sources. All consumers of this layer can access data from all sources of intelligence (indexers, lockfile parsers, tree-sitter, syntect, ctags, etc) in a uniform way: LSIF-Typed formatted elements. Downstream consumers of this layer may themselves return data of a similar shape (for example, the GraphQL schema will have a lot of the same concepts (e.g., symbols, documents, syntax kinds) as LSIF-Typed) but will be encoded in a format more appropriate for the given client (e.g., GraphQL responses for Apollo SPA clients).

Services overview

The six services defined in the diagram above are explained in more detail below.

Documents service

Exposes data about a single text document. This service can be used for high-level information about a document (constructing a symbols outline) as well as in-depth code navigation (bulk calculate all symbol occurrence information in a range). This service can be seen as the layer that mixes all information about a single document, and filters out what isn’t currently necessary.

Background tasks may create indexes for code, or enhance existing indexes with additional information.

Service API methods:

  • Document(ctx context.Context, opts DocumentOpts) ([]BadgedDocument, error)
    • DocumentOpts includes repo, rev, path, optional range of occurrences to return (return all occurrences in document by default); The GraphQL transport layer will also allow us to toggle on or off parts of the result which may be relevant for some users but not others
    • We return a slice here instead of a single document as we may return data from more than one source. Relevantly, BadgedDocument contains three major pieces of data:
      • The fidelity of information (search-based, or from a particular upload from this commit, etc)
      • A list of occurrences in the document (or provided range), including fields like range, symbol, role, syntaxKind, etc, matching LSIF-Typed. Occurrence information should give us stencils for the current blob view.
      • A list of symbols referenced by the document. This includes some rudimentary things like hover strings, but maybe not any locations except for something cheaply calculable like local variable navigation. Symbol information can be used to construct a symbols hierarchy for the sidebar.

Symbols service

Exposes data about a single symbol (or small set of symbols) across multiple documents and indexes (may use the dependencies service to do so). Exposes deep data about a single symbol that the document-level information may not be able to efficiently aggregate. Data will generally be calculated online during a user query and will not be persisted for later queries (save for the use of high-locality caches if proven to be beneficial on common query paths).

The GraphQL transport layer for this service will make the following methods appear as a single call, and each field will be separately paginateable. Specifically, the new reference panel will be able to ask for the hover text as well as the first page of all locations (definitions, refs, implementations, type defs, etc). Subsequent pages for additional locations can be made with a subsequent GraphQL request (specific to that location type) in response to a specific user interaction.

Service API methods:

  • Hover(ctx context.Context, symbol SymbolName) (BadgedHover, error)
    • Note: SymbolName here may be expanded to include an optional source location. This applies to the operations below as well. We can likely just talk about this at the PR level once we make progress.
  • **{Operation}**s(ctx context.Context, symbol SymbolName, opts **{Operation}**Opts) ([]BadgedLocation, error)
    • Where {Operation} is one of Definition, Reference, Implementation, or TypeDefinition.
    • Options include filtering, pagination, etc. Filters for references (for example) can be things like “assignments only” at this level (we can analyze syntax kind/occurrence roles in LSIF-typed), not just “filename containing”-type filtering
    • BadgedLocations contains a way to navigate to another piece of code on the same Sourcegraph instance, but also references the fidelity of information (search-based, or from a particular upload from this commit, etc).

Dependencies service

Exposes data about relationships between repositories and, eventually, dependencies between other resources such as uploads, symbols, etc. This service responds with data that cannot be answered by analysis of a single document (in isolation) and the response data will generally need to be refined. This service is best used as a cache to enable linking of elements across index boundaries.

Service API methods:

  • Dependencies(ctx context.Context, repoRevs map[api.RepoName]types.RevSpecSet) (map[api.RepoName]types.RevSpecSet, error)
    • The current implementation of this method takes a bag of repo/revision pairs, and returns a second set of repo/revision pairs describing the transitive dependencies of the inputs (for those that resolve). We currently will try to eagerly load package repos to fulfill these repos.

Uploads service

Exposes bookkeeping data (not raw data) about a precise code intelligence upload. This service may also expose imprecise upload records created automatically by lightweight analysis mechanisms in the future, if we would like such a unified interface. For now, we assume we only expose precise code uploads (index files uploaded via CI or auto-indexing), and not sources such as ctags or tree-sitter analysis.

Background tasks should include current janitorial tasks as well as commit graph updates (which are hidden behind this service boundary).

Service API methods:

  • List(ctx context.Context, opts ListOpts) ([]Upload, error)
    • ListOpts include filtering, pagination, ordering, etc
  • Get(ctx context.Context, id int) (Upload, bool, error)
  • GetBatch(ctx context.Context, ids …int) ([]Upload, error)
    • In the related PR, consider partial success of batch operations
  • Enqueue(ctx context.Context, state State, reader io.Reader) error
    • State contains identity and progress of an incomplete upload
    • Reader contains the a full or partial LSIF upload that should be associated in full or in part with the upload identified by state
  • Delete(ctx context.Context, id int) error
  • CommitsVisibleToUpload(ctx context.Context, id int) ([]string, error)
  • UploadsVisibleToCommit(ctx context.Context, commit string) ([]Upload, error)

Policies service

Exposes the management of code intelligence (retention, auto-indexing, etc) policies. This service should also encapsulate the logic needed to match arbitrary items (commits, uploads, indexes, etc) against the current set of policies, which can be useful in other services (such as the upload expirer subpackage).

In the background, this service is responsible for ensuring that policies containing repository patterns are efficiently resolved (without a full regex search over all repositories in Cloud). We currently maintain such a denormalized pattern expansion in the database with a periodic worker.

Service API methods:

  • List(ctx context.Context, opts ListOpts) ([]Policy, error)
    • ListOpts include filtering, pagination, ordering, etc
  • Get(ctx context.Context, id int) (Policy, bool, error)
  • Create(ctx context.Context, policy Policy) (hydratedPolicy Policy, _ error)
  • Update(ctx context.Context, policy Policy) (hydratedPolicy Policy, _ error)
  • Delete(ctx context.Context, id int) error
  • CommitsMatchingRetentionPolicies(ctx context.Context, repoID int, policies []Policy, instant time.Time, commitSubset …string) (map[string][]Policy, error)
  • CommitsMatchingIndexingPolicies(ctx context.Context, repoID int, policies []Policy, instant time.Time) (commitsToPolicies map[string][]Policy, error)

Auto-indexing service

Exposes information about auto-indexing configuration and jobs, and allows users to schedule explicit jobs to be performed by an external executor service. This service encapsulates the (Sourcegraph-instance internal) logic about job configuration and inference.

Service API methods:

  • List(ctx context.Context, opts ListOpts) ([]IndexJob, error)
    • ListOpts include filtering, pagination, ordering, etc
  • Get(ctx context.Context, id int) (IndexJob, bool, error)
  • GetBatch(ctx context.Context, ids …int) ([]IndexJob, error)
  • Delete(ctx context.Context, id int) error
  • Enqueue(ctx context.Context, jobs []IndexJob) error
  • Infer(ctx context.Context, repoID int) ([]IndexJob, error)
  • Repository index configuration CRUD methods

Proposal

We transform the current code intelligence backend code into the end state described above. This can be done piecewise, moving or copying existing code into a new skeleton service boundary and updating callers to use the new API.

In code, each service will be implemented as a simple Go package in (enterprise/)internal/codeintel that follows several conventions. See internal/codeintel/dependencies as a proto-example. Each such package must:

  • Be lazily self-instantiable. This is implemented in the proto-dependencies API as a package-level GetService function that instantiates and returns a (possibly singleton, but that’s an implementation detail) production instance of the service. It takes a database handle as an argument, which is guaranteed to exist in any client location that could query this service.
  • Expose its public behaviors in a single, focused Service struct defined at the top of the package. This service struct should be instantiated by the package’s factory function. The domain of this service should be fairly cohesive and large services should be decomposed further when natural. The internals of this service should be completely opaque to the service’s callers. A service should be largely in charge of its own persistence and caching strategies. A service may make use of any other downstream in-process or over-the-network service to fulfill its request.
  • Be fully instrumented at the top level. The observability package must wrap each public method of the Service struct.
  • Implementation details should be organized hierarchically under the service in which it is used. Code shared by multiple services must be stateless. Keep the “public API surface” portion of the package as clean as possible. We’re trying to promote cross-team use of these APIs, and having clear ownership boundaries will be necessary for efficient inter-team execution.

Milestones

We can roughly break the remainder of the work up into several deliverables that build off of each other semi-linearly. The exact implementation plan may be difficult to make correctly on the first try up-front. The milestones below should be done mostly in the order listed, but unobvious dependencies in code may cause us to re-evaluate the order that we implement these services, or how many of them we are working on concurrently.

Milestone #1: Generate service skeletons

Generate a new directory/trivial code structure (mirroring the approved API above) for the documents, symbols, uploads, policies, and auto-indexing services. The dependencies service already exists, but we might want to add skeleton background jobs for proactive lockfile indexing. Each service method will initially return an Unimplemented error. Later milestones will move code into the correct services without changing the behavior of the relocated code significantly.

Milestone #2: Initial documents service implementation

Move (or copy) the following existing code into the skeleton documents service. Whether or not all consumers of the old code are immediately able to move to the new service or not depends on each service and is difficult to predict in advance. For each of these initial service definitions, we may need to maintain a new but unused version while we remove all references to the old one and abandon it over a short period of time.

Milestone #3: Initial symbols service implementation

Move (or copy) the following existing code into the skeleton symbols service. See M2 for notes on success criteria.

Milestone #4: Initial uploads service implementation

Move (or copy) the following existing code into the skeleton uploads service. See M2 for notes on success criteria.

Milestone #5: Initial policies service implementation

Move (or copy) the following existing code into the skeleton policies service. See M2 for notes on success criteria.

Milestone #6: Initial auto-indexing service implementation

Move (or copy) the following existing code into the skeleton auto-indexing service. See M2 for notes on success criteria.

Milestone #7: Consolidate dependencies service

Symbols service does a lot of cross-index/cross-repository queries, which requires duplicate work that belongs to the dependencies service. Move the relevant code behind the correct boundary.

Milestone #8: Consolidate policies service

Upload expiration and auto-indexing tasks do a lot of work that belongs to the policies service. Move the relevant code behind the correct boundary.