Skip to content
This repository was archived by the owner on Jun 4, 2026. It is now read-only.

rfc-642-code-intel-protocol-strategy.md

Latest commit

 

History

History
217 lines (178 loc) · 10.6 KB

rfc-642-code-intel-protocol-strategy.md

File metadata and controls

217 lines (178 loc) · 10.6 KB
Raw

RFC 642 REVIEW: Code Intel Index Format Strategy

Recommender: Varun Gandhi
Date: Mar 25, 2022~Apr 15, 2022
Status: REVIEW
Decider: Varun Gandhi
Input providers: María Craig, Owen Convey, Olaf Geirsson, TJ Devries, Eric Fritz, Thorsten Ball
Approvers (please review by EOD 2022-04-22): Owen Convey, Olaf Geirsson
Approvals: Olaf Geirsson, TJ Devries, María Craig

Summary

The Code Intel team has been working on a new data format that captures code intelligence in a language-agnostic way. (See RFC 619 WIP: Code Intelligence Platform for more details.) This data format will be at the core of how indexers (written by Sourcegraph or external developers) interact with the Sourcegraph application, and enables advanced use cases such as incremental indexing. We need to have a consistent name and messaging (what + why + how) for this data format, for clearer communication internally and externally. I propose that:

  • We use the name “LSIF Typed” for the format.
  • We frame it as an evolution of LSIF that:
    • Is easier to write indexers for because:
      • It can be understood more easily.
      • Has good tooling that “just works.”
      • Is easier to debug when things go wrong.
    • Makes it easier to run incremental indexing.
    • Will support conversion from vanilla LSIF within some window.
  • We briefly mention LSIF Typed in the upcoming release (“hey, this is a cool thing we have in the works, stay tuned”), and say that we will share more details in the future.
  • We “reserve” some time in Q2 FY23 for providing centralized tooling + on-boarding documentation for LSIF Typed. This should live in its own separate repository. Once the tooling + docs are in good shape, we should publish a blog post describing the framing above in greater detail.

Background

The Code Intel team has been working on a new format for indexer output, with the working title of “LSIF Typed” (sometimes spelt as “LSIF typed”, and sometimes with hyphenation). The upcoming lsif-typescript indexer currently emits data in this format.

LSIF itself seems to be gaining traction outside Sourcegraph. For example, we’ve seen work on Rust, C# and PHP indexing by open source developers who are not Sourcegraph employees. GitHub Next is also working on something related to LSIF based on some Twitter convos, potentially for F#.

Problem

  • We have not settled on a final name for the new index format.
  • We have not settled on whether the new index format will be largely framed as an implementation detail of Sourcegraph, or whether we want to advertise it loudly.
  • We have not settled on how the new index format will be framed in contrast to vanilla LSIF: as an evolution, as a roughly interchangeable (or drop-in) alternative, as a competitor, a mix of these or something else entirely.
  • If open source communities continue investing in vanilla LSIF indexers, the development of indexers will run into the same problems we were running into with LSIF:
    • Retrofitting incremental indexing on top of vanilla LSIF would be difficult, if not impossible.
    • Progress on indexers is likely to be slower, due to:
      • Poor tooling (debugging, testing etc.).
      • The LSIF spec being harder to understand than it needs to be, since it is written using LSP terminology.

Proposal

We use the name “LSIF Typed”

Rationale:

  • Retaining LSIF in the name seems useful, because we will still support some level of interop between the two.
  • “Typed” carries positive connotations (TypeScript vs JavaScript); the new format is more “strongly typed” in some sense (it has a proper schema), compared to vanilla LSIF.
  • We haven’t been able to come up with any alternatives that are substantially better in some way.

We frame it as an evolution of LSIF that…

  • Is easier to write indexers for because:
    • It can be understood more easily.
    • Has good tooling that “just works”.
    • Is easier to debug when things go wrong.
  • Makes it easier to run incremental indexing.
  • Will support conversion from vanilla LSIF within some window
  • Enables faster serialization/deserialization performance and smaller disk usage

Rationale:

  • For framing it as an evolution: Framing it as competition doesn’t really help us (it might mislead people into thinking that adding support for LSIF Typed may be quite time-consuming).
  • We can cite examples of indexers in practice which stalled because the semantics of certain parts of LSIF were too unclear.
  • Tooling + Incremental indexing are “carrots”, which are better than using a “stick” (e.g. “vanilla LSIF uploads will stop working after some time”). Devs love tools that make their life easier.
  • We can start encouraging people to switch over to LSIF Typed gradually.

We briefly mention LSIF Typed in the upcoming release as…

“Hey, this is a cool thing we have in the works, stay tuned”, and say that we will share more details in the future.
Rationale:

  • For not sharing more details right now:
    • We don’t really have a lot of time left in Q1 to polish tools + writing docs. And we have other things to do.
  • For at least sharing that this is something we are working on:
    • We’ve gotten interest from various customers about incremental indexing. This shows them that we are investing in this area. Hopefully, this also grabs some eyeballs from people working on indexers (or potentially interested in working on indexers), and we can potentially get some feedback/have informal convos before we publish a blog post.

We improve tooling & docs in Q2, then publish a blog post

We should “reserve” some time in Q2 FY23 for providing centralized tooling + on-boarding documentation for LSIF Typed.
Rationale:

  • The tooling + on-boarding experience is going to be key to get open source contributors and other companies to be interested in LSIF Typed.
  • Right now, the experience with using LSIF Typed is a little hodge-podge. The lsif-typed CLI tool lives in sourcegraph/sourcegraph, some of the snapshot functionality lives in lsif-java and is reimplemented in lsif-typescript’s test suite.

This should live in its own separate repository.
Rationale:

  • It isn’t really coupled to sourcegraph/sourcegraph or a specific indexer. It would be good to be able to tag releases of the lsif-typed CLI and publish binaries, so that people don’t need to build the binaries themselves. However, that would be out of place to publish in the sourcegraph/sourcegraph repo.
  • This makes it easier to monitor issues etc. (the current issue templates in the sourcegraph repo are a mess IMO)
  • I don’t think we need CLA bot functionality, which is another speed bump for contributions.

Once the tooling + docs are in good shape, we should publish a blog post describing the framing above in greater detail.
Rationale:

  • This is a good opportunity to explain/“sell” LSIF Typed.

Definition of success

  • We have ~rough decisions for:
    • Will the new index format be advertised as part of the next release blog post.
    • Will the new index format have its own separate blog post.
    • What the messaging is for the new index format.
  • The handbook is updated with the outcome of this RFC:
    • The Code Intel strategy page mentions the new index format, and why we’re investing in it. This should capture the “messaging” mentioned above.
    • On the Terminology Guidelines page, we should have the recommended spelling for the protocol name.
  • We should update all spellings across the code and docs for consistency.

Prependix: why we shouldn’t stick with the “LSIF Typed” name

See this proposal: RFC 679 WIP: SCIP to the good part

Appendix: why we are sticking with the “LSIF Typed” name

Olaf Geirsson The “LSIF Typed” name was the most controversial part of this RFC so I’ve added this section after the RFC was approved to explain the motivation for keeping the “LSIF Typed” name.

Naming proposals were roughly split in two camps by whether they include “LSIF” in the name or not.

  • If we want to include LSIF in the name then we are most likely sticking with “LSIF Typed” because that’s what we’ve been using everywhere so far. The main motivation to keep “LSIF” in the name is because we want to avoid confusion for CEs, salespeople and customers who are already accustomed to the LSIF name and interpret it as synonymous with “precise code intel”. We acknowledge that keeping “LSIF” in the name might still be confusing for members of our team and external contributors who will need to learn that “LSIF” and “LSIF Typed” have very different shapes.

    • Sidenote: several people voted for using “TypedLSIF’’ instead of “LSIF Typed” because it doesn’t have a space in the name and might reduce ambiguity of how to reference it from code (lsif-typed vs. lsif_typed vs. lsiftyped). We didn’t put serious effort into resolving the tradeoffs between “LSIF Typed” and “TypedLSIF” because most of the discussion was focused on whether we wanted LSIF in the name at all.

  • If we don’t want to include LSIF in the name then we need to first decide what alternative name we want to use. This Slack thread is a brainstorm session where we evaluated several alternatives. Once we have a concrete alternative name that we like, then somebody who is personally motivated to change the name should write an RFC to document the benefits of changing the name. We should follow the Sourcegraph handbook guidelines when making this decision. Some of the strongest arguments in favor of not using LSIF in the name include:

    • LSIF Typed has a very different shape from LSIF so it will be confusing for people who work with both LSIF Typed and LSIF.
    • We could benefit from a marketing push by advertising LSIF Typed as a Sourcegraph innovation.