Tracking Issue for `git2` deprecation

[bsdinis]

There has been a latent sentiment amongst contributors that jj has outgrown its need to depend on libgit2, and as such we are moving to replace it and eventually deprecate it. This issue tracks the reasoning for this move, as well as the steps needed to make it happen and a deprecation plan.

Steps

• Implement shelling out to a git subprocess for push/fetch. git: spawn a separate git process for network operations #5228
• Migrate remote operations with gitoxide.
• Migrate git tests to gitoxide
  • cli/tests: move a lot of the cli tests to gitoxide #5575  #5639
  • lib/tests: port lib tests to gitoxide #5757
  • cli/tests: move bookmark_command test to gitoxide #5754
  • cli/tests: migrate more tests from git2 to gitoxide #5655
  • lib/tests: move remaining git test to gitoxide #5813
  • cli/tests: move test_git_colocated to gitoxide #5803
  • lib: move git_backend tests to gitoxide #5770
  • cli/tests: move remaining tests to gitoxide #5846
• Switch the default push/fetch code path to use the git subprocess git: enable subprocessing by default #5652
• Barricade the git2 code behind a (enabled by default) feature flag git: make git2 support optional #5956
• Enable a deprecation warning for git2 git: add deprecation warnings for git.subprocess #6047
• Remove git2 git: remove git.subprocess setting #6048

Outstanding bugs

We list here bugs that have arisen with git subprocessing. Although sometimes git2 might not be the solution (some of them were use-cases that were wholly blocked by git2), it is good to have them registered and try to figure them out before we fully deprecate git2.

• authentication problems
  • bug: interactive login exits early if there is anything in the output with git.subprocess = true #5924
  • BUG: Missing output messages from the git subprocess #5760
  • ssh_dispatch_run_fatal when git.subprocess=true #5968
• git-push ref failures
  • jj git push thinks the ref moved on remote unexpectedly, but jj git fetch does nothing #5978

Rationale

Issues with git2

libgit2 (with git2 being its Rust bindings) is a project to implement git as a library so that external applications can interact programmatically with git repos without having to shell out to git and parse its output.

This effort is commendable and git2 has been crucial in making jj git compatible from the get go. I'd like to emphasize that this is by no means a bash on libgit2. However, we have found some pain points with its usage:

• Missing support for SSH. This regularly comes up when users try to push to Github, with over 20 issues opened about it for jj plus multiple reports a week.
• Packaging problems. libgit2 only supports one version at a time, meaning packagers have a problematic window of time while jj has not updated git2 to its latest version.
• Performance. For the main (current) usage of libgit2, which is interacting with git remotes, libgit2 has been reported to be slow.

Change to what

jj actually uses another git library to interact with repositories: gitoxide. This means that, as it stands, we have two libraries for interacting with git, which is arguably one too many.
As gitoxide has become more feature complete, relying on git2 becomes less desirable.
For the remaining cases (which turn out to be simply the push/fetch operations), the consensus is to rely on shelling out to a git subprocess.

[yuja]

Barricade the git2 code behind a (enabled by default) feature flag

I think this is optional, and can be omitted unless required by package maintainers who want to drop libgit2 earlier.

[bsdinis]

I think this is optional, and can be omitted unless required by package maintainers who want to drop libgit2 earlier.

Agree that it's not super important. Besides the use case you've described, it'd make adding more git2 code harder, which would be desirable at that stage of the pipeline

[emilazy]

I have remote management fully ported locally and will send a PR tomorrow.

I think supporting/testing a feature flag would be nice because it would keep us honest about the ability to drop it.

[qaqland]

I’m curious if will use this library in the future:

https://lore.kernel.org/git/cover.1723054623.git.steadmon@google.com/

[PhilipMetzger]

I’m curious if will use this library in the future:

https://lore.kernel.org/git/cover.1723054623.git.steadmon@google.com/

This was already discussed in the Discord, and it will probably be done whenever a version of it will be available.

[hotsphink]

This doesn't affect anything here, but I wanted to add in another issue with fetching via libgit2: at Mozilla, we're currently using a git plugin to communicate with a mercurial remote. The plugin modifies the repository URL syntax in a way that libgit2 (or jj using libgit2?) doesn't support. (Example: hg::https://hg.mozilla.org/mozilla-central). As a result, we've all been using manual git fetch instead of jj git fetch. This is no longer necessary with git.subprocess = true, and besides is only a temporary measure until we migrate the repository of record from mercurial -> git, but it's still a pain point where the git shell command can be modified to support things that libgit2 does not.

(That said, I assume gitoxide or cgit-rs would have the same issue? As long as the subprocess option is available, it doesn't actually matter for us.)

[emilazy]

git.subprocess = true is due to be default in the next stable release now that #5652 has been merged, so hopefully that will eliminate that particular pain point. (Though of course that’s no guarantee it’ll ship if there are issues.)

gix doesn’t implement pushing, so being able to use it for remote operations is not on any imminent horizon. I assume that any future use of cgit-rs should be able to be 1:1 compatible with what git(1) does, but I imagine that’s a way off as well. I think we’ll be using a git(1) subprocess for a fairly long time. (Probably longer than Mozilla will be using Mercurial, from my understanding, at least!)

[KyleFin]

I've been loving finally using jj this week with GitLab and HTTP since #469 was resolved.

I just wanted to add a reminder here that https://github.com/jj-vcs/jj/blob/main/docs/git-compatibility.md#supported-features could probably use a refresh when some of the dust settles (especially the first 2 bullets: Configuration and Authentication)

Thanks for all the great work here!