How to delete commits (and git branches) completely ?

[simonmichael]

Thank you for jj, which is awesome!

My question:

I'm cleaning up a repo (because jj log showed me that I have hundreds of stale old branches in it).

I have found jj better than git for finding old branches to delete.
Eg: jj log --no-graph -r 'bookmarks() ~ person(simon)'

And I've found that jj bookmark delete BOOKMARK also deletes the git branch.

But not the commits themselves, I think because they are still referenced by git refs created by jj like refs/jj/keep/HASH.
I'd like to cleanly delete those, so git (or jj) can garbage collect these commits.

I also tried jj abandoning them, which didn't help. Then I saw

To prevent GC from deleting commits that are still reachable from the operation log, the GitBackend stores a ref for each commit in the operation log in the refs/jj/keep/ namespace.

So do I need to jj op abandon all operations that mention them, as well ? It seems hard.

[martinvonz]

So do I need to jj op abandon all operations that mention them, as well ? It seems hard.

Yes, that's what you need to do. If you just deleted the branches, then you'll have to abandon most operations in your repo. jj op abandon ..@- will delete all operations before the current operation. Then follow that by jj util gc to remove the unused refs/jj/keep/ refs and run git gc (i.e. jj util gc does that for you).

[simonmichael]

Thanks @martinvonz! Losing all operation history of course sounds bad. Is there any way to see which operations reference those commits ? I didn't see an obvious one.

[simonmichael]

Found them this way!

for o in `jj op log --no-graph -T 'id.short()++"\n"'`; do echo $o:; jj op show $o | grep HASH; done

I had a hunch it would be the first "import git refs" operation, and it was. But also today's attempts to forget/delete the bookmark and then abandon the commit.

[simonmichael]

I tried abandoning all the ops listed by the above, which seemd successful; afterward, none were listed. On the other hand, the deleted git branches and commits were back. I deleted them again with git. I gc'd on both sides. I ended up abandoning all ops as Martin showed above. Perhaps I made mistakes, but I could not eradicate these commits. If someone else comes up with a successful procedure, I'd be glad to know it.

[yuja]

You'll have to use jj util gc --expire=now to delete jj/keep refs immediately.

Also note all predecessor commits are preserved. For example, old commits displayed in jj evolog -r <REVISIONS_SHOULD_BE_KEPT> won't be deleted even if old operations got abandoned.

[simonmichael]

Thank you, --expire=now (or --prune=now on the git side) was another important clue. And noted about evolog.

For the record, I got these stale branches cleaned out in this way:

1. Take advantage of jj to save a list of the branches to be deleted. Eg: jj log --no-graph -r 'bookmarks() ~ person(simon)' -T 'bookmarks++"\n"'
2. Give names to any unnamed heads created with jj: jj bookmark create ...
3. Discard jj state, in the simplest way: rm -rf .jj
4. Delete each branch with git: for b in ...; do git branch -D $b; done
5. Also delete the "keep" references created by jj: gsed -i '/\/keep\//d' .git/info/refs; gsed -i '/\/keep\//d' .git/packed-refs
6. Check one or two hashes to see if they're still referenced: rg SOMEHASH .git
7. If still mentioned in .git/logs/HEAD, that means the git reflog still references it. As with jj's operations log, the simplest way to fix that is discard the entire reflog: git reflog expire --expire=now --all.
   I'm not sure if a more surgical removal is feasible.
8. Now all references should be gone; discard the commits with: git gc --prune=now
9. And verify they're gone: git show SOMEHASH (should fail)
10. And finally bring back jj (starting fresh): jj git init --colocate

Bottom line, neither jj nor git give up data easily, even when you want them to !

And as #git pointed out, the easier way when you really want to clean up is:
clone the repo and transfer the branches you really want to keep.

[lgarron]

I ended up with bloated data from a branch switch one too many times, so I tried to give the full process a go. After making sure the visible branch heads corresponded to the ones I wanted to keep, I did the following:

1. Make a backup copy of the entire repo folder so I can start over and compare outcomes as needed.
2. Convert the repo into a colocated repo using the steps from: https://github.com/jj-vcs/jj/blob/main/docs/git-compatibility.md#converting-a-repo-into-a-co-located-repo
3. Create a gigantic merge commit holding on to every visible head (other than bookmarks, which are already tracked):

# fish
set LEAVES (jj log --no-graph -r 'visible_heads() ~ bookmarks() ~ remote_bookmarks()' -T "change_id ++ \"\n\"")
jj new $LEAVES # This will almost certainly be a huge merge conflict. That's fine for our purposes.
jj bookmark set --revision @ BACKUP_MEGA_MERGE

4. Run the steps from @simonmichael above.
5. Abandon the the mega merge commit.
6. Open both the repo and its backup in gg to perform a quick visual inspection that the graph matches.
7. Perform repo initialization bookkeeping:

• Re-track remote branches.
• Re-run any per-repo jj git config commands.

Here's the core of what I ran on the commandline:

# Gitify: https://github.com/jj-vcs/jj/blob/main/docs/git-compatibility.md#converting-a-repo-into-a-co-located-repo
# Ignore the .jj directory in Git
echo '/*' > .jj/.gitignore
# Move the Git repo
mv .jj/repo/store/git .git
# Tell jj where to find it
echo -n '../../../.git' > .jj/repo/store/git_target
# Make the Git repository non-bare and set HEAD
git config --unset core.bare
# Convince jj to update .git/HEAD to point to the working-copy commit's parent
jj new && jj undo
set LEAVES (jj log --no-graph -r 'visible_heads() ~ bookmarks() ~ remote_bookmarks()' -T "change_id ++ "\n"")

echo $LEAVES

jj new $LEAVES # This will almost certainly be a huge merge conflict. That's fine for our purposes.

jj bookmark set --revision @ BACKUP_MEGA_MERGE
rm -rf .jj

gsed -i '//keep//d' .git/info/refs; gsed -i '//keep//d' .git/packed-refs

git reflog expire --expire=now --all

git gc --prune=now

jj git init --colocate .
jj abandon BACKUP_MEGA_MERGE
Run any jj config steps here (e.g. set the author if you don't have a default)

Some notes:

• I'd still like to figure out the .git dir for this repo, as this doc states that "In co-located repos with a very large number of branches or other refs, jj commands can get noticeably slower because of the automatic jj git import executed on each command.". Presumably these steps can be run in reverse?
• It's important to be careful that the gigantic merge commit and the working copy don't result in tracking files you want to abandon. There's probably a way to do this by checking out a commit that ignores the whole repo.

[lgarron]

Oh my, I transferred focus to the gg app while a repo was briefly without a .gitignore and this turned out to be an expensive mistake. I now have 300MB instead of 3MB in my .jj. jj abandon and jj op abandon ..@- && jj util gc --expire=now didn't remove much of it.

I've now performed too many operations to feel comfortable using jj undo for this, and I have way too many branches to feel comfortable doing this by clearing .jj. Is there any other workaround available now?

I'm okay losing an arbitrary amount of the op log in this case, if it has to be.

[lgarron]

This happened to me again, but in a different way. We're using cargo-run-bin, which places binaries in a .bin folder that is ignored¹.

When I checked out an old commit where .bin was not ignored yet, jj immediately tracked the binaries. Now I'm once again stuck in a situation where the .jj is an order of magnitude larger than it should be.

At the moment the most reliable way I have to solve this is:

• Push all branches and tags and hope I haven't forgotten anything.
• Clone the repo from scratch and re-create the local bookmarks.

This is not a great flow, and it's only a matter of time until I lose something that I actually wanted to keep.
(It also requires pushing everything publicly, which may not be desirable for draft branches. I'm sure I can figure out workarounds like pulling between local checkouts, but that seems even more prone to mistakes.)

I understand that jj can't necessarily implement everything easily or quickly, but this is a really bad UX footgun and I think it should be considered table-stakes to have a solution that works directly inside a (non-colocated) .jj checkout.

Footnotes

1. It's ignored at the repo root .gitignore because cargo-run-bin does not place a .gitignore inside .bin itself. I can obviously fix this for a given project, but stuff like this comes up in various ways and it's impossible to account for all of it ahead of time. ↩

[martinvonz]

#5596 is the tracking bug