Previously ignored files should not become tracked on checkout

[martinvonz]

Description

When updating from a commit where some files are ignored to a commit where they are not ignored (typically an older version), the ignored files will be picked up by subsequent snapshotting. That's rarely what one wants, and it's especially problematic if the ignored files are many (e.g. target/ in Rust projects).

Steps to Reproduce the Problem

cd $(mktemp -d)
jj git init
jj new
echo ignored > .gitignore
touch ignored
jj edit @-
jj status

Actual Behavior

The ignored file is tracked.

Expected Behavior

The ignored file does not get tracked. Perhaps jj status reports that it's in some kind of intermediate state.

We could implement it by having the WorkingCopy keep track of this type of file. Files would enter this state when updating the working copy and the ignored paths change. They would exit it when they're either either deleted or properly ignored again.

This idea was originally suggested by @ony.

Specifications

• Platform: All
• Version: 0.25.0 (and probably all before it)

[yuja]

Duplicates of #3204?

Files would enter this state when updating the working copy and the ignored paths change. They would exit it when they're either either deleted or properly ignored again.

fwiw, git submodule directories are processed in a similar manner. They're marked as submodule (= ignored) internally.

[martinvonz]

Duplicates of #3204?

Ah, that's the issue I was looking for! I spent at least 5 minutes searching for it (e.g. "tracked ignored" doesn't work). I guess I'll mark that one as duplicate instead so it will be easier to find the issue (i.e. this one) next time.

[bts]

In case it's helpful, I ran into this when I tried to use the git pattern of adding a (.env) file in one commit and then adding it to .gitignore in a subsequent commit. (By the way -- thanks for jj! It's great.)

[niklaswimmer]

I just pushed a secret to a remote due to this, it was only a test one though so no biggies 😄

Basically, I was working on something that required me to temporarily put a secret in a file for test purposes and I took special care to .gitignore it, knowing that jj would otherwise start to track it automatically. A few days later (today), someone asked me about some code I had written in some sibling commit so I jj edited it and pushed it up to a temporary branch. I did not look at what I would push, because I just wanted to quickly share code with someone else (actually thought "because it is jj I do not even have to add any changes, I can just quickly do them and push them..."). Because that sibling commit did not include the changes to .gitignore and the untracked secret was moved over when I ran jj edit, I ended up pushing it.

I guess what I am saying is that whatever comes out of this issue, should have either warned me when I ran jj edit that some files will no longer be ignored or should have ignored them automatically again. Notably, it should not just delete that file, because I would like to still have it around when I work on the original change again. I think this is what the issue description is suggesting anyway, I am just repeating it here explicitly 👍

[evmar]

I was thinking about this issue and why it is so vexing for me, and I thought of this principle:

jj edit x, followed by jj edit y to go back to where I was before, should have no effect on the contents of changes x and y

This problem underlying this issue is about that principle being violated. The ignored-ness of the file is one way to look at it (since I think that is the only way for this to break?) but another way of looking at it is that immediately after running jj edit y the resulting on-disk state (as in what would be captured in the next jj snapshot) does not match the contents of y that I just requested.

[jceb]

I started using JJ today and this issue was the first one that I ran into with existing git repositories. In some I had legacy files that I kept around but hadn't put on the ignore list. JJ picked them up immediately and then I struggled with defining ignore lists on the main branch. The fix on the main branch led to this issue when I switched to another branch that didn't have the ignore list yet.

[jceb]

One work around that comes to my mind is git's internal ignore list that is kept inside the .git folder and is not tracked by version control. If there was something similar in JJ, I could work around the issue by manually copying the current ignore list into this special list. Definitely not perfect, however it feels like the issue is caused by JJs architecture and therefore difficult to solve.

[hotsphink]

jj respects .git/info/exclude already. (Note that adding stuff there won't untrack it if jj has already picked it up; you'll need to jj file untrack it or abandon @.)

[jceb]

Ah, this is good to know! Thank you!

[johanneshardt]

For accidentally added files, is there a way to comb through the oplog and filter them out? I guess it's not possible since they get added as a side effect of other commands right? I ran into this when a vscode lsp extension generated ~1000 build files, which now makes commands such as jj operation log --patch difficult to use. It would be nice if there was a way to solve this without needing to delete + clone the repo again!

[yuja]

~1000 build files, which now makes commands such as jj operation log --patch difficult to use. It would be nice if there was a way to solve this without needing to delete + clone the repo again!

If you're okay to nuke these operations, you can use jj op abandon <first-bad-op>-..<last-bad-op>.

[johanneshardt]

Oh, I totally had the wrong mental model of jj op abandon 😅 I thought changing it would actually modify history, like unapplying the change made at that step, but its just erasing the history entries, that makes sense! I still had the problem of operations being slow and .jj growing quite large. I tried doing jj op abandon ..@- and jj util gc --expire now, which did not really help. I guess the objects still exist in the evolog of the change where they were accidentally committed, so they don't get collected (like the documentation mentions)? For what it's worth i really like the auto-tracking behavior in general, its just unfortunate how it interacts with .gitignores currently.