repro: .gitignore files keep growing with same entries

[rlleshi]

Bug Report

Description

Whenever I run dvc repro the same entries keep being appended to the corresponding .gitignore files.

Sample content of .gitignore file:

/X.csv
/groups.csv
/y.csv
/X.csv
/groups.csv
/y.csv
/X.csv
/groups.csv
/y.csv

Reproduce

dvc repro -> same entries appended in .gitignore files.

Expected

DVC should only add a line to the .gitignore file once for each file that's being version controlled by DVC

Environment information

❯ dvc doctor
DVC version: 2.57.0 (pip)
-------------------------
Platform: Python 3.10.6 on Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.51.0
        dvc_objects = 0.22.0
        dvc_render = 0.5.2
        dvc_task = 0.2.1
        scmrepo = 1.0.3
Supports:
        azure (adlfs = 2023.4.0, knack = 0.10.1, azure-identity = 1.13.0),
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3)
Config:
        Global: /home/user/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/32582e8b1552224ea25e5d697a41250a

[dberenbaum]

Does it add another duplicate every time you run dvc repro? I'm not able to reproduce it from a quick test:

git init -q repo
cd repo
dvc init -q
dvc stage add -q -n test -o test.txt 'echo test > test.txt'
dvc repro -q
echo test2 > test.txt
dvc repro -q
cat .gitignore

The output is:

/test.txt

[rlleshi]

Yep, it adds another duplicate every time I run dvc repro. Any other logs I can give you?

[dberenbaum]

What do you get running the test script above?

[rlleshi]

@dberenbaum here's a minimal example to reproduce it following your commands above:

git init -q repo
cd repo
mkdir data
touch .gitignore
echo -e 'data/*\n!data/.gitignore\n!data/test.txt' > .gitignore
dvc init -q
dvc stage add -q -n test -o data/test.txt 'echo test > data/test.txt'
dvc repro -q
echo test2 > data/test.txt
dvc repro -q
cat data/.gitignore

will output:

/test.txt
/test.txt
/test.txt

In my case I basically have some files I don't want to version, hence the content of the root .gitignore.

[daavoo]

I am able to reproduce. Haven't looked into details yet but it seems like a similar bug to #7815 , that comes from the dulwich implementation of gitignore parsing

[dberenbaum]

@daavoo Do you think we should mark as duplicate? Should we create a bug in dulwich?

[dberenbaum]

@daavoo will check if the same problem happens in pygit2