Imported plans copy the full source tree per child plan; discover `prune` does not apply to the worktree

[thrix-vault]

Summary

discover: prune: true prunes the discover test directory (discover/<phase>/tests), but it does not touch the plan worktree (<plan_workdir>/tree). The worktree is always a full rsync of the metadata tree root. When a plan imports and fans out into many child plans, each child materializes its own full-tree worktree, so the whole source repository gets copied once per child plan on the runner.

Impact

Seen in a Testing Farm run where an imported openssl plan expanded into ~40 child plans. Each child had a 17M tree/ worktree holding the entire openssl test repo (every test dir plus shared helpers), even though each plan runs a single test. That is ~680M of near-identical copies for one run, and prune: true on those plans does not reduce it.

Reproducer

D=$(mktemp -d); cd "$D"

# "remote" repo: 3 plans, each runs ONE small test, all with prune: true.
# heavy/blob.bin (5MB) belongs to no test.
mkdir remote && cd remote && git init -q && mkdir -p .fmf && echo 1 > .fmf/version
mkdir -p tests/t1 tests/t2 tests/t3 heavy Plans
printf '/t1:\n    test: ./r.sh\n    path: /tests/t1\n/t2:\n    test: ./r.sh\n    path: /tests/t2\n/t3:\n    test: ./r.sh\n    path: /tests/t3\n' > tests/main.fmf
for t in t1 t2 t3; do echo "echo $t" > tests/$t/r.sh; done
head -c 5000000 /dev/zero > heavy/blob.bin
for n in 1 2 3; do
  printf 'discover:\n    how: fmf\n    test:\n      - /tests/t%s\n    prune: true\nprovision:\n    how: local\nexecute:\n    how: tmt\n' "$n" > Plans/tier$n.fmf
done
git add -A && git commit -qm init
R=$(pwd); cd "$D"

# local repo: import all the plans (fan-out)
mkdir local && cd local && git init -q && mkdir -p .fmf && echo 1 > .fmf/version
printf 'plan:\n    import:\n        url: file://%s\n        name: /Plans\n        scope: all-plans\n        importing: become-parent\n' "$R" > importer.fmf
git add -A && git commit -qm init

run=$(mktemp -d)
tmt run --scratch -i "$run" discover

echo "=== worktrees (tree/) ==="
find "$run" -name tree -type d | while read t; do printf '%s\t%s\n' "$(du -sh "$t"|cut -f1)" "${t#$run/}"; done
echo "=== one worktree contents ==="
t=$(find "$run" -name tree -type d | head -1); ls -A "$t"; ls "$t/tests"
echo "=== matching pruned discover tests ==="
d=$(find "$run" -path '*discover*tests' -type d | head -1); ls "$d/tests"

Actual output

=== worktrees (tree/) ===
4.9M	importer/Plans/tier3/tree
4.9M	importer/Plans/tier2/tree
4.9M	importer/Plans/tier1/tree
=== one worktree contents ===
.fmf  heavy  Plans  tests
main.fmf  t1  t2  t3        # all three tests + the 5MB blob, in every worktree
=== matching pruned discover tests ===
main.fmf  t3               # discover IS pruned to the single selected test

Each imported child plan gets a full 4.9M copy of the whole repo (heavy/blob.bin, all of tests/t1 t2 t3), while its discover/.../tests is correctly pruned to the one selected test.

Expected

With prune: true, the on-runner footprint of an imported/fan-out plan should not scale with the full repository size times the number of child plans. Either the worktree should honor pruning, or sibling child plans sharing the same source should not each copy the whole tree.

Notes on the mechanism

• The worktree is created in Plan._initialize_worktree() (tmt/base/plan.py), which rsyncs the whole tree root into <plan_workdir>/tree. It returns early only for is_remote_plan_reference, which is the importing plan, not the materialized child plans, so every child gets a full worktree.
• prune/prune_tree() (tmt/steps/discover/__init__.py) operates on the discover step's test_dir only. It never touches the worktree, so prune: true cannot shrink the copies above.
• prune_tree() itself works correctly for imported plans; the discover tree is pruned as expected. The bloat is the worktree.

Version

Reproduced with tmt 1.76.0.

Generated-by: Claude Code

[LecrisUT]

Afaict this is default-3 case in #4559 (comment).

I don't think there is a good way to handle this one because the plan work-tree could be used for prepare. The pruning of the work tree would have to be defined and done somewhere globally for the whole plan/step, not the individual phase.

[thrix]

The real plan which blowed up to multi-million inodes on the artifact storage:

https://gitlab.com/redhat/rhel/tests/fips-provider-next/-/blob/master/plans/external.fmf?ref_type=heads

[thrix]

From triage meeting:

• cleanup after run should be done, should be easy to fix
• seems related to the remote plans importing
• @thrix mention the concrete leftovers from that fips plan