[Bug]: Mutagen sync leaves empty directories in .garden/build causing desync (e.g. with Prisma)

[nullxx]

Bug

Current behavior

When using Garden with the default build system (Mutagen), file deletions are propagated to .garden/build, but empty directories left behind after file moves or deletions are not removed. This results in outdated, empty folders being left in .garden/build, which can cause problems with tools like Prisma that rely on accurate directory structures (e.g., prisma/migrations).

Expected behavior

Empty directories in .garden/build should be removed when files inside them are moved or deleted, reflecting the actual state of the source directory.

Reproducible example

1. Create a folder with a file, e.g., migrations/001-init/file.sql
2. Move file.sql to another folder (e.g., migrations/002-changed/)
3. Run garden deploy
4. Observe that .garden/build/migrations/001-init still exists as an empty directory

Workaround

You can avoid the issue in three different ways:

1. Delete .garden/ manually after cleanup or after moving files:

   rm -rf .garden

2. Use rsync instead of Mutagen for syncing:

   GARDEN_K8S_BUILD_SYNC_MODE=rsync GARDEN_LEGACY_BUILD_STAGE=true garden deploy

3. Patch the Garden source code to use two-way-resolved sync mode instead of one-way-replica at:
   

   garden/core/src/plugins/kubernetes/container/build/common.ts

   Line 143 in ee31fdb

   mode: "one-way-replica",

Suggested solution(s)

Change the default sync mode in the Garden source code to use two-way-resolved instead of one-way-replica, or introduce a mechanism to clean up empty directories left in .garden/build after sync.

Additional context

This behavior breaks workflows that rely on a strict directory structure such as Prisma's migration system. These leftover empty folders can cause unexpected issues.

Your environment

• OS: Fedora Linux 42 (Workstation Edition)
• How I'm running Kubernetes: EKS

garden version
garden version: 0.13.61

[thsig]

Hi @nullxx, apologies for the late reply here! And thanks for taking a look at this : )

There are few misconceptions in this issue:

• The .garden/build directory is designed to contain clean copies of the source directories of the Builds managed by Garden. Any directory removals that happen later in a build process (e.g. in a local or remote Docker build) shouldn't affect the contents of the build directory.
• In your example, if you were to delete the empty migrations/001-init directory after moving file.sql to another directory, a subsequent garden build command run would result in a build directory without the migrations/001-init. However, we don't auto-delete empty directories in the build dir—this is by design, since most of the time users would expect the directory structure in the build directory to reflect the source directory (with any excludes applied). If you want the directory to disappear from the build directory, the recommended approach would be to delete it from your source directory and run garden build again.
• Using a different build sync method (rsync or mutagen) would not have an effect on whether emptying a source directory and rebuilding results in that directory not showing up in the build directory—build syncing refers to the method used to sync files from the build directory into a remote builder (e.g. when using in-cluster building).
  Could you elaborate a bit more on the use-case you had in mind? Maybe there are other patterns or workarounds that would be more suitable to address your needs here.

Thanks again!

[nullxx]

Hello @thsig

• The .garden/build directory is designed to contain clean copies of the source directories of the Builds managed by Garden. Any directory removals that happen later in a build process (e.g. in a local or remote Docker build) shouldn't affect the contents of the build directory.

• In your example, if you were to delete the empty migrations/001-init directory after moving file.sql to another directory, a subsequent garden build command run would result in a build directory without the migrations/001-init. However, we don't auto-delete empty directories in the build dir—this is by design, since most of the time users would expect the directory structure in the build directory to reflect the source directory (with any excludes applied). If you want the directory to disappear from the build directory, the recommended approach would be to delete it from your source directory and run garden build again.

Please test this approach because I can confirm it is not correct.

1. The startpoint

.
└── migrations
    └── 1
        └── file.sql

2. Run garden build. It appears

.
├── .garden
│   └── build
│       ├── .DS_Store
│       └── migrations
│           ├── .DS_Store
│           └── 1
│               └── file.sql
└── migrations
    └── 1
        └── file.sql

3. Run mv migrations/1 migrations/2. It appears

.
├── .garden
│   └── build
│       └── migrations
│           └── 1
│               └── file.sql
└── migrations
    └── 2
        └── file.sql

4. Run garden build. It appears

.
├── .garden
│   └── build
│       └── migrations
│           └── 1 # << this folder does no longer exist inside source, but is not removed by garden
│           └── 2
│               └── file.sql
└── migrations
    └── 2
        └── file.sql

This is caused by the sync method used between the src directory and .garden/build by mutagen or rsync. That is why changing it fixes this issue.

[eysi09]

Sorry for the late reply @nullxx! It just so happens that @thsig went on paternity leave shortly after his reply so this one sort of slipped between cracks.

You're absolutely right that this is a bug but it's not related to Mutagen per se but rather the build staging mechanism.

Tagging @edvald who's our in-house expert on build staging and will take a look.