Go backend does not delete module download sandboxes

[wfscheper]

Describe the bug
The Go backend does not delete sandboxes created to download 3rd party modules. This means pants quickly fills up my temp space with sandboxes, and I'm forced to manually clean them up.

Pants version
2.14.0

OS
MacOS ARM64 (13.0.1 (22A400))
Linux x86_64 (Fedora 35)

Additional info
This problem is exacerbated (or maybe caused?) by go setting the file permissions in the modcache to 0600 (ie. read-only).

[tdyas]

What platform (OS and architecture) are you on? What specific version of OS?

[wfscheper]

So far I'm unable to reproduce this on a ARM MacBook Pro. Weirdly, on linux pants was constantly re-running the go module downloads every time I did anything (list, test, etc), but that doesn't appear to be happening on my mac. I'll see if I can get some time later today to come up with a reproduction repo I can share (and confirm if I can recreate the problem on a mac).

[wfscheper]

Okay, I was able to reproduce this on a M1 MacBook Pro, with a very simple main.go with only one 3rd party import.

package main

import (
	"fmt"

	"gopkg.in/yaml.v3"
)

func main() {
	out, err := yaml.Marshal(map[string]string{"foo": "bar"})
	if err != nil {
		fmt.Println("error:", err)
	}

	fmt.Println(string(out))
}

$ rm -r .pids/ .pants.d/ ~/.cache/pants/*
$ ./pants list ::
Bootstrapping Pants using /opt/homebrew/bin/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex verified.
Installing pantsbuild.pants==2.14.0 into a virtual environment at $HOME/.cache/pants/setup/bootstrap-Darwin-arm64/2.14.0_py39
New virtual environment successfully created at $HOME/.cache/pants/setup/bootstrap-Darwin-arm64/2.14.0_py39.
19:55:09.64 [INFO] Initializing scheduler...
19:55:09.70 [INFO] Scheduler initialized.
//:0
//:bin0
//:root
//:root#github.com/creack/pty
//:root#github.com/kr/pretty
//:root#github.com/kr/pty
//:root#github.com/kr/text
//:root#github.com/kr/text/cmd/agg
//:root#github.com/kr/text/colwriter
//:root#github.com/kr/text/mc
//:root#gopkg.in/check.v1
//:root#gopkg.in/yaml.v3
//go.mod:root
//go.sum:root
$ ls -d /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-*
/private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-As3riK/ /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-ZUhJhL/
/private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-NJ7z50/ /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-lgGAAg/
/private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-TOL0RY/ /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-pFSr5Y/

[tdyas]

What are the permissions on the directories? (And what files if any remain in the sandboxes?)

Also, I assume you removed all previous sandboxes from /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/? I didn't see a rm -rf for that in your log.

[stuhood]

These directories should be deleted asynchronously by pantsd: if pantsd stays up, they should eventually be removed.

[wfscheper]

pantsd is still live, but the sandboxes aren't cleaned up:

$ ps aux | grep pants
user           60557   0.9  0.2 971639600  26832   ??  S    Sun07PM   8:12.73 pantsd [/Users/user/git/pants-17698-example]
user           73614   0.0  0.0 408636096   1440 s001  S+    2:14PM   0:00.00 grep --color=auto pants
$ ls -dl /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-*
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-As3riK/
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-NJ7z50/
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-TOL0RY/
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-ZUhJhL/
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-lgGAAg/
drwxr-xr-x  3 user  staff  96 Dec 11 19:55 /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-pFSr5Y/

Looks like pants cleaned out everything but the sandboxed modcache

$ tree /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-As3riK/
/private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-As3riK/
└── gopath
    └── pkg
        └── mod
            └── github.com
                └── kr
                    └── text@v0.2.0
                        ├── License
                        ├── Readme
                        ├── cmd
                        │   └── agg
                        │       ├── doc.go
                        │       ├── main.go
                        │       ├── num.go
                        │       └── string.go
                        ├── colwriter
                        │   ├── Readme
                        │   ├── column.go
                        │   └── column_test.go
                        ├── doc.go
                        ├── go.mod
                        ├── indent.go
                        ├── indent_test.go
                        ├── mc
                        │   ├── Readme
                        │   └── mc.go
                        ├── wrap.go
                        └── wrap_test.go
$ ls -l /private/var/folders/rs/yp1rkctx697gyvsnhkskcwtm0000gn/T/pants-sandbox-As3riK/gopath/pkg/mod/github.com/kr/text@v0.2.0/
total 64
-r--r--r--  1 user  staff  1052 Dec 11 19:55 License
-r--r--r--  1 user  staff   127 Dec 11 19:55 Readme
dr-xr-xr-x  3 user  staff    96 Dec 11 19:55 cmd/
dr-xr-xr-x  5 user  staff   160 Dec 11 19:55 colwriter/
-r--r--r--  1 user  staff   100 Dec 11 19:55 doc.go
-r--r--r--  1 user  staff    68 Dec 11 19:55 go.mod
-r--r--r--  1 user  staff  1537 Dec 11 19:55 indent.go
-r--r--r--  1 user  staff  1940 Dec 11 19:55 indent_test.go
dr-xr-xr-x  4 user  staff   128 Dec 11 19:55 mc/
-r--r--r--  1 user  staff  2225 Dec 11 19:55 wrap.go
-r--r--r--  1 user  staff  1177 Dec 11 19:55 wrap_test.go

[wfscheper]

Circling back to this now that the holidays are behind us. Is there any more info I can provide?

[wfscheper]

So I can confirm this is still happening with pants 2.15.0. I'm interested in adopting pants more widely at work, but this problem effectively prevents me from trying it on several of our bigger go projects. The volume of dependencies downloaded completely fills my development environment's tmpfs.

Would it be feasible to run go clean -modcache after pants has captured the mod cache contents?

[tdyas]

As it has been described, this issue still seems like an issue with deletion of the temporary directory and not with go per se other than the fact that go triggers the issue because of the read-only permissions on the files in the module cache.

Are you able to manually rm -rf the directories in question? If so, Pants should be able to do the same thing. And if Pants cannot do that, I would question why Pants cannot do that. The Go rules in Pants do not handle deletion of sandbox directories; the process_execution crate in the Pants Rust-based core does that.

[wfscheper]

Go creates the modcache with read-only permissions to keep you from easily modifying the contents (for instance by editing files in an IDE that lets you jump to a definition). However, this also means that you cannot simply delete the contents of the modcache either. If I first run chmod -r u+w /path/to/sandbox/* then I can successfully run rm -r /path/to/sandbox. The command go clean -modcache effectively does this for you, by first updating the permissions on the files and then removing them.

[wfscheper]

So I suppose there are two options. In the general case, process_execution could ensure that all files in the sandbox have the user write bit set as part of the cleanup process. Or specifically for the go backend, it could run go clean -modcache to clean up the sandboxed modcache itself.