[Bug]: The /tmp filesystem is filling up with Trivy directories

[zachelnet]

zot version

v2.1.11

Describe the bug

Hello,

Since updating from v2.1.7 to v2.1.11, the file system under /tmp/ has been filling up with 'trivy-xxxxxx'.

To reproduce

Updating from v2.1.7 to v2.1.11

1. Configuration

{
    "storage": {
        "rootDirectory": "/srv/zot",
        "GC": true,
        "gcDelay": "1h",
        "gcInterval": "168h",
        "dedupe": true,
        "retention": {
            "dryRun": false,
            "delay": "24h",
            "policies": [{
                "repositories": ["**"],
                "deleteReferrers": true,
                "deleteUntagged": true,
                "keepTags": [{
                    "pulledWithin": "720h",
                    "pushedWithin": "720h"
                }]
            }]
        }
    },
    "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "externalUrl": "https://oci.dev.local",
        "compat": ["docker2s2"],
        "tls": {
            "cert": "/etc/zot/fullchain.pem",
            "key": "/etc/zot/privkey.pem"
        },
        "auth": {
            "openid": {
                "providers": {
                    "oidc": {
                        "credentialsFile": "/etc/zot/oidc-credentials.json",
                        "issuer": "https://keycloak.local/realms/sub",
                        "name": "SSO",
                        "keypath": "",
                        "scopes": ["openid"]
                    }
                }
            }
        },
        "accessControl": {
            "metrics":{
                "users": ["admin@local"]
            },
            "repositories": {
                "**": {
                    "policies": [
                        {
                            "groups": ["S_admin"],
                            "actions": ["read", "create", "update", "delete"]
                        }
                    ],
                    "anonymousPolicy": ["read"],
                    "defaultPolicy": ["create", "read"]
                }
            }
        }
    },
    "log": {
        "level": "warn"
    },
    "extensions": {
       "metrics": {
            "enable": true,
            "prometheus": {
                "path": "/metrics"
            }
       },
       "scrub": {
            "enable": true,
            "interval": "24h"
        },
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "48h"
            }
        },
        "ui": {
            "enable": true
        },
        "sync": {
            "credentialsFile": "/etc/zot/sync-auth-filepath.json",
            "registries": [
                {
                    "urls": ["https://ghcr.io"],
                    "onDemand": true
                },
                {
                    "urls": ["https://docker.gitea.com"],
                    "onDemand": true
                }
            ]
        }
    }
}

2. Client tool used
3. Seen error

:~]# du -sh /tmp 
59G     /tmp
:~]# ll /tmp/
total 2780
drwxrwxrwt 17 root root     4096 Nov 28 08:19 ./
drwxr-xr-x 21 root root     4096 Nov 24 22:48 ../
....
drwxr-xr-x  3 zot  zot      4096 Nov 27 15:25 trivy-455860/
drwxr-xr-x  2 zot  zot   2777088 Nov 28 08:09 trivy-456294/
....

Expected behavior

No response

Screenshots

No response

Additional context

No response

[andaaron]

Hi @zachelnet, wasn't this also the case before the upgrade?

Note those folders are created by the Trivy library we import for CVE scanning, we don't manage them directly. Do they contain only files such as /tmp/trivy-3359096/analyzer-file-286123778?

[zachelnet]

Hi @zachelnet, wasn't this also the case before the upgrade?

Note those folders are created by the Trivy library we import for CVE scanning, we don't manage them directly. Do they contain only files such as /tmp/trivy-3359096/analyzer-file-286123778?

For the upgrade, the folder wasn't on /tmp

[zachelnet]

@andaaron on v2.1.7 is the trivy database on rootDirectory/_trivy/

~]# ll /srv/zot/_trivy/
total 20
drwxr-xr-x  5 zot zot 4096 Nov 28 21:47 ./
drwxr-xr-x 75 zot zot 4096 Nov 28 21:46 ../
drwxr-xr-x  2 zot zot 4096 Nov 28 21:47 db/
drwx------  2 zot zot 4096 Aug 29 14:34 fanal/
drwxr-xr-x  2 zot zot 4096 Nov 27 15:31 java-db/
~]# ll /srv/zot/_trivy/db/
total 863492
drwxr-xr-x 2 zot zot      4096 Nov 28 21:47 ./
drwxr-xr-x 5 zot zot      4096 Nov 28 21:47 ../
-rw-r--r-- 1 zot zot       152 Nov 28 21:47 metadata.json
-rw-r--r-- 1 zot zot 884199424 Nov 28 19:23 trivy.db

after v2.1.11 is changing on /tmp/trivy-xxxxxx.

It's possible to change this?

[andaaron]

Nothing has changed AFAIK. Trivy has been downloading the content under /tmp/ and moving it to /srv/zot/_trivy for as far as I can remember. I mentioned this in one of my comments: project-zot/project-zot.github.io#138 (comment), which will need to get into the documentation.

The analyzer files may be new, I don't remember seeing them before.
I will take a look to see if they added a new customization to fix this.
If not, maybe setting a different path for the TMPDIR environment variable for the zot process may help.

[andaaron]

aquasecurity/trivy#9457 (reply in thread)

[andaaron]

It seems like they made this change aquasecurity/trivy@8f5b560
in Trivy 0.65.0 which we picked in #3292.

Seems like they have new apis for creating and cleaning up the temporary folders, and the Cleanup is not explicitly called from our code, causing the "temporary" files to keep piling up under the default trivy directory (I see they have a 77 MB tarball with the database under /tmp/trivy-84376/oci-download-2700474122/db.tar.gz). Calling that Cleanup should be the correct fix in our case.

[andaaron]

Hi @zachelnet, I have merged #3618 to main. From now on, the temporary files should be deleted as soon as they are no longer needed. Would you be willing to try it out?

The already existing /tmp/trivy should be cleaned up manually. They are related to previous zot executions, and are no longer needed.
All the important data we need to keep is already under /srv/zot/_trivy/, this has been the case in both v2.1.7 to v2.1.11.

[rchincha]

@zachelnet pls verify this PR (merged) is working for your use case.
#3618

If so, pls close this issue.

[zachelnet]

@rchincha thanks, I will verify