go.sum not updated if local package is missing due to go generate

[tfaller]

How are you running Renovate?

Self-hosted Renovate

Which platform you running Renovate on?

Gitea

Which version of Renovate are you using?

43.0.8

Please tell us more about your question or problem

The issue

If you generate code with go generate e.g. for API endpoints from a spec file, you often don't commit these generated files into the repo. This can lead to "missing" packages. Renovate performs a go get -t ./... to update the go.sum file after changing the go.mod. The issue here is, that this command fails, because it can't find the local "missing" packages, because go generate was not run. Because go get fails, the go.sum will never be updated -> the PR will just contain the go.mod. CI jobs will fail, because they need an updated go.sum.

I see two possible solutions:

Option 1: Run go generate

The current existing option goGenerate runs after vendoring. Not before go get. Maybe run it also before go getor introduce a second option for this.

Option 2: Create the missing packages

We could simply create the missing packages beforehand. It is easy to identify these by running go list ./.... Output like:

internal/endpoint.go:14:2: no required module provides package example.com/my-module/internal/api; to add it:
        go get example.com/my-module/internal/api
example.com/my-module/cmd
example.com/my-module/internal

It lists all existing and missing packages. If a missing package matches the module name in the go.mod we know it is a local one -> create a folder with a dummy package.gofile, just containing package name. Also make sure we don't include these in the PR.

Final thoughts

Instead of go get -t ./... we could run it specifically with the modules that were updated in the go.mod. That would at least update the go.sum. But if goModTidyis enabled, it will not work -> it will also error out because of missing packages.

"Option 2" could be the safest play, because the goGenerateoption is an "unsafe" one.

We could also say: The user must themself create these dummy package.gofiles and check them in. I see two issues with that:

• They are "junk". If some times later "go generate" creates these packages not anymore, the user has to cleanup these dummy files. The user also needs to have a .gitignore that makes sure, that only his dummy files get committed, not the generated one.
• The user has currently no idea that these missing packages cause renovate not to update the go.sum. There is no info in the PR. I only found out by looking into the Renovate debug log. If some command fails to run, we probably should add this info to the PR. But this is a whole other discussion.

What do you think?

Logs (if relevant)

Logs

DEBUG: Failed to update go.sum (repository=example, branch=renovate/github.com-aws-aws-lambda-go-1.x)
       "err": {
         "cmd": "go get -t ./...",
         "stderr": "go: example.com/my-module/cmd imports\n\texample.com/my-module/internal/api: cannot find module providing package example.com/my-module/internal/api\n",
         "stdout": "",
         "options": {
           "cwd": "/tmp/renovate/repos/gitea/example",
           "env": [
             "GOPATH",
             "GOFLAGS",
             "GIT_CONFIG_KEY_0",
             "GIT_CONFIG_VALUE_0",
             "GIT_CONFIG_KEY_1",
             "GIT_CONFIG_VALUE_1",
             "GIT_CONFIG_KEY_2",
             "GIT_CONFIG_VALUE_2",
             "GIT_CONFIG_COUNT",
             "GIT_CONFIG_KEY_3",
             "GIT_CONFIG_VALUE_3",
             "GIT_CONFIG_KEY_4",
             "GIT_CONFIG_VALUE_4",
             "GIT_CONFIG_KEY_5",
             "GIT_CONFIG_VALUE_5",
             "HOME",
             "PATH",
             "LC_ALL",
             "LANG",
             "CONTAINERBASE_CACHE_DIR"
           ],
           "maxBuffer": 10485760,
           "timeout": 900000,
           "stdin": "pipe",
           "stdout": "pipe",
           "stderr": "pipe"
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: go get -t ./...\ngo: example.com/my-module/cmd imports\n\texample.com/my-module/internal/api: cannot find module providing package example.com/my-module/internal/api\n",
         "stack": "ExecError: Command failed: go get -t ./...\ngo: example.com/my-module/cmd imports\n\texample.com/my-module/internal/api: cannot find module providing package example.com/my-module/internal/api\n\n    at ChildProcess.<anonymous> (/workspaces/renovate/lib/util/exec/common.ts:3:131)\n    at ChildProcess.emit (node:events:520:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       }