envoy@v1.32.3/go.mod checksum mismatch

[jarrodhroberson]

I am trying to build a project and when I updated dependencies I started getting this error.

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build b1770871-d733-4430-bc45-892600527fa9 status: FAILURE
go: downloading go1.23.4 (linux/amd64)
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod: checksum mismatch
        downloaded: h1:F6hWupPfh75TBXGKA++MCT/CZHFq5r9/uwt/kQYkZfE=
        go.sum:     h1:c955gQjaXHsMxMjHjEZ7nwIzMJYxXpN+sJIGufsSbg4=

I can get around it locally by setting GONOSUMDB='github.com/envoyproxy/*' but this does not let me deploy it to google app engine where I can not specify that.

Is there a solution to this?

[mmorel-35]

Hi @jarrodhroberson ,

Would you like to share your project setup ?

[emilhauk]

Here's an example: https://github.com/emilhauk/envoy-checksum-error
This does not install the exact same version, but the problem remains the same.

[mmorel-35]

Can you try this

go get github.com/envoyproxy/go-control-plane/envoy@latest
go mod tidy
go build ./...

[LeeBrotherston]

For what it's worth, I am experiencing the same issue and so tried the suggested fix above which yields the same error:

> go get github.com/envoyproxy/go-control-plane/envoy@latest
verifying github.com/envoyproxy/go-control-plane/envoy@v1.32.3/go.mod: checksum mismatch
	downloaded: h1:c955gQjaXHsMxMjHjEZ7nwIzMJYxXpN+sJIGufsSbg4=
	go.sum:     h1:F6hWupPfh75TBXGKA++MCT/CZHFq5r9/uwt/kQYkZfE=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

I would note that in my case, the downloaded and go.sum values are the opposite way around to the original poster.

Looking at the changelog I see that there are changes to envoy/go.mod which references this issue: 4b91141

I suspect that this change may be the root cause of the issue, I have not tested but this line raised some suspicion for me:

go-control-plane/envoy/go.mod

Line 6 in 2d07f5a

replace github.com/envoyproxy/go-control-plane@v0.13.4 => ../

[emilhauk]

I also tried your suggestion, but to no avail. I get different hashes though (same as in the example repo linked above).

Then, I tried to install this using a basic docker container, which apparently works just fine 🙈:

$ docker run --rm -it golang:alpine
Unable to find image 'golang:alpine' locally
alpine: Pulling from library/golang
41974eb6247f: Download complete 
0a2ab5391ffc: Download complete 
4f4fb700ef54: Already exists 
1f3e46996e29: Download complete 
06f05ace1117: Download complete 
Digest: sha256:c23339199a08b0e12032856908589a6d41a0dab141b8b3b21f156fc571a3f1d3
Status: Downloaded newer image for golang:alpine

/go # go version
go version go1.23.4 linux/amd64

/go # go mod init example.com/m
go: creating new go.mod: module example.com/m
go: to add module requirements and sums:
	go mod tidy

/go # go get github.com/envoyproxy/go-control-plane/envoy@latest
go: warning: ignoring go.mod in $GOPATH /go
go: downloading github.com/envoyproxy/go-control-plane v0.13.4
go: downloading github.com/envoyproxy/go-control-plane/envoy v1.32.3
go: downloading google.golang.org/protobuf v1.35.2
go: downloading github.com/cncf/xds/go v0.0.0-20240723142845-024c85f92f20
go: downloading github.com/envoyproxy/protoc-gen-validate v1.1.0
go: downloading github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
go: downloading google.golang.org/grpc v1.67.1
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142
go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
go: downloading cel.dev/expr v0.16.0
go: downloading golang.org/x/net v0.28.0
go: downloading golang.org/x/sys v0.24.0
go: downloading golang.org/x/text v0.17.0
go: added cel.dev/expr v0.16.0
go: added github.com/cncf/xds/go v0.0.0-20240723142845-024c85f92f20
go: added github.com/envoyproxy/go-control-plane v0.13.4
go: added github.com/envoyproxy/go-control-plane/envoy v1.32.3
go: added github.com/envoyproxy/protoc-gen-validate v1.1.0
go: added github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10
go: added golang.org/x/net v0.28.0
go: added golang.org/x/sys v0.24.0
go: added golang.org/x/text v0.17.0
go: added google.golang.org/genproto/googleapis/api v0.0.0-20240814211410-ddb44dafa142
go: added google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142
go: added google.golang.org/grpc v1.67.1
go: added google.golang.org/protobuf v1.35.2

The issue seems to be local or system related, but affecting several others as well it seems. I have tried the following:

• go clean -cache -modcache
• manually deleting content from $GOMODCACHE

I don't get how this is the only package of countless on my machine that's acting up. Any insights and/or suggestions would be greatly appreciated 😃

[jarrodhroberson]

I can not share my project, it is not public.

I am not using this directly, it is being accessed as a dependency of a dependency. I think it a Google Cloud SDK dependency.

I tracked it down to "opentelemetry" using this version.

I can get it to work LOCALLY by either setting GONOSUMDB or replacing my local hash with the expected one.

But, when I try and deploy to Google App Engine Standard, I am NOT in control of the build environment and can not set GONOSUMDB (which I think is a hack work around).

I did finally get an absolutely crap workaround to get it to deploy by replacing the hash in go.sum with the one that GAE build expects. Which is "correct" I have no idea because the next time I tried to deploy, the hashes were reversed and I got the error again.

The only reason that the checksum would be different for the exact same version is someone replaced a published binary and did not bump the version number for some reason. I know this because when I first started using Go and releasing my code I did the same thing and it bit me immediately.

Deleting the contents of go.mod and deleting go.sum completely can affect the hashes that are retrieved. I have tried everything, including doing that. I have experienced the order of the hashes reversing as a previous poster says. That is why, replacing the hash in my go.sum before pushing to Google App Engine for deployment is not a solution, the hash that GAE pulls seems to be non-deterministic.

I am using GOPROXY=direct, which should pull directly from the source,
But, if the source is a proxy itself, OR you can not control GOPROX (I can not) this is useless. This is a fundamental error that only the project team can resolve deterministically.

There is no acceptable client side fix for this if you do not control the version of the library directly.

[jarrodhroberson]

I created a new empty project, and the very first time I used go mod tidy this is what I got.

GOROOT= #gosetup
GOPATH=/home/jhr/go #gosetup
/usr/local/go/bin/go get -t -v cloud.google.com/go/storage/... #gosetup
go: downloading google.golang.org/api v0.214.0
go: downloading google.golang.org/grpc v1.67.3
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576
go: downloading go.opentelemetry.io/otel v1.29.0
go: downloading go.opentelemetry.io/otel/trace v1.29.0
go: downloading go.opentelemetry.io/otel/metric v1.29.0
github.com/envoyproxy/go-control-plane/envoy@v1.32.3: verifying module: checksum mismatch
	downloaded: h1:c1EIw4vwYCaovxRZtyycws8aX6dJ9W2p+4bCi7mcDgw=
	sum.golang.org: h1:hVEaommgvzTjTd4xCaFd+kEQ2iYBtGxP6luyLrx6uOk=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

this is the imports

import (
	"context"
	"fmt"
	"io"
	"os"
	"time"

	"cloud.google.com/go/storage"
)

and since go mod tidy will not run this is what my go.mod looks like

module github.com/jarrodhroberson/youbroketheinternet

go 1.23.4

whatever you guys did, you broke Google Cloud Platform libraries in a very bad way.

[zmiklank]

Hello.
Running go mod tidy results in the error described in this thread also for https://github.com/istio/proxy when running the go mod tidy on my localhost.

[valerian-roche]

Hey, it seems pkg.dev cached a very short release prior to the main one (only partially).
I will cut another release to hopefully improve on this issue, but given (other) go.mod issues it's quite complicated and long to do

[jarrodhroberson]

Hey, it seems pkg.dev cached a very short release prior to the main one (only partially). I will cut another release to hopefully improve on this issue, but given (other) go.mod issues it's quite complicated and long to do

this is why you should never "release" different binaries for the exact same version. the hash for a specific exact version should NEVER change once it is RELEASED. That is what RELEASE means!

[mjnowen]

Hi, any update on this? ETA? @valerian-roche

[klauspost]

Please release a clean (new) version. My proxy is also picking this up.

[LeeBrotherston]

Not the desired fix I know, but for others who want a workaround whilst this is resolved, you can:

• Remove the package:
  • go get github.com/envoyproxy/go-control-plane@none
• Remove any locally cached copies of the module
• Re-add the module forcing a SUMDB & Proxy that has the "right" version:
  • GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go get github.com/envoyproxy/go-control-plane
  • GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go mod tidy

[mjnowen]

Corporate AppSec policy forbids use of any proxy, so using GOPROXY=direct where it is broken.

[jarrodhroberson]

this is not a solution for those of us that can not set GOSUMDB or GOPROXY because builds are happening in a CI environment or a cloud build environment like Google Cloud Platform. I finally got it to work by adding `go.sum` to my `.gitcloudignore` but this is a TERRIBLE work around security wise. I also tried using `vendor` directory and it fails if the `go.sum` is included with the build as well, since it still checks and picks up the wrong one. Why did you change the binary of a release?

…

On Mon, Jan 27, 2025 at 10:00 AM Lee Brotherston ***@***.***> wrote: Not the desired fix I know, but for others who want a workaround whilst this is resolved, you can: - Remove the package: - go get ***@***.*** - Remove any locally cached copies of the module - Re-add the module forcing a SUMDB & Proxy that has the "right" version: - GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go get github.com/envoyproxy/go-control-plane - GOSUMDB="sum.golang.org" GOPROXY="https://proxy.golang.org/" go mod tidy — Reply to this email directly, view it on GitHub <#1083 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABF7725KZMW7NDDZFG4EEL2MZCZ7AVCNFSM6AAAAABU5KE63OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJVHE4TKMRYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Jarrod Roberson 678.551.2852