Closed
Description
Go version
go 1.23.1
Output of go env in your module/workspace:
go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\dphulkar\AppData\Local\go-build
set GOENV=C:\Users\dphulkar\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\dphulkar\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\dphulkar\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\dphulkar\go\pkg\mod\golang.org\t[email protected]
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\Users\dphulkar\go\pkg\mod\golang.org\t[email protected]\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.23.1
set GODEBUG=
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\dphulkar\AppData\Roaming\go\telemetry
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=C:\Users\dphulkar\azure-storage-azcopy\go.mod
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\dphulkar\AppData\Local\Temp\go-build1895105759=/tmp/go-build -gno-record-gcc-switchesWhat did you do?
-
Used AzCopy version 10.27.0, built with Go version 1.23.1, to copy a dataset with the following characteristics:
- Total Data Size: 567 GB
- Total Files: 1,415,230 files
-
Observed memory usage behavior during the operation.
-
Customers also reported similar memory issues when using AzCopy versions built with Go 1.23.1: Issue #2901
-
To identify the root cause, multiple experiments were conducted:
- AzCopy 10.26.0 (built with Go 1.22.x) showed stable memory usage (~54% RAM).
- AzCopy 10.27.0 (built with Go 1.23.1) exhibited a gradual memory increase, peaking at 96%.
- Downgrading the Go runtime in AzCopy 10.27.x from 1.23.1 to 1.22.5/1.22.7 mitigated the issue, with memory usage stabilizing (~54%).
- Applying a workaround by setting runtime.MemoryProfileRate=0 in AzCopy 10.27.0 (built with Go 1.23.1) failed to resolve the problem, as memory usage still peaked at 99%.
- Tried setting the environment variable GODEBUG=profstackdepth=32 to limit profiling stack depth, but memory usage remained high, peaking at 97% RAM.
- Also experimented with lowering the profstackdepth value further, but it did not resolve the issue.
What did you see happen?
- Memory Usage: Gradual increase, peaking at 96% RAM.
- CPU Usage: Fluctuating between 51% and 100%.
- Memory Profile: Did not stabilize, leading to high resource consumption over time.
Observations:
- Significant memory usage difference compared to prior versions (e.g., AzCopy 10.26).
- Downgrading the Go runtime from version 1.23.1 to 1.22.5 or 1.22.7 mitigated the issue, with memory usage stabilizing around 54%.
- The workaround of setting runtime.MemoryProfileRate=0 did not resolve the issue, as memory usage still reached 99% RAM.
What did you expect to see?
- Consistent memory usage profile, similar to AzCopy 10.26 or when using Go 1.22.5/1.22.7.
- Stabilized memory usage without a gradual increase over time.
Activity