S3 with minio: IncompleteBody

[guskovd]

I'm trying to run nixery with s3 backend (minio):

NIX_TIMEOUT=3600 S3_ACCESS_KEY=<my_access_key> S3_SECRET_KEY=<my_secret_key> S3_ENDPOINT=<my_minio_endpoint> PORT=8080 NIXERY_CHANNEL=nixos-25.11 NIXERY_STORAGE_BACKEND=s3 S3_BUCKET=nix ./result/bin/server

$ docker pull localhost:8080/hello

Trying to pull localhost:8080/hello:latest...
Error: unable to copy from source docker://localhost:8080/hello:latest: initializing source docker://localhost:8080/hello:latest: reading manifest latest in localhost:8080/hello: received unexpected HTTP status: 500 Internal Server Error

Nixery logs:

{"time":"2026-05-28T20:39:15.136555166Z","level":"INFO","source":{"function":"main.main","file":"github.com/google/nixery/cmd/server/main.go","line":245},"msg":"initialised logging","service":"nixery","version":"depot"}
{"time":"2026-05-28T20:39:15.13662678Z","level":"INFO","source":{"function":"github.com/google/nixery/config.pkgSourceFromEnv","file":"github.com/google/nixery/config/pkgsource.go","line":121},"msg":"using Nix package set from Nix channel or commit","channel":"nixos-25.11"}
{"time":"2026-05-28T20:39:15.136829593Z","level":"INFO","source":{"function":"main.main","file":"github.com/google/nixery/cmd/server/main.go","line":267},"msg":"initialised storage backend","backend":"S3 (nix)"}
{"time":"2026-05-28T20:39:15.136845218Z","level":"INFO","source":{"function":"main.main","file":"github.com/google/nixery/cmd/server/main.go","line":293},"msg":"starting Nixery","version":"depot","port":"8080"}
{"time":"2026-05-28T20:39:17.115350027Z","level":"INFO","source":{"function":"main.(*registryHandler).serveManifestTag","file":"github.com/google/nixery/cmd/server/main.go","line":142},"msg":"requesting image manifest","image":"hello","tag":"latest"}
{"time":"2026-05-28T20:39:17.11622237Z","level":"INFO","source":{"function":"github.com/google/nixery/builder.callNix","file":"github.com/google/nixery/builder/builder.go","line":190},"msg":"invoked Nix build","cmd":"nixery-prepare-image","image":"hello"}
{"time":"2026-05-28T20:39:17.543464056Z","level":"INFO","source":{"function":"github.com/google/nixery/builder.prepareImage","file":"github.com/google/nixery/builder/builder.go","line":243},"msg":"finished image preparation via Nix","image":"hello","tag":"latest"}
{"time":"2026-05-28T20:39:17.594975308Z","level":"DEBUG","source":{"function":"github.com/google/nixery/builder.layerFromCache","file":"github.com/google/nixery/builder/cache.go","line":157},"msg":"failed to retrieve cached layer from storage backend","err":"operation error S3: GetObject, https response error StatusCode: 404, RequestID: 18B3D54219EC5BBF, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, NoSuchKey: The specified key does not exist.","layer":"ee68a7492a010fd928c60c04c4ca23fa773e1041","backend":"S3 (nix)"}
{"time":"2026-05-28T20:39:21.574171062Z","level":"INFO","source":{"function":"github.com/google/nixery/builder.prepareLayers.func1","file":"github.com/google/nixery/builder/builder.go","line":290},"msg":"created image layer","layer":"ee68a7492a010fd928c60c04c4ca23fa773e1041","packages":["hello-2.12.2","glibc-2.40-224","xgcc-14.3.0-libgcc","libunistring-1.4.1","libidn2-2.3.8"],"tarhash":"sha256:af9971b972a60092eacc78b67947ea7ba0329216d4332ff71ea8366a12d52008"}
{"time":"2026-05-28T20:39:21.645283291Z","level":"ERROR","source":{"function":"github.com/google/nixery/storage.(*S3Backend).Persist","file":"github.com/google/nixery/storage/s3.go","line":114},"msg":"failed to write to S3","err":"operation error S3: PutObject, https response error StatusCode: 400, RequestID: 18B3D5430806BDB8, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error IncompleteBody: You did not provide the number of bytes specified by the Content-Length HTTP header.","key":"staging/ee68a7492a010fd928c60c04c4ca23fa773e1041"}
{"time":"2026-05-28T20:39:21.645356884Z","level":"ERROR","source":{"function":"github.com/google/nixery/builder.uploadHashLayer","file":"github.com/google/nixery/builder/builder.go","line":393},"msg":"failed to create and store layer","err":"operation error S3: PutObject, https response error StatusCode: 400, RequestID: 18B3D5430806BDB8, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error IncompleteBody: You did not provide the number of bytes specified by the Content-Length HTTP header.","layer":"ee68a7492a010fd928c60c04c4ca23fa773e1041","backend":"S3 (nix)"}
{"time":"2026-05-28T20:39:21.645440895Z","level":"ERROR","source":{"function":"main.(*registryHandler).serveManifestTag","file":"github.com/google/nixery/cmd/server/main.go","line":150},"msg":"failed to build image manifest","err":"operation error S3: PutObject, https response error StatusCode: 400, RequestID: 18B3D5430806BDB8, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error IncompleteBody: You did not provide the number of bytes specified by the Content-Length HTTP header.","image":"hello","tag":"latest"}