Nix 2.21.x -> 2.22.x `download buffer is full` on CentOS 7

[MatthewCroughan]

Describe the bug

When using CentOS 7, I ran into an issue with nix versions 2.21.x through to 2.22.x, where builtin fetchers such as those
triggered when running nix build nixpkgs#hello fill up a buffer and fail to fetch.

Downgrading to Nix 2.18.1 or 2.18.2 makes this issue stop.

When running nix build nixpkgs#hello, nix will fail to download the GitHub tarball. Running with -vvv shows the following

download thread waiting for 100 ms
download buffer is full; going to sleep

Eventually, the download fails and the following error is emitted

error:
       … while fetching the input 'github:NixOS/nixpkgs/nixpkgs-unstable'

       error: cannot get archive member name: truncated gzip input

Steps To Reproduce

1. Get CentOS 7
2. Install Nix via the nixos.org installer
3. Get nix 2.21.x or 2.22.x
4. nix build nixpkgs#hello

Expected behavior

For Nix to successfully download and unpack the tarball.

nix-env --version output

$ nix-env --version
nix-env (Nix) 2.21.2

Additional context

I'm sorry I can't provide more helpful steps to reproduce the bug, though I'm happy to help if anyone can instruct me on what to run.

Priorities

Add 👍 to issues you find important.

[bodokaiser]

@MatthewCroughan have you found a workaround?

[MatthewCroughan]

@bodokaiser Other than using a Nix release prior to this, no. I wanted to perform a time consuming git bisect, but wouldn't do that unless paid to do so, due to the esoteric nature of the regression effecting ancient Linuxes that I don't have much motivation to touch otherwise.

[bodokaiser]

@MatthewCroughan How did you perform the downgrade? Did you uninstall nix and just installed an older nix, or could you use nix nix-upgrade?

Did you disable SELinux? Are you using the multi-user install? Is there a proxy in your network?

(I also have multiple problems including the cache's SSL due to a proxy - trying to disentangle them)

[lentilus]

I am having similar issues but on a different setup. I am running docker on a fedora host. Everything is fine on the host but in the docker container (devpod) nix builds are really slow. They dont fail, just take really long. The logs (-vvvvv) show hundreds of lines like this:

download thread waiting for 100 ms
download thread waiting for 100 ms
download thread waiting for 100 ms
download buffer is full; going to sleep
download buffer is full; going to sleep
download thread waiting for 100 ms
download buffer is full; going to sleep

I have not verified that downgrading makes things better but right now I am on 2.23. I am really not sure but maybe the issue is related: #11249

[bodokaiser]

Have you checked if you have enough space for your cache dir? This was the limitation for me in the endOn 7. Aug 2024, at 15:26, Lentilus ***@***.***> wrote: I am having similar issues but on a different setup. I am running docker on a fedora host. Everything is fine on the host but in the docker container (devpod) nix builds are really slow. They dont fail, just take really long. The logs (-vvvvv) show hundreds of lines like this: download thread waiting for 100 ms download thread waiting for 100 ms download thread waiting for 100 ms download buffer is full; going to sleep download buffer is full; going to sleep download thread waiting for 100 ms download buffer is full; going to sleep I have not verified that downgrading makes things better but right now I am on 2.23. I am really not sure but maybe the issue is related: #11249 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[lentilus]

Can you elaborate on that? Do you mean the store directory or is there an additional cache that I don't know about? On my host I have about 100GiB of free disk.
The logs from devpod tell me that the container is started using the following command

19:49:05 debug Running docker command: docker run --sig-proxy=false --mount type=bind,src=/home/lentilus/git/2ndpod,dst=/workspaces/2ndpod -u root -e DEVPOD=true -e REMOTE_CONTAINERS=true -l dev.containers.id=2ndpod-def-e4a1c -l devcontainer.metadata=[{"id":"ghcr.io/devcontainers/features/common-utils:2"},{"id":"ghcr.io/devcontainers/features/git:1"},{"remoteUser":"vscode"},{"entrypoint":"/usr/local/share/nix-entrypoint.sh"},{"onCreateCommand":{"":["sudo chsh -s /usr/bin/zsh $USER"]}}] -l devpod.user=root -d --entrypoint /bin/sh vsc-2ndpod-1e65f:devpod-1a149c9eba5e1e523404f67734dea86e -c echo Container started
trap "exit 0" 15
/usr/local/share/nix-entrypoint.sh
exec "$@"
while sleep 1 & wait $!; do :; done -

I think this means that the container should have access to the full 100GiB, because there is no flag that suggests otherwise...
And nixpkgs#hello should not be all that big after all.

[bodokaiser]

Yes, sure. In addition to /store, nix uses $USER/.cache.In our case, the user directories were on a network drive which limits the total size of a user directory to 10 GB. As you said that you are using nix inside a Docker container, it could very well be that you have some space restrictions.Maybe you can mount $user/.cache to the host systemFor us, we changed the env variable XDG_CACHE_HOME (double check if this is the correct name, I am not at home right now to check the correct name), to something pointing on the local hard drive instead of the users home drive being a network share.On 7. Aug 2024, at 20:26, Lentilus ***@***.***> wrote: Can you elaborate on that? Do you mean the store directory or is there an additional cache that I don't know about? On my host I have about 100GiB of free disk. The logs from devpod tell me that the container is started using the following command 19:49:05 debug Running docker command: docker run --sig-proxy=false --mount type=bind,src=/home/lentilus/git/2ndpod,dst=/workspaces/2ndpod -u root -e DEVPOD=true -e REMOTE_CONTAINERS=true -l dev.containers.id=2ndpod-def-e4a1c -l devcontainer.metadata=[{"id":"ghcr.io/devcontainers/features/common-utils:2"},{"id":"ghcr.io/devcontainers/features/git:1"},{"remoteUser":"vscode"},{"entrypoint":"/usr/local/share/nix-entrypoint.sh"},{"onCreateCommand":{"":["sudo chsh -s /usr/bin/zsh $USER"]}}] -l devpod.user=root -d --entrypoint /bin/sh vsc-2ndpod-1e65f:devpod-1a149c9eba5e1e523404f67734dea86e -c echo Container started trap "exit 0" 15 /usr/local/share/nix-entrypoint.sh exec "$@" while sleep 1 & wait $!; do :; done - I think this means that the container should have access to the full 100GiB, because there is no flag that suggests otherwise... And nixpkgs#hello should not be all that big after all. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[roberth]

They dont fail, just take really long. The logs (-vvvvv) show hundreds of lines like this:

download thread waiting for 100 ms
download thread waiting for 100 ms
download thread waiting for 100 ms
download buffer is full; going to sleep

@lentilus This sounds like it could be solved by #11171 ie master or one of the backports.

And nixpkgs#hello should not be all that big after all.

Nixpkgs itself is pretty big though, and it's written to a cache that's implemented by means of a git repo in ~/.cache/nix/tarball-cache/.

[lentilus]

Thanks @bodokaiser for the details! Unfortunately the size of the cache dir did not seem to be the issue. This may be a stupid question, but is there any way around building nix from source to check if #11171 fixes it? The commit is quite recent and not present in any of the releases, right?

[roberth]

It's in Nix 2.24, available as nixVersions.latest in the nixos-unstable channel.
2.23-maintenance has not been tagged for this yet.