nix garbage collection is suspiciously slow

[jpds]

Is your feature request related to a problem? Please describe.

nix-collect-garbage is... slow on a multicore Ryzen box as it appears to be single-threaded.

The same appears to apply for nix store gc.

[roberth]

I believe most time is spent deleting things from .thrash, which is "embarrassingly parallel" as it's called. I.e. it has no interdependencies in that phase of the gc process and should be easy to run in parallel.

[thufschmitt]

that phase of the gc process and should be easy to run in parallel

Would we gain a lot from that? I would suspect that the speed of it is purely IO bound.

A non-trivial part of the GC work is the root tracing though, and that could certainly be made faster by parallelizing it if it isn't already. It's also partially IO bound because a large chunk of it is querying the database, but slightly less so.

[roberth]

I/O bound workloads can still benefit from "pipelining" usually. Concurrent requests fill the time that is otherwise lost because of request/response roundtrip latency, both in terms of CPU as well as I/O queue. The latter is probably most significant in this case (not that it matters much).

[roberth]

Also worth considering is a form of GC invocation that detaches from the terminal after, say, 5 GB has been freed up. That's generally sufficient space to start doing other things, and let it continue in the background.

Ultimately though, I feel that in everyday use, auto GC should work well enough, so that you never have to wait. If it doesn't work well enough, that's probably the real issue.

[trofi]

There is a huge CPU load aspect of garbage collection as well. For example my system has an NVME storage with ~500 000 packages.

$ time nix store gc --dry-run

real    7m15,307s
user    0m0,247s
sys     0m0,158s

perf top says all of the time is spent hammering sqlite database for queryPathInfo.

[roberth]

queryPathInfo

This method is cached, but to a limit of some 50k store paths iirc.
Also iirc 1 KB is an ok back of the envelope estimate for the memory size of such an object, so that could be 50 MB in memory. I don't know how high is acceptable to keep in memory, so perhaps it'd be good to look into caching specific columns only, and/or having something like the SymbolTable but for store paths, as references cause a lot of duplication of store paths.

Low hanging fruit is to increase the path info cache during GC to a higher but still reasonable amount.

[AlexLJordan]

If sqlite is a limiting factor, have there ever been considerations of switching database software?

[thufschmitt]

If sqlite is a limiting factor, have there ever been considerations of switching database software?

I'd be surprised if sqlite was really the limiting factor. More likely we could optimize the queries or the database layout first (contributions welcome ;) )

[roberth]

If

is the keyword here.
We don't know this, so step one is to measure time spent in sqlite, e.g. using profiling.