Frequent OscarCI/HeckeCI/... job failures on AA, Nemo" in `epoll_wait` via `uv__io_poll`, `uv_run`, `ijl_task_get_next`

[fingolfin]

I am currently seeing this error all the time AA and Nemo tests running OscarCI (i.e. running the Oscar.jl test suite on pull requests in AA and Nemo):

epoll_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:236
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/partr.c:478

Logs where I've seen this:

• OscarCI: https://github.com/Nemocas/AbstractAlgebra.jl/actions/runs/20215109728/job/58026848541?pr=1729
• HeckeCI: https://github.com/Nemocas/Nemo.jl/actions/runs/20237565853/job/58096617305?pr=2206
• regular test: https://github.com/thofma/Hecke.jl/actions/runs/20188615842/job/57974286171?pr=2106
• regular test: https://github.com/thofma/Hecke.jl/actions/runs/20229015491/job/58067211704?pr=2110
• HeckeCI: https://github.com/Nemocas/AbstractAlgebra.jl/actions/runs/20247943776/job/58132962750?pr=2254
• ...

I'll add logs to the list as I encounter them.

Anyway, it'd be good to get to the bottom of this. Could it be as "simple" as running out of RAM?

[fingolfin]

Not sure if it is related or not, but I've also seen this error a couple times now (but not as frequently as the other):

in expression starting at none:1
uv_return_spawn at ./process.jl:60
jfptr_uv_return_spawn_75909.1 at /opt/hostedtoolcache/julia/1.10.10/x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/gf.c:3077
jlcapi_uv_return_spawn_77563.1 at /opt/hostedtoolcache/julia/1.10.10/x64/lib/julia/sys.so (unknown line)
uv__wait_children at /workspace/srcdir/libuv/src/unix/process.c:167
uv__chld at /workspace/srcdir/libuv/src/unix/process.c:90
uv__signal_event at /workspace/srcdir/libuv/src/unix/signal.c:435
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:388
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/partr.c:478

See e.g.

• OscarCI: https://github.com/Nemocas/AbstractAlgebra.jl/actions/runs/20237430784/job/58095973024?pr=2252#step:7:10611

[benlorenz]

Signal 15 (Terminate) usually points to the process being killed by the kernel because of an oom situation.

Not sure if it is related or not, but I've also seen this error a couple times now (but not as frequently as the other):

in expression starting at none:1
uv_return_spawn at ./process.jl:60
jfptr_uv_return_spawn_75909.1 at /opt/hostedtoolcache/julia/1.10.10/x64/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/gf.c:3077
jlcapi_uv_return_spawn_77563.1 at /opt/hostedtoolcache/julia/1.10.10/x64/lib/julia/sys.so (unknown line)
uv__wait_children at /workspace/srcdir/libuv/src/unix/process.c:167
uv__chld at /workspace/srcdir/libuv/src/unix/process.c:90
uv__signal_event at /workspace/srcdir/libuv/src/unix/signal.c:435
uv__io_poll at /workspace/srcdir/libuv/src/unix/epoll.c:388
uv_run at /workspace/srcdir/libuv/src/unix/core.c:400
ijl_task_get_next at /cache/build/builder-amdci5-7/julialang/julia-release-1-dot-10/src/partr.c:478

See e.g.

• OscarCI: Nemocas/AbstractAlgebra.jl/actions/runs/20237430784/job/58095973024?pr=2252#step:7:10611

This backtrace looks like the process that is terminated is the main process where Pkg.test is called, not the subprocess that is actually running the tests. Hence I don't think it is caused by our tests.

I guess we will now see if this also happens on the RPTU runners, and if it does happen we can try to check the logs on the server directly.

[fingolfin]

Thing is, I am now also seeing this a lot in HeckeCI runs.

My understanding is that both OscarCI and HeckeCI, when run on GitHub runners, still execute test runs in parallel, using multiple CPUs, right?

So perhaps we need to limit the number of those... I think it detects 4 CPUs in those runners, so does it then run 4 things in parallel? Perhaps we can limit that to 3 or even just 2, at least on the GitHub runners ? Or instead of basing this on the runners, it could be based on the available RAM?

Perhaps we also need something like @benlorenz' PR #5654 for Hecke?

[benlorenz]

OscarCI will only use one runner unless NUMPROCS is set, which we do only for the selfhosted runners. We never used the github runners for parallel testing of Oscar due to the limited memory.
Both OscarCI runs that failed were in the same test-file:
experimental/DoubleAndHyperComplexes/test/spectral_sequences.jl, maybe we should check the memory usage/requirements of these tests. According to our test-stats it does 17292.8MB allocations, which is rather high but not directly a problem, depending on whether this is needed at the time or not.

I made a PR for a heap size hint in thofma/Hecke.jl#2113