Is there an existing issue for this?
Is this happening on an up to date version of Incus?
Incus system details
Current Incus main branch, testing in a ricsv64 VM with 4 CPUs and 4GiB RAM.
Instance details
No response
Instance log
No response
Current behavior
There's some sort of a race condition in the internal/server/endpoints tests that causes occasional failures due to timeout. The issue seems to be more easily triggered on slower, heavily loaded systems. I encounter a failure 20-30% of the time in a riscv64 VM with 4 CPUs and 4GiB RAM when running all defined unit tests. Simply running the individual problematic test in a loop doesn't seem to trigger the race condition.
Three different tests fail (TestEndpoints_LocalCreateUnixSocket, TestEndpoints_LocalSocketBasedActivation, and TestEndpoints_DevIncusCreateUnixSocket), but the root cause looks identical in all cases. I've attached full tracebacks from each failure.
panic: test timed out after 10m0s
running tests:
TestEndpoints_LocalCreateUnixSocket (9m59s)
[snip]
goroutine 42 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0x0?, 0xf4?)
/usr/lib/go-1.26/src/runtime/sema.go:114 +0x42
sync.(*WaitGroup).Wait(0x9adcb9f6690)
/usr/lib/go-1.26/src/sync/waitgroup.go:206 +0x84
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Wait(...)
/home/gibmat/incus/internal/server/endpoints/endpoints.go:556
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).Down(0x9adcb9f6640)
/home/gibmat/incus/internal/server/endpoints/endpoints.go:401 +0x1be
github.com/lxc/incus/v6/internal/server/endpoints_test.newEndpoints.func1()
/home/gibmat/incus/internal/server/endpoints/endpoints_test.go:48 +0x34
github.com/lxc/incus/v6/internal/server/endpoints_test.TestEndpoints_LocalCreateUnixSocket(0x9adcbbb2248)
/home/gibmat/incus/internal/server/endpoints/local_test.go:28 +0xd6
testing.tRunner(0x9adcbbb2248, 0x80f3f8)
/usr/lib/go-1.26/src/testing/testing.go:2036 +0xc4
created by testing.(*T).Run in goroutine 1
/usr/lib/go-1.26/src/testing/testing.go:2101 +0x38a
TestEndpoints_DevIncusCreateUnixSocket.txt
TestEndpoints_LocalCreateUnixSocket.txt
TestEndpoints_LocalSocketBasedActivation.txt
### Expected behavior
_No response_
### Steps to reproduce
Repeatedly run `go test -count=1 ./...`
Is there an existing issue for this?
Is this happening on an up to date version of Incus?
Incus system details
Current Incus main branch, testing in a ricsv64 VM with 4 CPUs and 4GiB RAM.Instance details
No response
Instance log
No response
Current behavior
There's some sort of a race condition in the
internal/server/endpointstests that causes occasional failures due to timeout. The issue seems to be more easily triggered on slower, heavily loaded systems. I encounter a failure 20-30% of the time in a riscv64 VM with 4 CPUs and 4GiB RAM when running all defined unit tests. Simply running the individual problematic test in a loop doesn't seem to trigger the race condition.Three different tests fail (
TestEndpoints_LocalCreateUnixSocket,TestEndpoints_LocalSocketBasedActivation, andTestEndpoints_DevIncusCreateUnixSocket), but the root cause looks identical in all cases. I've attached full tracebacks from each failure.TestEndpoints_DevIncusCreateUnixSocket.txt
TestEndpoints_LocalCreateUnixSocket.txt
TestEndpoints_LocalSocketBasedActivation.txt