Skip to content

Race in internal/server/endpoints tests causes occasional failures #3252

@gibmat

Description

@gibmat

Is there an existing issue for this?

  • There is no existing issue for this bug

Is this happening on an up to date version of Incus?

  • This is happening on a supported version of Incus

Incus system details

Current Incus main branch, testing in a ricsv64 VM with 4 CPUs and 4GiB RAM.

Instance details

No response

Instance log

No response

Current behavior

There's some sort of a race condition in the internal/server/endpoints tests that causes occasional failures due to timeout. The issue seems to be more easily triggered on slower, heavily loaded systems. I encounter a failure 20-30% of the time in a riscv64 VM with 4 CPUs and 4GiB RAM when running all defined unit tests. Simply running the individual problematic test in a loop doesn't seem to trigger the race condition.

Three different tests fail (TestEndpoints_LocalCreateUnixSocket, TestEndpoints_LocalSocketBasedActivation, and TestEndpoints_DevIncusCreateUnixSocket), but the root cause looks identical in all cases. I've attached full tracebacks from each failure.

panic: test timed out after 10m0s
        running tests:
                TestEndpoints_LocalCreateUnixSocket (9m59s)

[snip]

goroutine 42 [sync.WaitGroup.Wait, 9 minutes]:
sync.runtime_SemacquireWaitGroup(0x0?, 0xf4?)
        /usr/lib/go-1.26/src/runtime/sema.go:114 +0x42
sync.(*WaitGroup).Wait(0x9adcb9f6690)
        /usr/lib/go-1.26/src/sync/waitgroup.go:206 +0x84
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Wait(...)
        /home/gibmat/incus/internal/server/endpoints/endpoints.go:556
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).Down(0x9adcb9f6640)
        /home/gibmat/incus/internal/server/endpoints/endpoints.go:401 +0x1be
github.com/lxc/incus/v6/internal/server/endpoints_test.newEndpoints.func1()
        /home/gibmat/incus/internal/server/endpoints/endpoints_test.go:48 +0x34
github.com/lxc/incus/v6/internal/server/endpoints_test.TestEndpoints_LocalCreateUnixSocket(0x9adcbbb2248)
        /home/gibmat/incus/internal/server/endpoints/local_test.go:28 +0xd6
testing.tRunner(0x9adcbbb2248, 0x80f3f8)
        /usr/lib/go-1.26/src/testing/testing.go:2036 +0xc4
created by testing.(*T).Run in goroutine 1
        /usr/lib/go-1.26/src/testing/testing.go:2101 +0x38a

TestEndpoints_DevIncusCreateUnixSocket.txt
TestEndpoints_LocalCreateUnixSocket.txt
TestEndpoints_LocalSocketBasedActivation.txt


### Expected behavior

_No response_

### Steps to reproduce

Repeatedly run `go test -count=1 ./...`

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions