CI test suites are flaky

[3nprob]

Description

Test failures have been increasing on the develop branch. Spurious/"flaky" failures mean that actual failures/regressions in affected units are not visible.

• Revert "Add the ability to exclude remote users in user directory search results (#18300)" #18388
• Add the ability to exclude remote users in user directory search results #18300 (comment)

Steps to reproduce

• merge commit on develop branch

Homeserver

n/a

Synapse Version

n/a

Installation Method

Other (please mention below)

Database

n/a

Workers

Single process

Platform

GitHub Actions

Configuration

No response

Relevant log output

test_package.go:183: CreateDirtyDeployment failed: CreateDirtyServer: Failed to deploy image complement-synapse : Error response from daemon: Conflict. The container name "/complement_fed_dirty_hs1" is already in use by container "8ac075e21564947ffad89d884e3201f09eb2d7844fd73364f953caef6396807e". You have to remove (or rename) that container to be able to reuse that name.

Anything else that would be useful to know?

No response

[anoadragon453]

It appears to be something that will need to be fixed in Complement: matrix-org/complement#775, as we can't pin GitHub runner image versions.

[3nprob]

It appears to be something that will need to be fixed in Complement: matrix-org/complement#775, as we can't pin GitHub runner image versions.

@anoadragon453 The errors here reproduce locally for me (also here) but not after #18388 and look unrelated to complement?

Conflation of possibly unrelated errors makes accepting those kinds of failure modes risky.

Failing jobs in GHA on the develop branch could be triggered to rerun/retried after the fact by someone with the proper permissions. In face of failing checks, shouldn't failing jobs on develop be retried and confirmed passing before additional commits are merged/pushed on top of the failing ones until they have passed? This would significantly reduce the risk of regressions.

Accepting "flaky tests" also often translates into accepting instable software (how do you tell the difference)?

Additionally, this hampers external contributions (we just see our PRs fail checks while we lack permission to initiate a retry on our own PRs).

[anoadragon453]

The spurious flakes resulting in:

The container name "/complement_fed_dirty_hs1" is already in use by container

and

Error starting userland proxy: listen tcp4 127.0.0.1:32837: bind: address already in use]

...errors are fixed by matrix-org/complement#776. Synapse automatically pulls in the main branch of Complement on the develop branch, so this fix has already been pulled in. Subsequently, the develop branch has started to pass tests again.

There are other flakes that need to be solved, but those are tracked in separate issues. I believe we can close this issue as it was referring to the recent regression in the Complement test suite.