Skip to content

Commit 46d235c

Browse files
MadLittleModsMadLittleMods
authored
Add in-repo Complement tests (#19406)
This is useful so we can test Synapse specific behaviors like our admin API. (see docs in PR, `complement/README.md`) ``` COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo ``` Complement calls these ["out-of-repo"](https://github.com/matrix-org/complement/blob/78c255edcebfcb0ac5e3c86d49d76cb21fdd035a/OUT-OF-REPO-TESTS.md) tests but it's a bit of a misnomer once they're in your project. (just depends on the perspective) There has been [previous desire](#19021 (comment)) for this kind of thing but this is spawning from wanting to have some tests for our purge history admin API (element-hq/synapse-rust-apps#430). There are some Sytest tests ([`matrix-org/sytest` -> `tests/48admin.pl#L91-L618`](https://github.com/matrix-org/sytest/blob/1be04cce46c0f84abac736390dc3fab17f35a756/tests/48admin.pl#L91-L618)) for this already but I'd much rather work in Complement instead of Sytest. I'm wanting these tests to ensure that our new `event-cache` rust app for Synapse Pro doesn't break these kind of erasure features (element-hq/synapse-rust-apps#366 and element-hq/synapse-rust-apps#153). Interestingly, there is already [`matrix-org/complement` -> `tests/csapi/admin_test.go`](https://github.com/matrix-org/complement/blob/78c255edcebfcb0ac5e3c86d49d76cb21fdd035a/tests/csapi/admin_test.go) (added in matrix-org/complement#322) in the Complement repo iteslf that tests the `/_synapse/admin/v1/send_server_notice` endpoint but it's a bit of an interesting case as [Dendrite also supports this endpoint](matrix-org/dendrite#2180). I don't think it's good practice to continually shove in more and more Synapse-specific behavior into the Complement repo itself. We already have success with other out-of-repo tests for projects like the [SBG](https://github.com/element-hq/sbg/tree/b76b05b53e40bf6890e51dd1b83cec3460274eb2/complement_tests), [TI-Messenger Proxy](https://github.com/element-hq/ti-messenger-proxy/tree/c8fa87feccc743c01cccbbc2685321206b532925/complement), and our [Synapse Pro for small hosts](https://github.com/element-hq/synapse-small-hosts/tree/c2ea7eabf3e1d7c26a5312ebef326b254937be99/complement).
1 parent f6105b7 commit 46d235c

12 files changed

Lines changed: 1144 additions & 21 deletions

File tree

‎.github/workflows/latest_deps.yml‎

Lines changed: 45 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,6 @@ jobs:
126126
-exec echo "::endgroup::" \;
127127
|| true
128128
129-
130129
sytest:
131130
needs: check_repo
132131
if: needs.check_repo.outputs.should_run_workflow == 'true'
@@ -181,7 +180,6 @@ jobs:
181180
/logs/results.tap
182181
/logs/**/*.log*
183182
184-
185183
complement:
186184
needs: check_repo
187185
if: "!failure() && !cancelled() && needs.check_repo.outputs.should_run_workflow == 'true'"
@@ -214,11 +212,53 @@ jobs:
214212
cache-dependency-path: complement/go.sum
215213
go-version-file: complement/go.mod
216214

217-
- run: |
215+
- name: Run Complement Tests
216+
id: run_complement_tests
217+
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
218+
# are underpowered and don't like running tons of Synapse instances at once.
219+
# -json: Output JSON format so that gotestfmt can parse it.
220+
#
221+
# tee /tmp/gotest-complement.log: We tee the output to a file so that we can re-process it
222+
# later on for better formatting with gotestfmt. But we still want the command
223+
# to output to the terminal as it runs so we can see what's happening in
224+
# real-time.
225+
run: |
218226
set -o pipefail
219-
TEST_ONLY_IGNORE_POETRY_LOCKFILE=1 POSTGRES=${{ (matrix.database == 'Postgres') && 1 || '' }} WORKERS=${{ (matrix.arrangement == 'workers') && 1 || '' }} COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | synapse/.ci/scripts/gotestfmt
227+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest-complement.log
220228
shell: bash
221-
name: Run Complement Tests
229+
env:
230+
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
231+
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
232+
TEST_ONLY_IGNORE_POETRY_LOCKFILE: 1
233+
234+
- name: Formatted Complement test logs
235+
# Always run this step if we attempted to run the Complement tests.
236+
if: always() && steps.run_complement_tests.outcome != 'skipped'
237+
run: cat /tmp/gotest-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
238+
239+
- name: Run in-repo Complement Tests
240+
id: run_in_repo_complement_tests
241+
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
242+
# are underpowered and don't like running tons of Synapse instances at once.
243+
# -json: Output JSON format so that gotestfmt can parse it.
244+
#
245+
# tee /tmp/gotest-in-repo-complement.log: We tee the output to a file so that we can re-process it
246+
# later on for better formatting with gotestfmt. But we still want the command
247+
# to output to the terminal as it runs so we can see what's happening in
248+
# real-time.
249+
run: |
250+
set -o pipefail
251+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh --in-repo -p 1 -json 2>&1 | tee /tmp/gotest-in-repo-complement.log
252+
shell: bash
253+
env:
254+
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
255+
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
256+
TEST_ONLY_IGNORE_POETRY_LOCKFILE: 1
257+
258+
- name: Formatted in-repo Complement test logs
259+
# Always run this step if we attempted to run the Complement tests.
260+
if: always() && steps.run_in_repo_complement_tests.outcome != 'skipped'
261+
run: cat /tmp/gotest-in-repo-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
222262

223263
# Open an issue if the build fails, so we know about it.
224264
# Only do this if we're not experimenting with this action in a PR.

‎.github/workflows/tests.yml‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -707,20 +707,19 @@ jobs:
707707
cache-dependency-path: complement/go.sum
708708
go-version-file: complement/go.mod
709709

710-
# use p=1 concurrency as GHA boxes are underpowered and don't like running tons of synapses at once.
711710
- name: Run Complement Tests
712711
id: run_complement_tests
713712
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
714713
# are underpowered and don't like running tons of Synapse instances at once.
715714
# -json: Output JSON format so that gotestfmt can parse it.
716715
#
717-
# tee /tmp/gotest.log: We tee the output to a file so that we can re-process it
716+
# tee /tmp/gotest-complement.log: We tee the output to a file so that we can re-process it
718717
# later on for better formatting with gotestfmt. But we still want the command
719718
# to output to the terminal as it runs so we can see what's happening in
720719
# real-time.
721720
run: |
722721
set -o pipefail
723-
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest.log
722+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest-complement.log
724723
shell: bash
725724
env:
726725
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
@@ -729,7 +728,30 @@ jobs:
729728
- name: Formatted Complement test logs
730729
# Always run this step if we attempted to run the Complement tests.
731730
if: always() && steps.run_complement_tests.outcome != 'skipped'
732-
run: cat /tmp/gotest.log | gotestfmt -hide "successful-downloads,empty-packages"
731+
run: cat /tmp/gotest-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
732+
733+
- name: Run in-repo Complement Tests
734+
id: run_in_repo_complement_tests
735+
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
736+
# are underpowered and don't like running tons of Synapse instances at once.
737+
# -json: Output JSON format so that gotestfmt can parse it.
738+
#
739+
# tee /tmp/gotest-in-repo-complement.log: We tee the output to a file so that we can re-process it
740+
# later on for better formatting with gotestfmt. But we still want the command
741+
# to output to the terminal as it runs so we can see what's happening in
742+
# real-time.
743+
run: |
744+
set -o pipefail
745+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh --in-repo -p 1 -json 2>&1 | tee /tmp/gotest-in-repo-complement.log
746+
shell: bash
747+
env:
748+
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
749+
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
750+
751+
- name: Formatted in-repo Complement test logs
752+
# Always run this step if we attempted to run the Complement tests.
753+
if: always() && steps.run_in_repo_complement_tests.outcome != 'skipped'
754+
run: cat /tmp/gotest-in-repo-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
733755

734756
cargo-test:
735757
if: ${{ needs.changes.outputs.rust == 'true' }}

‎.github/workflows/twisted_trunk.yml‎

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,9 @@ on:
1212
twisted_ref:
1313
description: Commit, branch or tag to checkout from upstream Twisted.
1414
required: false
15-
default: 'trunk'
15+
default: "trunk"
1616
type: string
1717

18-
1918
concurrency:
2019
group: ${{ github.workflow }}-${{ github.ref }}
2120
cancel-in-progress: true
@@ -89,7 +88,6 @@ jobs:
8988
poetry add --extras tls git+https://github.com/twisted/twisted.git#trunk
9089
poetry install --no-interaction --extras "all test"
9190
- run: poetry run trial --jobs 2 tests
92-
9391
- name: Dump logs
9492
# Logs are most useful when the command fails, always include them.
9593
if: ${{ always() }}
@@ -199,11 +197,53 @@ jobs:
199197
poetry lock
200198
working-directory: synapse
201199

202-
- run: |
200+
- name: Run Complement Tests
201+
id: run_complement_tests
202+
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
203+
# are underpowered and don't like running tons of Synapse instances at once.
204+
# -json: Output JSON format so that gotestfmt can parse it.
205+
#
206+
# tee /tmp/gotest-complement.log: We tee the output to a file so that we can re-process it
207+
# later on for better formatting with gotestfmt. But we still want the command
208+
# to output to the terminal as it runs so we can see what's happening in
209+
# real-time.
210+
run: |
203211
set -o pipefail
204-
TEST_ONLY_SKIP_DEP_HASH_VERIFICATION=1 POSTGRES=${{ (matrix.database == 'Postgres') && 1 || '' }} WORKERS=${{ (matrix.arrangement == 'workers') && 1 || '' }} COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -json 2>&1 | synapse/.ci/scripts/gotestfmt
212+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest-complement.log
205213
shell: bash
206-
name: Run Complement Tests
214+
env:
215+
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
216+
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
217+
TEST_ONLY_SKIP_DEP_HASH_VERIFICATION: 1
218+
219+
- name: Formatted Complement test logs
220+
# Always run this step if we attempted to run the Complement tests.
221+
if: always() && steps.run_complement_tests.outcome != 'skipped'
222+
run: cat /tmp/gotest-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
223+
224+
- name: Run in-repo Complement Tests
225+
id: run_in_repo_complement_tests
226+
# -p=1: We're using `-p 1` to force the test packages to run serially as GHA boxes
227+
# are underpowered and don't like running tons of Synapse instances at once.
228+
# -json: Output JSON format so that gotestfmt can parse it.
229+
#
230+
# tee /tmp/gotest-in-repo-complement.log: We tee the output to a file so that we can re-process it
231+
# later on for better formatting with gotestfmt. But we still want the command
232+
# to output to the terminal as it runs so we can see what's happening in
233+
# real-time.
234+
run: |
235+
set -o pipefail
236+
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh --in-repo -p 1 -json 2>&1 | tee /tmp/gotest-in-repo-complement.log
237+
shell: bash
238+
env:
239+
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
240+
WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
241+
TEST_ONLY_SKIP_DEP_HASH_VERIFICATION: 1
242+
243+
- name: Formatted in-repo Complement test logs
244+
# Always run this step if we attempted to run the Complement tests.
245+
if: always() && steps.run_in_repo_complement_tests.outcome != 'skipped'
246+
run: cat /tmp/gotest-in-repo-complement.log | gotestfmt -hide "successful-downloads,empty-packages"
207247

208248
# open an issue if the build fails, so we know about it.
209249
open-issue:

‎changelog.d/19406.misc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add in-repo Complement tests so we can test Synapse specific behavior at an end-to-end level.

‎complement/.golangci.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Docs: https://golangci-lint.run/docs/configuration/file/
2+
#
3+
# Go formatting/linting rules
4+
#
5+
# This is run as part of the normal linting utility script,
6+
# `poetry run ./scripts-dev/lint.sh`
7+
8+
version: "2"
9+
10+
linters:
11+
# Default set of linters.
12+
# The value can be:
13+
# - `standard`: https://golangci-lint.run/docs/linters/#enabled-by-default
14+
# - `all`: enables all linters by default.
15+
# - `none`: disables all linters by default.
16+
# - `fast`: enables only linters considered as "fast" (`golangci-lint help linters --json | jq '[ .[] | select(.fast==true) ] | map(.name)'`).
17+
# Default: standard
18+
default: standard
19+
20+
# Enable specific linter.
21+
# enable:
22+
# - example
23+
24+
# Disable specific linters.
25+
disable:
26+
# FIXME: Ideally, we'd enable the `bodyclose` lint but there are many
27+
# false-positives (like https://github.com/timakin/bodyclose/issues/39) and just is
28+
# not well-suited for our use case (https://github.com/timakin/bodyclose/issues/11 and
29+
# https://github.com/timakin/bodyclose/issues/76).
30+
- bodyclose
31+
32+
formatters:
33+
# Enable specific formatter.
34+
# Default: [] (uses standard Go formatting)
35+
enable:
36+
- gofmt
37+
- goimports
38+
- golines

‎complement/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Complement testing
2+
3+
Complement is a black box integration testing framework for Matrix homeservers. It
4+
allows us to write end-to-end tests that interact with real Synapse homeservers to
5+
ensure everything works at a holistic level.
6+
7+
8+
## Setup
9+
10+
Nothing beyond a [normal Complement
11+
setup](https://github.com/matrix-org/complement?tab=readme-ov-file#running) (just Go and
12+
Docker).
13+
14+
15+
## Running tests
16+
17+
Run tests from the [Complement](https://github.com/matrix-org/complement) repo:
18+
19+
```shell
20+
# Run the tests
21+
./scripts-dev/complement.sh
22+
23+
# To run a whole group of tests, you can specify part of the test path:
24+
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
25+
# To run a specific test, you can specify the whole name structure:
26+
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate/Parallel/POST_/createRoom_makes_a_public_room
27+
# Generally though, the `-run` parameter accepts regex patterns, so you can match however you like:
28+
scripts-dev/complement.sh ./tests/... -run 'TestRoomCreate/Parallel/POST_/createRoom_makes_a_(.*)'
29+
```
30+
31+
Typically, if you're developing the Synapse and Complement tests side-by-side, you will
32+
run something like this:
33+
34+
```shell
35+
# To run a specific test
36+
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
37+
```
38+
39+
40+
### Running in-repo tests
41+
42+
In-repo Complement tests are tests that are vendored into this project. We use the
43+
in-repo test suite to test Synapse specific behaviors like the admin API.
44+
45+
To run the in-repo Complement tests, use the `--in-repo` command line argument.
46+
47+
```shell
48+
# Run only a specific test package.
49+
# Note: test packages are relative to the `./complement` directory in this project
50+
./scripts-dev/complement.sh --in-repo ./tests/...
51+
52+
# Similarly, you can also use `-run` to specify all or part of a specific test path to run
53+
scripts-dev/complement.sh --in-repo ./tests/... -run TestIntraShardFederation
54+
```

0 commit comments

Comments
 (0)