test: parametrize functional tests for light client mode#7409
test: parametrize functional tests for light client mode#7409glitchminer wants to merge 2 commits into
Conversation
|
Jenkins BuildsClick to see older builds (15)
|
f513ad4 to
f3a0d7a
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #7409 +/- ##
===========================================
- Coverage 61.09% 60.99% -0.10%
===========================================
Files 836 836
Lines 117120 117120
===========================================
- Hits 71553 71439 -114
- Misses 38193 38297 +104
- Partials 7374 7384 +10
Flags with carried forward coverage won't be shown. Click here to find out more. |
at0m1x19
left a comment
at0m1x19
left a comment
There was a problem hiding this comment.
Lets add a marker tied to the bug (e.g. light_client_7393) on the light client parametrization, and exclude it from the nightly in .github/workflows/test-reliability.yml:
-m "reliability and not light_client_7393
|
|
||
| # Waku light client test IDs | ||
| FULL_NODE = "wakuV2LightClient_False" | ||
| LIGHT_CLIENT = "wakuV2LightClient_True" |
There was a problem hiding this comment.
develop already has 5 files using inline string IDs:
ids=["wakuV2LightClient_False", "wakuV2LightClient_True"]
Here is the list of files: test_wakuext_messages_sending.py, test_wakuext_message_reactions.py, test_wakuext_contact_requests.py, test_wakuext_messages_fetching.py, test_wakuext_messages_interacting.py
Introducing FULL_NODE/LIGHT_CLIENT constants and use them in the new files we leaving the repo in a mixed state. I can suggest update the existing 5 files to use the new constants.
There was a problem hiding this comment.
Thanks @at0m1x19. Done, I've set xfail instead of the marker just now but fine to align on marker if you prefer.
f3a0d7a to
8ff93da
Compare
8ff93da to
023fde6
Compare
Parametrize all functional tests to run in both full node and light client mode. Light client params marked xfail(strict=False) for #7393. Replace inline string IDs with constants.
Aligns with the convention introduced in #7412: tag the LIGHT_CLIENT parametrize entry with a named, issue-tied marker and exclude it from the reliability nightly via the workflow selector instead of using xfail strict=False. Single grep target for cleanup once #7393 lands. - Register light_client_7393 marker in tests-functional/pytest.ini - Update test-reliability.yml: -m "reliability and not light_client_7393" - Replace xfail mark with light_client_7393 across 11 test files
023fde6 to
3bba0c0
Compare
|
Switched to the marker approach. |
4 participants
Summary
Adds
waku_light_clientparametrization to 11 functional test files so each test runs in both full Waku node and light client mode.Light client is the transport layer for mobile. These tests previously only ran in full node mode, meaning light-client-specific failures were invisible to CI.
Changes
Each test class gets a
@pytest.mark.parametrize("waku_light_client", [False, True], indirect=True)decorator. Per-class fixtures are updated to pass the parameter through tobackend_new_profile. No test logic,assertions, or timeouts are changed.
The light-client parametrize entry is tagged with
@pytest.mark.light_client_7393(registered intests-functional/pytest.ini) so the known filter-subscription race in #7393 doesn't poison the nightly. The nightlyselector becomes
-m "reliability and not light_client_7393". This follows the same convention introduced in #7412 — single grep target for cleanup once #7393 lands.Reliability files (run by
test-reliability.ymlnightly):tests/reliability/test_one_to_one_messages.pytests/reliability/test_private_groups_messages.pytests/reliability/test_create_private_groups.pytests/reliability/test_contact_request.pyWakuext files (parametrized for future coverage; not currently in any selector):
tests/test_wakuext_group_chats.pytests/test_wakuext_contact_verification.pytests/test_wakuext_contact_requests.pytests/test_wakuext_message_reactions.pytests/test_wakuext_messages_fetching.pytests/test_wakuext_messages_interacting.pytests/test_wakuext_messages_sending.pyInitial results
Light client variants surface real failures. Across 4 CI runs (3 GitHub Actions 2-core, 1 Codespace 4-core):
All light client failures are
TimeoutError: Expected 1 signal(s) of type SignalType.MESSAGES_NEW, but got 0 in 60 seconds— consistent with the filter subscription race in #7393. Private group message delivery is mostaffected; contact requests are largely unaffected (they don't depend on the post-acceptance filter subscription path). Results are worse under resource pressure (2-core runner vs 4-core), potentially mirroring mobile
device constraints.
Cleanup when #7393 lands
grep -rn light_client_7393 tests-functional/ .github/workflows/finds:pytest.inipytest.mark.light_client_7393on eachLIGHT_CLIENTparametrize entrynot light_client_7393exclusion intest-reliability.ymlStrip all three.
Test plan
pytest --collect-onlyconfirms expected counts:-m "reliability"→ 100 tests (50 full + 50 light)-m "reliability and not light_client_7393"→ 57 tests (full-node only)-m "light_client_7393"→ 91 tests (all light-client variants across reliability + wakuext)