improvement(nemesis): Rework nemesis discovery #10502

pehala · 2025-03-25T14:35:22Z

Problem statement

Currently, the Nemesis class has too many responsibilities, which causes problems, one of the problems is that it requires parameters to initialize and does a lot of logic in the init function and you need to provide those even if you need only part of the class. Problem manifests can be seen in test_nemesis_sisyphus which needs to mock Tester, Cluster and Node to filter nemesis, but none of methods used required any of this, we need to provide it because of the aforementioned problem.

Solution

Extract Nemesis discovery (i.e. Gathering all disrupt method and matching it with subclasses) to a separate class. While doing so, also reduce the nemesis discovery methods needed. Only one method for filtering is now present (NemesisRegistry.get_disrupt_methods) and input is a logical phrase. To allow also extracting properties to nemesis.yaml/nemesis_classes.yaml add gather_properties.

nemesis.yaml was also changed from list containing strings, to a full on dict. Dict is sorted so the desired output is essentially the same, but it requires less processing to write/read.

All Monkey which only filtered by flags (LimitedChaosMonkey, GeminiNonDisruptiveChaosMonkey, GeminiChaosMonkey, NetworkMonkey, NonDisruptiveMonkey, DisruptiveMonkey, FreeTierSetMonkey)
were removed and replaced by nemesis_selector usage

Changelog

Add NemesisRegistry class
- Responsible for gathering and filtering available disrupt methods
- Semi-generic, theoretically it could be used on any class that has disrupt method, not only Nemesis
  - Currently only requirement is that is has disrupt method
- Requires no runtime information to initialize
Limit source code searching to disrupt method
- This improvement changes time of test_nemesis_sisyphus from 13 sec to 3 sec, including python env initialization
Rewrite test_nemesis_sisyphus to demonstrate improvements
Remove Monkeys that use filtering by flags
Change nemesis.yaml structure into a dict

Testing

https://jenkins.scylladb.com/job/scylla-staging/job/phala/job/longevity-schema-changes-3h-test/40/
- Based on test-cases/longevity/longevity-sla-100gb-4h test, as that is one of the test affected by the changes
- Failed due to a known issue https://github.com/scylladb/scylla-enterprise/issues/2572, but the nemesis was executed
https://jenkins.scylladb.com/job/scylla-staging/job/phala/job/longevity-100gb-4h-test/10/
- General longevity to verify Sisyphus correctness

PR pre-checks (self review)

I added the relevant backport labels
I didn't leave commented-out/debugging code

sdcm/nemesis.py

sdcm/nemesis_registry.py

fruch

In general the direction LGTM

I think we can throw away a few more thing while we are at it. (with too much work)

sdcm/nemesis.py

sdcm/nemesis_registry.py

pehala · 2025-04-01T15:17:11Z

v2 - Drop the backwards compatibility of this patch and simplified the logic:

Unify filtering of nemesis
- Should simplify a lot of unnecessary logic, only filtering method is now get_disrupt_methods(logical_phrase)
Remove DEPRECATED_NEMESIS
Change structure of nemesis.yml
- It is now a dict, instead of list of strings. I think this is easier to parse
Make ComplexNemesis, which were only filtering based on flag, inherit from SisyphusMonkey (Needs testing)
- Reduces code and unifies logic with the SisyphusMonkey
Remove kubernetes flag from NemesisRegistry, move the logic into build_list_of_disruptions

sdcm/nemesis.py

vponomaryov · 2025-04-02T14:12:35Z

The improvement(nemesis): Remove deprecated Nemesis commit can and should be moved out to a separate PR and be merged.

The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.

After it, first and 4th commits could be combined into one merging their nice descriptions.

It will simplify reviewing changes of this PR.

Then,
I am already lost in the following nemesis-related PRs:

Can we consider this one (#10502) as a first in the chain and get it merged sooner than later?
Idea is great, let's not waste time...

pehala · 2025-04-02T14:19:29Z

The improvement(nemesis): Remove deprecated Nemesis commit can and should be moved out to a separate PR and be merged.
The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.

I will extract those

The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.
Can we consider this one (#10502) as a first in the chain and get it merged sooner than later?
Idea is great, let's not waste time...

We can make this one first, I can edit the rest to rely on this one

EDIT: Created #10575 and #10574

pehala · 2025-04-02T16:14:54Z

v3:

Removed Nemesis, which filter only by flags
- LimitedChaosMonkey, GeminiNonDisruptiveChaosMonkey, GeminiChaosMonkey, NetworkMonkey,
  NonDisruptiveMonkey, DisruptiveMonkey, FreeTierSetMonkey

next version should be the final after the extracted PR are incorporated and final comments are addressed

pehala · 2025-04-03T07:35:48Z

v4:

Rebased on top of master and squashed commits
Add docstrings

I also updated cover letter and started a test run to verify correctness, as this patch now affects actual testcases. I will take this out of draft, once the run is finished and passing

sdcm/nemesis.py

vponomaryov · 2025-04-10T12:10:06Z

@pehala
The second commit called improvement(treewide): Remove Nemesis, which filter by flags is applicable on the current master as-is and doesn't really depend on the first commit, right?

If so, can you please move this second commit out of this PR to a separate one so we could merge it separately (earlier)?

Also, what is expected to be done in scope of this PR to make it be "ready for review" and stop being "work in progress"?

pehala · 2025-04-10T12:41:14Z

The second commit called improvement(treewide): Remove Nemesis, which filter by flags is applicable on the current master as-is and doesn't really depend on the first commit, right?

If so, can you please move this second commit out of this PR to a separate one so we could merge it separately (earlier)?

Will do

Also, what is expected to be done in scope of this PR to make it be "ready for review" and stop being "work in progress"?

I need to finish testing it to verify I did not break anything. Latest test seemed to be working but I need to dive deeper

…lass * Add NemesisRegistry class * It is Responsible for discovering and filtering NemesisClasses and disrupt methods * Doesnt need to instance Nemesis class * Reduced number of method comapred to previous: * get_disrupt_methods for filtering, takes in logical_phrase * gather_properties for exporting * Change nemesis.yml structure * Now it is a pure dict, instead of list of strings * test_nemesis_sisyphus.py no longer needs Fake classes to generate the .yml files * Speed up nemesis discovery by checking source code only for the disrupt method * Change nemesis binding to be based on Class instead of an Instance

…_methods Previously it worked because we already had a disrupt_add_remove_dc disrupt method. Change it so it actually tests @Nemesis.add_disrupt_method

Currently, two separate mechanisms exist to call disrupt methods: build_list_of_disruptions used by SisyphusMonkey call_random_disrupt_method used by all complex monkeys These two mechanism differ in yaml properties it supports, in usage of random_seed and how they discover disrupt methods. Former using NemesisRegistry, later filtering it directly. This commit unities both of the usecases under one code Remove AllMonkey and ChaosMonkey as they can be replaced by SisiphusMonkey

pehala · 2025-04-16T09:59:32Z

Tests passed, let me know if you would like more cases tested, so I am making this ready for review

vponomaryov

LGTM

fruch

LGTM

fruch · 2025-04-20T06:28:39Z

sdcm/nemesis.py

        self.log.info(f"List of Nemesis to execute: {self._disruption_list_names}")

+    @cached_property
+    def infinite_cycle(self):


it's not clear why we didn't apply this long long ago.

every time we hit it, we just enlarge the number on nemesis_multiply_factor which is quite silly, since it was meant for the shuffle and not for making sure we have nemesis running along the whole test.

fruch

LGTM

vponomaryov · 2025-04-22T10:36:25Z

@soyacz and @roydahan
Any objections? If not, then let's just merge it

vponomaryov · 2025-04-23T12:10:32Z

@soyacz and @roydahan Any objections? If not, then let's just merge it

ping

soyacz · 2025-04-23T12:41:40Z

@soyacz and @roydahan Any objections? If not, then let's just merge it

No objections from my side.

pehala requested a review from fruch March 25, 2025 14:35

github-actions bot assigned pehala Mar 25, 2025

pehala force-pushed the extract_nemesis_discovery branch 2 times, most recently from 6ba1735 to 2251dae Compare March 25, 2025 15:06

fruch requested a review from roydahan April 1, 2025 08:29

fruch reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

fruch reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis_registry.py Outdated Show resolved Hide resolved

fruch reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis_registry.py Outdated Show resolved Hide resolved

fruch reviewed Apr 1, 2025

View reviewed changes

fruch requested a review from a team April 1, 2025 08:46

vponomaryov reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

vponomaryov reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

vponomaryov reviewed Apr 1, 2025

View reviewed changes

sdcm/nemesis_registry.py Outdated Show resolved Hide resolved

pehala force-pushed the extract_nemesis_discovery branch from 2251dae to df1154c Compare April 1, 2025 15:11

soyacz reviewed Apr 2, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

sdcm/nemesis.py Outdated Show resolved Hide resolved

fruch reviewed Apr 2, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

vponomaryov mentioned this pull request Apr 2, 2025

improvement(nemesis): speed up nemesis method discovering #10551

Closed

2 tasks

vponomaryov mentioned this pull request Apr 2, 2025

improvement(nemesis): Remove deprecated Nemesis #10574

Merged

3 tasks

pehala force-pushed the extract_nemesis_discovery branch 3 times, most recently from cdbab07 to 3eb6b15 Compare April 3, 2025 07:23

soyacz reviewed Apr 3, 2025

View reviewed changes

sdcm/nemesis.py Outdated Show resolved Hide resolved

sdcm/nemesis.py Outdated Show resolved Hide resolved

pehala force-pushed the extract_nemesis_discovery branch from 3eb6b15 to b82d4d3 Compare April 3, 2025 08:46

pehala mentioned this pull request Apr 4, 2025

Make test_nemesis independent from Nemesis disruption methods #10592

Closed

pehala force-pushed the extract_nemesis_discovery branch from b82d4d3 to 2b8b3e0 Compare April 4, 2025 13:01

pehala changed the title ~~improvement(nemesis): Extract nemesis discovery~~ improvement(nemesis): Rework nemesis discovery Apr 4, 2025

pehala force-pushed the extract_nemesis_discovery branch 3 times, most recently from 3f3fa05 to dfeca9c Compare April 8, 2025 05:14

pehala added 3 commits April 16, 2025 11:11

improvement(test_nemesis): Improve test_list_nemesis_of_added_disrupt…

ab0f4be

…_methods Previously it worked because we already had a disrupt_add_remove_dc disrupt method. Change it so it actually tests @Nemesis.add_disrupt_method

pehala force-pushed the extract_nemesis_discovery branch from dfeca9c to 761806d Compare April 16, 2025 09:12

pehala marked this pull request as ready for review April 16, 2025 09:58

vponomaryov approved these changes Apr 18, 2025

View reviewed changes

vponomaryov requested review from soyacz and fruch April 18, 2025 10:33

fruch approved these changes Apr 20, 2025

View reviewed changes

fruch reviewed Apr 20, 2025

View reviewed changes

fruch approved these changes Apr 20, 2025

View reviewed changes

vponomaryov removed request for soyacz and roydahan April 23, 2025 13:28

vponomaryov merged commit 86f814e into scylladb:master Apr 23, 2025
7 checks passed

scylladbbot added the promoted-to-master label Apr 23, 2025

vponomaryov added the backport/2025.1 label Apr 28, 2025

scylladbbot mentioned this pull request Apr 28, 2025

[Backport 2025.1] improvement(nemesis): Rework nemesis discovery #10713

Merged

4 tasks

scylladbbot added backport/2025.1-done and removed backport/2025.1 labels Apr 29, 2025

soyacz mentioned this pull request May 27, 2025

improvement(nemesis_tests): Make tests independant from production Nemesis #10912

Merged

3 tasks

improvement(nemesis): Rework nemesis discovery #10502

improvement(nemesis): Rework nemesis discovery #10502

Uh oh!

Conversation

pehala commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem statement

Solution

Changelog

Testing

PR pre-checks (self review)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fruch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pehala commented Apr 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vponomaryov commented Apr 2, 2025

Uh oh!

pehala commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pehala commented Apr 2, 2025

Uh oh!

pehala commented Apr 3, 2025

Uh oh!

Uh oh!

Uh oh!

vponomaryov commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pehala commented Apr 10, 2025

Uh oh!

pehala commented Apr 16, 2025

Uh oh!

vponomaryov left a comment

Choose a reason for hiding this comment

Uh oh!

fruch left a comment

Choose a reason for hiding this comment

Uh oh!

fruch Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

fruch left a comment

Choose a reason for hiding this comment

Uh oh!

vponomaryov commented Apr 22, 2025

Uh oh!

vponomaryov commented Apr 23, 2025

Uh oh!

soyacz commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

pehala commented Mar 25, 2025 •

edited

Loading

pehala commented Apr 2, 2025 •

edited

Loading

vponomaryov commented Apr 10, 2025 •

edited

Loading