improvement(nemesis): Rework nemesis discovery

[pehala]

Problem statement

Currently, the Nemesis class has too many responsibilities, which causes problems, one of the problems is that it requires parameters to initialize and does a lot of logic in the init function and you need to provide those even if you need only part of the class. Problem manifests can be seen in test_nemesis_sisyphus which needs to mock Tester, Cluster and Node to filter nemesis, but none of methods used required any of this, we need to provide it because of the aforementioned problem.

Solution

Extract Nemesis discovery (i.e. Gathering all disrupt method and matching it with subclasses) to a separate class. While doing so, also reduce the nemesis discovery methods needed. Only one method for filtering is now present (NemesisRegistry.get_disrupt_methods) and input is a logical phrase. To allow also extracting properties to nemesis.yaml/nemesis_classes.yaml add gather_properties.

nemesis.yaml was also changed from list containing strings, to a full on dict. Dict is sorted so the desired output is essentially the same, but it requires less processing to write/read.

All Monkey which only filtered by flags (LimitedChaosMonkey, GeminiNonDisruptiveChaosMonkey, GeminiChaosMonkey, NetworkMonkey, NonDisruptiveMonkey, DisruptiveMonkey, FreeTierSetMonkey)
were removed and replaced by nemesis_selector usage

Changelog

• Add NemesisRegistry class
  • Responsible for gathering and filtering available disrupt methods
  • Semi-generic, theoretically it could be used on any class that has disrupt method, not only Nemesis
    • Currently only requirement is that is has disrupt method
  • Requires no runtime information to initialize
• Limit source code searching to disrupt method
  • This improvement changes time of test_nemesis_sisyphus from 13 sec to 3 sec, including python env initialization
• Rewrite test_nemesis_sisyphus to demonstrate improvements
• Remove Monkeys that use filtering by flags
• Change nemesis.yaml structure into a dict

Testing

• https://jenkins.scylladb.com/job/scylla-staging/job/phala/job/longevity-schema-changes-3h-test/36/
  • Based on test-cases/longevity/longevity-sla-100gb-4h test, as that is one of the test affected by the changes
• https://jenkins.scylladb.com/job/scylla-staging/job/phala/job/longevity-100gb-4h-test/2/
  • General longevity to verify Sisyphus correctness

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

[fruch]

In general the direction LGTM

I think we can throw away a few more thing while we are at it. (with too much work)

[pehala]

v2 - Drop the backwards compatibility of this patch and simplified the logic:

• Unify filtering of nemesis
  • Should simplify a lot of unnecessary logic, only filtering method is now get_disrupt_methods(logical_phrase)
• Remove DEPRECATED_NEMESIS
• Change structure of nemesis.yml
  • It is now a dict, instead of list of strings. I think this is easier to parse
• Make ComplexNemesis, which were only filtering based on flag, inherit from SisyphusMonkey (Needs testing)
  • Reduces code and unifies logic with the SisyphusMonkey
• Remove kubernetes flag from NemesisRegistry, move the logic into build_list_of_disruptions

[vponomaryov]

The improvement(nemesis): Remove deprecated Nemesis commit can and should be moved out to a separate PR and be merged.

The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.

After it, first and 4th commits could be combined into one merging their nice descriptions.

It will simplify reviewing changes of this PR.

Then,
I am already lost in the following nemesis-related PRs:

• improvement(nemesis_jobs): Improve nemesis job generation #10381
• improvement(nemesis_jobs): Rework nemesis jobs generation #10385
• improvement(nemesis): Rework nemesis discovery #10502

Can we consider this one (#10502) as a first in the chain and get it merged sooner than later?
Idea is great, let's not waste time...

[pehala]

The improvement(nemesis): Remove deprecated Nemesis commit can and should be moved out to a separate PR and be merged.
The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.

I will extract those

The same about the fix(nemesis.yml): Update nemesis.yml and nemesis_classes.yml.
Can we consider this one (#10502) as a first in the chain and get it merged sooner than later?
Idea is great, let's not waste time...

We can make this one first, I can edit the rest to rely on this one

EDIT: Created #10575 and #10574

[pehala]

v3:

• Removed Nemesis, which filter only by flags
  • LimitedChaosMonkey, GeminiNonDisruptiveChaosMonkey, GeminiChaosMonkey, NetworkMonkey,
    NonDisruptiveMonkey, DisruptiveMonkey, FreeTierSetMonkey

next version should be the final after the extracted PR are incorporated and final comments are addressed

[pehala]

v4:

• Rebased on top of master and squashed commits
• Add docstrings

I also updated cover letter and started a test run to verify correctness, as this patch now affects actual testcases. I will take this out of draft, once the run is finished and passing

[soyacz]

I wonder how it worked with NemesisSequence - this one should also be excluded. How code knows that we should not include disrupt_run_unique_sequence?

[pehala]

It does not, it seems like disrupt_run_unique_sequence was always collected, see

scylla-cluster-tests/data_dir/nemesis.yml

Line 309 in 93e7488

- disrupt_run_unique_sequence:

[soyacz]

ineed, I see it was run 41 times in 2025.1 testing. But I'm not sure if we should - this is mainly used in perf tests.

[pehala]

v5 - Unify disrupt method execution mechanisms:
Currently, two separate mechanisms exist to call disrupt methods:
build_list_of_disruptions used by SisyphusMonkey
call_random_disrupt_method used by all complex monkeys
These two mechanism differ in yaml properties it supports, in usage of random_seed
and how they discover disrupt methods.
Former using NemesisRegistry, later filtering it directly.

• Change how build_list_of_disruptions works
  • Rename it to build_disruptions_by_selector
  • Change it to return a list instead of working on self.disruptions_list
• Add build_disruptions_by_name as a replacement to call_random_disrupt_method
• Move nemesis_multiply_factor option to shuffle_list_of_disruptions instead
  • So it can be used in build_disruptions_by_name
• Remove factor scaling based on test duration
  • It seemed inconsistent and in conflict with nemesis_multiply_factor
• Complex Nemesis now use build_disruptions_by_name and call_next_nemesis() similar to SisyphusMonkey
• Remove AllMonkey and ChaosMonkey, they were the same as SisyphusMonkey after the changes

This change could be theoretically extracted out of this PR, but it would mean only one of the execution patch would call the new code. I consider ComplexNemesis as mostly outdated so this should not be a big issue