test(latency): enable hinted handoff-during-upgrades

[temichus]

enable hinted handoff for perf-regression-latency-650gb-upgrade

Testing

• (https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/hints/job/scylla-master-perf-regression-latency-650gb-roling-upgrade-v2/)
• https://jenkins.scylladb.com/view/staging/job/scylla-staging/job/artsiom_mishuta/job/hints/job/scylla-master-perf-regression-latency-650gb-roling-upgrade-tablest-v2/

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[temichus]

(latency during upgrades 650gb) - all good

[temichus]

(latency during upgrades / tablets 650gb) - all good

[soyacz]

@temichus Is there a proof it works correctly? E.g. a picture from monitoring showing hints sent?

[temichus]

@temichus Is there a proof it works correctly? E.g. a picture from monitoring showing hints sent?
@soyacz
proof (latency during upgrades 650gb write test)
hydra investigate show-monitor d7a4ad8e-de0b-4530-b8d2-c0b712f58da9

[roydahan]

Same as the other one.
We won't change the test while there is an ongoing investigation for regression.

[temichus]

fresh bisect runs for hinted-handoff-latency-during-upgrades and branch-perf-v15 branches

write hinted-handoff

write branch-perf-v15

write hinted-handoff(tablets)

write branch-perf-v15(tablets)

[temichus]

read hinted-handoff

read branch-perf-v15

read hinted-handoff (tablets)

read branch-perf-v15 (tablets)

[temichus]

mixed hinted-handoff

mixed branch-perf-v15

mixed hinted-handoff(tablets)
Test id: a2218a66-b341-466a-a792-cb5c8f548b03

mixed branch-perf-v15(tablets)
Test id: cea1ce69-0120-4792-a4eb-90d5bdbe06d6

[temichus]

in mixed, both show bad results but hinted handoff - worst, ain 2 reruns, picture are the same
cc @piodul