Screenshot created with API has broken layout and without data sometimes

[soyacz]

Packages

Scylla version: 2024.3.0~dev-20241229.307c8f6dce8b with build-id 11053057c6c9439a10501c8cf06bdd71b8d49157

Monitoring version: 4.8, used by SCT

Kernel Version: 6.8.0-1021-aws

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

Sometimes screenshots layout is broken (one stacked on another) and data is missing when creating screenshots with API (grafana renderer).
Often it works, so issue is not easily reproducible (still happens quite often).
It is like that for several months already but it is getting more annoying recently (especially when we take screenshots during the test - it happens more often than doing it after). E.g. run from August, see screenshot tab
Examples:

Impact

useless screenshots

How frequently does it reproduce?

often for during-test-screenshot take, rarely after the test

Installation details

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 192ba60e-91ad-4e55-bf03-2faaa1cc655c
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 192ba60e-91ad-4e55-bf03-2faaa1cc655c

Logs:

• perf-latency-nemesis-ubuntu-db-node-192ba60e-5 - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_130725/perf-latency-nemesis-ubuntu-db-node-192ba60e-5-192ba60e.tar.gz
• perf-latency-nemesis-ubuntu-db-node-192ba60e-7 - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_130725/perf-latency-nemesis-ubuntu-db-node-192ba60e-7-192ba60e.tar.gz
• perf-latency-nemesis-ubuntu-db-node-192ba60e-3 - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_130725/perf-latency-nemesis-ubuntu-db-node-192ba60e-3-192ba60e.tar.gz
• perf-latency-nemesis-ubuntu-db-node-192ba60e-6 - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_130725/perf-latency-nemesis-ubuntu-db-node-192ba60e-6-192ba60e.tar.gz
• db-cluster-192ba60e.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_215447/db-cluster-192ba60e.tar.gz
• sct-runner-events-192ba60e.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_215447/sct-runner-events-192ba60e.tar.gz
• sct-192ba60e.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_215447/sct-192ba60e.log.tar.gz
• loader-set-192ba60e.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_215447/loader-set-192ba60e.tar.gz
• monitor-set-192ba60e.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/192ba60e-91ad-4e55-bf03-2faaa1cc655c/20241230_215447/monitor-set-192ba60e.tar.gz

Jenkins job URL
Argus