Add a job in SM to collect Scylla Doctor from whole cluster

[tarzanek]

SM and its agents can be leveraged to collect from central place the state of all nodes using
Scylla Doctor ( https://github.com/scylladb/scylla-doctor )

this will help support to properly and quickly help customers to verify their clusters and their health and ev. config drifts

Can a job be added to run SD on all nodes and collect its outputs?

[tarzanek]

@karol-kokoszka can you triage?
we can certainly add knowledge how to run SD or internal ways how this is gathered for Scylla Cloud or others

[karol-kokoszka]

@tarzanek
How to call Scylla-Doctor ?
Is it CLI that needs to be executed on the hosts or it has some API ?
Or maybe it can be called from any server (let's say manager server VM) ?

Scylla Manager is not SSHing, it's calling Agent's API, that why I'm asking about the way to execute the job.

You want to merge it with the Scylla Manager task scheduler ?

[tarzanek]

it's cli command that needs to be executed on target hosts ( https://github.com/scylladb/scylla-doctor/tree/master/scylla-doctor#usage ) as root

it results will be in a vitals file that will need a download to SM

[karol-kokoszka]

it's cli command that needs to be executed on target hosts

It means that we would need to call agent to execute the CLI and collect the output.

it results will be in a vitals file that will need a download to SM

Assuming the doctor is executed through API call to agent, it's not a problem, as it will be just in the payload.

@tarzanek could the scylla doctor be imported to the agent codebase somehow ? Through the golang dependency for example ?
UPDATE: It's python, it couldn't.

How do you see scheduling this job ? Part of the task scheduler in manager (the same as we use for repair, backup) ? Is it needed to be scheduled ? Or it's rather "ad-hoc" job ?