Enhance tool registration to support unknown hosts

[Maxusmusti]

This issue was prompted by Robert Krawitz' pbench-agent use-case with kata-vms in openshift, which have changing hostnames due to their lifespans matching the pods themselves, resulting in an inability to automate without significant added work.

Currently, pbench-agent can only register tools for known hosts (before the benchmark run). This means that the agent by itself does not work in any use case where the hosts are not already known to the controller before starting the run.

Solution: Allow tools to be registered under a "label" (rather than only under a host name)

• Remote tool meisters will then be able to connect to the redis server with a specific label to determine the tools to run
• The tool-meister hostname can then be discovered once connected, allowing for a much wider net of use-cases

What this change will require:

• Tools will need the option to be registered under a label
• The tool-meister-start and redis -> tds -> tm control flow will have to be adjusted
• Redis keys will have to be adjusted as well
• The hostnames for tms will have to be discovered through redis by the tool-data-sink, rather than having that info fed directly through the tool registration process

Additional changes that can come of this work:

• Pbench runs no longer shutting down when network connection is lost with one remote node
• Control over groups of nodes with shared labels

[portante]

Tools will need the option to be registered under a label

What is kewl about this is that one can consider a hostname being a label. So the syntax for a label must support the syntax for a host name.

The difference is that we need the CLI tools to record the tools for that label, and then add a separate way to record the host names to be used for orchestration only by the CLI.

The tool-meister-start and redis -> tds -> tm control flow will have to be adjusted

Yes. Now the Tool Meister will report back saying "I need tools for label 'foo', and I am running on host 'bar.example.com'." And we also need to allow such as report any time during the life-cycle of the Tool Data Sink, so that it can tell that Tool Meister where to pick up in the flow of events (e.g. - we already started persistent tools, so go ahead and start yours, and we have started transient tools, so you might as well start yours as well).

Redis keys will have to be adjusted as well

This is very likely, though a label is a label.