Improve argument streaming for remote job execution (part 1)

[Shrews]

Changes to Remote Job Execution Argument Streaming

Reason For Change

Remote job execution involves a Transmitter node streaming the job keyword arguments as a JSON object to a Worker node. Adding new keyword arguments to the interface.run() or interface.run_async() methods can cause errors within the remote job streaming interface when an older Worker node receives the new, unrecognized keyword argument from a newer Transmitter node (worker/transmitter version mismatch).

For more background information on the streaming process, see this page of the documentation.

Fixes #1324

Solution

The total fix will require a multi-stage solution:

• First stage of the solution (implemented in this PR) will have the Transmitter process stream only keyword arguments that are actually specified and have a value different from their default value. This does not fix the case when a new keyword argument is introduced and used with a non-default value, but does fix older Worker nodes from failing when the new keyword is not used since the new keyword will not be sent in the argument stream.
• The second stage (not implemented here) will have the Worker process gracefully fail on unrecognized keywords.

Change Details

• RunnerConfig object will become the main source for supplying job parameters instead of kwargs.
  • Turns the BaseConfig and RunnerConfig classes into dataclasses to allow us to properly define arguments, their data types, and their default values.
    • Since the __init__() method is now autogenerated, care is taken to keep the same name for a few class attributes that did not have the same name as their argument. This is done through the use of Python class property decorators. Internally, the attribute is referenced by the non-alias version.
  • By default, all RunnerConfig attributes are streamable to a Worker. Attributes that are NEVER streamable will have explicit dataclass metadata to mark it as such (example, field(metadata={MetaValues.TRANSMIT: False})).
• interface.run() and interface.run_async() changed to accept a RunnerConfig object.
  • Can now be used in one of two ways:
    • Old-style: method takes only kwargs parameters; backwards compatible
    • New-style: method takes a RunnerConfig object and a few select parameters.
  • Non-public methods lower in the call hierarchy (init_runner(), dump_artifacts(), etc) are modified to allow for receiving a RunnerConfig object instead of keyword arguments.
• Transmitter modified to query the RunnerConfig object to retrieve the keyword arguments to transmit.
• Includes a new porting guide in the documentation describing the change.
  • Parameter docs for interface.run() method moved to BaseConfig and RunnerConfig docs.

TODO

• Testing with Controller?

[nitzmahone]

Didn't get all the way through the fine-toothed comb review before I ran out of days, but looking great- I think this solves the most important parts of the internal arg-splatting problem well.

[Shrews]

Have to revamp this a bit. The config objects come into play for transmit and process phases with this change, but did not originally. This could cause data transformations that didn't happen before.

[Akasurde]

Since you are here, on line 841, can you please change code to use os.path.isdir instead?

[Shrews]

Not sure I understand... change os.makedirs() to os.path.isdir()???

[Akasurde]

Suggested change

	debug('ssh-agent agrs added')
	debug('ssh-agent arguments added')