initial runner api (#1732)

* initial runner map api

* return Run objects

* WIP: click to method mapping

* WIP: click to method, adding groups

* remove comment

* pass in args and opts correctly

* fix validation for top level params

* restore metaflow_runner.py

* handle list in options

* fix nargs

* fix issue with validating missing args

* inject flow parameters as well

* handle include file

* internal click to chaining methods functionality

* subprocess manager

* simpler subprocess manager

* handle SIGINT

* wait() handles timeout + logs both

* improved log streaming with ability to resume + stream multiple times

* cleanup in context manager, otherwise user is responsible

* remove aiofiles

* split into CommandManager and SubprocessManager

* return components array instead

* remove assert

* add type hints

* return ExecutingRun encapsulating CommandManager and metaflow.Run

* print as string

* suggested improvements

* use correct variable name

* update usage of subprocess manager

* fix flags such as --pylint/--no-pylint

* cache loaded modules

* fix boolean flags

* remove aiofiles

* no f-strings

* get run object explicitly

* new higher level UX

* typeguard warning for 3.5/3.6 + add profile argument

* default None for profile

* detect early issues

* fix user defined params not being recognised

* kill all descendants

* do not use f-strings

* suggested changes

* use pkill instead of psutil

* minor fix

* vendor typeguard

* update error message

* remove comma and check in init

* Added docstrings; fixed stubs; moved stuff around

* change dir name to runner

* fix import in test

* fix import

* add status property to ExecutingRun

* add docstring for status property

---------

Co-authored-by: Romain Cledat <[email protected]>

Author: madhur-ob

metaflow/__init__.py

@@ -146,6 +146,10 @@ class and related decorators.
 # Import data class within tuple_util but not introduce new symbols.
 from . import tuple_util
 
+# Runner API
+if sys.version_info >= (3, 7):
+    from .runner.metaflow_runner import Runner
+
 __version_addl__ = []
 _ext_debug("Loading top-level modules")
 for m in _tl_modules:

metaflow/_vendor/importlib_metadata.LICENSE

@@ -0,0 +1,13 @@
+Copyright 2017-2019 Jason R. Coombs, Barry Warsaw
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.