Merge pull request PolusAI#200 from vjaganat90/gconf2

coalesce different txt based config files into one json

Author: sameeul

.gitignore

@@ -17,11 +17,8 @@ docs/_build/
 gromacs_mdp.html
 NCI*
 error_*.txt
+*config.json
 validation_*.txt
-cwl_dirs.txt
-yml_dirs.txt
-inference_rules.txt
-renaming_conventions.txt
 .hypothesis/
 .env
 node_modules/

docs/advanced.md

@@ -3,32 +3,38 @@
 ## Edge Inference Configuration
 
 ### Naming Conventions
-If `--inference_use_naming_conventions` is enabled, matches can be refined based on the naming conventions of the inputs and outputs in the CWL CommandLineTools. Specifically, first `input_` is removed from the input name and `output_` is removed from all output names. Then, the default renamings contained in `renaming_conventions.txt` (shown below) are iteratively applied to the input name (only), and then the modified input name and all of the output names are checked for equality.
+If `--inference_use_naming_conventions` is enabled, matches can be refined based on the naming conventions of the inputs and outputs in the CWL CommandLineTools. Specifically, first `input_` is removed from the input name and `output_` is removed from all output names. Then, the default renamings contained in `renaming_conventions` tag of `config.json` (shown below) are iteratively applied to the input name (only), and then the modified input name and all of the output names are checked for equality.
 
 ```
-# The biobb CWL files do not always use consistent naming
-# conventions, so we need to perform some renamings here.
-# Eventually, the CWL files themselves should be fixed.
-
-energy_     edr_
-structure_  tpr_
-traj_       trr_
+"renaming_conventions": [
+        [
+            "energy_",
+            "edr_"
+        ],
+        [
+            "structure_",
+            "tpr_"
+        ],
+        [
+            "traj_",
+            "trr_"
+        ]
+    ]
 ```
 
 If there is now a unique match, then great! If there are still multiple matches, it chooses the first (i.e. most recent) match. If there are now no matches, it ignores the naming conventions and chooses the first (i.e. most recent) match based on types and formats only. If there are still multiple matches, it again chooses the first (i.e. most recent) match. Note that there are cases (i.e. file format conversions) where using naming conventions may not yield the desired behavior, so again ***`users should always check that edge inference actually produces the intended DAG`***.
 
 ### Inference Rules
 
-Users can customize the inference algorithm using inference rules. The default inference rules stored in inference_rules.txt are shown below:
+Users can customize the inference algorithm using inference rules. The default inference rules stored in `inference_rules` tag of `config.json` are shown below:
 
 ```
-# Amber, gromacs (zipped 3880) topology
-edam:format_3881 continue
-edam:format_3987 continue
-
-# Amber, gromacs coordinates
-edam:format_3878 break
-edam:format_2033 break
+"inference_rules": {
+        "edam:format_3881": "continue",
+        "edam:format_3987": "continue",
+        "edam:format_3878": "break",
+        "edam:format_2033": "break"
+    }
 ```
 
 Currently, the only inference rule implemented is `break`, which stops the inference algorithm from considering any further outputs beyond the current output from matching the current input. (The current output is allowed, i.e. break is inclusive.) This is useful when the most recent output file is desired, but the inference algorithm for some reason doesn't match it and chooses a subsequent / earlier file. This can happen when converting from one file format, performing a workflow step, and converting back to the original format, where in some cases the inference algorithm may choose the original file, thus accidentally skipping the workflow step.
@@ -229,7 +235,7 @@ wic:
 
 ## Namespaces
 
-Namespaces can be used to distinguish two different tools / workflows with the same name from different sources. For example, suppose a collaborator has shared an alternative minimization protocol, which we have downloaded to `bar/min.yml`. We can use their protocol by adding the line `foo    bar/` to `yml_dirs.txt` and annotating the call site in `basic.yml` with `namespace: foo` as shown below.
+Namespaces can be used to distinguish two different tools / workflows with the same name from different sources. For example, suppose a collaborator has shared an alternative minimization protocol, which we have downloaded to `bar/min.yml`. We can use their protocol by adding the namespace tag `foo` to `search_paths_yml` tag of `config.json` and annotating the call site in `basic.yml` with `namespace: foo` as shown below.
 
 ```yaml
 ...

docs/tutorials/config_ci.json

@@ -1,4 +1,5 @@
-// NOTE: This file should be in one of the directories listed in yml_dirs.txt
+// NOTE: This file should be in one of the directories listed in 'search_paths_yml'
+// tag of config.json
 // (Technically comments are not allowed in JSON, but we are manually
 // stripping out all lines that start with // before parsing.)
 {

docs/userguide.md

@@ -10,24 +10,36 @@ See [overview](overview.md)
 
 Many software packages have a way of automatically discovering files which they can use. (examples: [pytest](https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery) [pylint](https://pylint.pycqa.org/en/latest/user_guide/usage/run.html))
 
-By default, wic will recursively search for tools / workflows within the directories (and subdirectories) listed in `~/wic/cwl_dirs.txt` and `~/wic/yml_dirs.txt`. The default cwl_dirs.txt is shown.
+By default, wic will recursively search for tools / workflows within the directories (and subdirectories) listed in the config file's json tags `search_paths_cwl` and `search_paths_yml`. The paths listed can be absolute or relative. The default `config.json` is shown.
 
 ***`We strongly recommend placing all repositories of tools / workflows in the same parent directory.`***
 
 (All your repos should be side-by-side in sibling directories, as shown.)
 
 ```
-# Namespace Directory
-global      ../workflow-inference-compiler/cwl_adapters/
-global      ../image-workflows/cwl_adapters/
-global      ../biobb_adapters/biobb_adapters/
-global      ../mm-workflows/cwl_adapters/
-gpu         ../mm-workflows/gpu/
-# foo       a/relative/path/
-# bar       /an/absolute/path/
+......
+"search_paths_cwl": {
+        "global": [
+            "../workflow-inference-compiler/cwl_adapters",
+            "../image-workflows/cwl_adapters",
+            "../biobb_adapters/biobb_adapters",
+            "../mm-workflows/cwl_adapters"
+        ],
+        "gpu": [
+            "../mm-workflows/gpu"
+        ]
+    },
+"search_paths_yml": {
+        "global": [
+            "./workflow-inference-compiler/docs/tutorials",
+            "../image-workflows/workflows",
+            "../mm-workflows/examples"
+        ]
+    }
+.....
 ```
 
-If you do not have these files in your `~/wic/` directory, they will be automatically created for you the first time you run wic. (Because of this, the first time you run wic you should be in the root directory of any one of your repos.) Then you can manually edit these files with additional sources of tools / workflows.
+If you do not specify config file using the command line argument `--config`, it will be automatically created for you the first time you run wic in `~/wic/global_config.json`. (Because of this, the first time you run wic you should be in the root directory of any one of your repos.) Then you can manually edit this file with additional sources of tools / workflows.
 
 To avoid dealing with relative file paths in YAML files, by default
 

pyproject.toml

@@ -128,14 +128,15 @@ version = {attr = "wic.__version__"}
 
 [tool.setuptools]
 package-dir = {"" = "src"}
-include-package-data = false
+include-package-data = true
 
 [tool.setuptools.packages.find]
 where = ["src"]
-namespaces = false
+namespaces = true
 
 [tool.setuptools.package-data]
 "*" = ["*.txt"]
+"wic" = ["*.json"]
 
 [tool.aliases]
 test = "pytest --workers 8"

src/wic/ast.py

@@ -93,7 +93,7 @@ def read_ast_from_disk(homedir: str,
             if paths_ns_i == {}:
                 wicdir = Path(homedir) / 'wic'
                 raise Exception(
-                    f'Error! namespace {plugin_ns} not found in yaml paths. Check {wicdir / "yml_dirs.txt"}')
+                    f"Error! namespace {plugin_ns} not found in yaml paths. Check 'search_paths_yml' in your config file")
             if stem not in paths_ns_i:
                 msg = f'Error! {stem} not found in namespace {plugin_ns} when attempting to read {step_id.stem}.yml'
                 if stem == 'in':

src/wic/cli.py

@@ -7,11 +7,13 @@
 parser = argparse.ArgumentParser(prog='main', description='Convert a high-level yaml workflow file to CWL.')
 parser.add_argument('--yaml', type=str, required=('--generate_schemas_only' not in sys.argv),
                     help='Yaml workflow file')
+parser.add_argument('--config_file', type=str, required=False, default=str(Path().home()/'wic'/'global_config.json'),
+                    help='User provided (JSON) config file')
 # version action exits the parser Ref : https://github.com/python/cpython/blob/1f515e8a109204f7399d85b7fd806135166422d9/Lib/argparse.py#L1167
 parser.add_argument('--version', action='version', version=__version__,
                     default='==SUPPRESS==', help='Current version of the Workflow Inference Compiler')
 parser.add_argument('--generate_schemas_only', default=False, action="store_true",
-                    help='Generate schemas for the files in ~/wic/cwl_dirs.txt and ~/wic/yml_dirs.txt')
+                    help='Generate schemas for the files in config.json (search_paths_yml and search_paths_cwl)')
 parser.add_argument('--homedir', type=str, required=False, default=str(Path().home()),
                     help='The users home directory. This is necessary because CWL clears environment variables (e.g. HOME)')
 # Change default to True for now. See comment in compiler.py

src/wic/config.json

@@ -0,0 +1,31 @@
+{
+    "search_paths_cwl": {
+        "global": [
+            "../workflow-inference-compiler/cwl_adapters",
+            "../image-workflows/cwl_adapters",
+            "../biobb_adapters/biobb_adapters",
+            "../mm-workflows/cwl_adapters"
+        ],
+        "gpu": [
+            "../mm-workflows/gpu"
+        ]
+    },
+    "search_paths_yml": {
+        "global": [
+            "../workflow-inference-compiler/docs/tutorials",
+            "../image-workflows/workflows",
+            "../mm-workflows/examples"
+        ]
+    },
+    "renaming_conventions": [
+        ["energy_", "edr_"],
+        ["structure_", "tpr_"],
+        ["traj_", "trr_"]
+    ],
+    "inference_rules": {
+        "edam:format_3881": "continue",
+        "edam:format_3987": "continue",
+        "edam:format_3878": "break",
+        "edam:format_2033": "break"
+    }
+}

src/wic/cwl_dirs.txt

@@ -258,14 +258,32 @@ def main() -> None:
     logfile.touch()
 
     args_vals = json.loads(args.config)
+    # In CWL all env variables are hidden by default so Path().home() doesn't work
+    # Also User may specify a different homedir
+    default_config_file = Path(args.homedir)/'wic'/'global_config.json'
+    global_config: Json = {}
+    if not Path(args.config_file).exists():
+        if Path(args.config_file) == default_config_file:
+            global_config = io.get_default_config()
+            # write the default config object to the 'global_config.json' file in user's ~/wic directory
+            # for user to inspect and or modify the config json file
+            io.write_config_to_disk(global_config, default_config_file)
+            print(f'default config file : {default_config_file} generated')
+        else:
+            print(f"Error user specified config file {args.config_file} doesn't exist")
+            sys.exit()
+    else:
+        # reading user specified config file only if it exists
+        # never overwrite user's config file or generate another file in user's non-default directory
+        # TODO : Validate the json inside 'read_config_from_disk' function
+        global_config = io.read_config_from_disk(Path(args.config_file))
 
-    tools_cwl = get_tools_cwl(args.homedir, quiet=args.quiet)
-    yml_paths = get_yml_paths(args.homedir)
+    tools_cwl = get_tools_cwl(global_config, quiet=args.quiet)
+    yml_paths = get_yml_paths(global_config)
 
     # Perform initialization via mutating global variables (This is not ideal)
-    wicdir = Path(args.homedir) / 'wic'
-    compiler.inference_rules = dict(io.read_lines_pairs(wicdir / 'inference_rules.txt'))
-    inference.renaming_conventions = io.read_lines_pairs(wicdir / 'renaming_conventions.txt')
+    compiler.inference_rules = global_config.get('inference_rules', {})
+    inference.renaming_conventions = global_config.get('renaming_conventions', [])
 
     # Generate schemas for validation
     yaml_stems = utils.flatten([list(p) for p in yml_paths.values()])