Merge pull request PolusAI#200 from vjaganat90/gconf2

coalesce different txt based config files into one json

Author: sameeul

.gitignore

@@ -17,11 +17,8 @@ docs/_build/
 gromacs_mdp.html
 NCI*
 error_*.txt
+*config.json
 validation_*.txt
-cwl_dirs.txt
-yml_dirs.txt
-inference_rules.txt
-renaming_conventions.txt
 .hypothesis/
 .env
 node_modules/

docs/advanced.md

@@ -3,32 +3,38 @@
 ## Edge Inference Configuration
 
 ### Naming Conventions
-If `--inference_use_naming_conventions` is enabled, matches can be refined based on the naming conventions of the inputs and outputs in the CWL CommandLineTools. Specifically, first `input_` is removed from the input name and `output_` is removed from all output names. Then, the default renamings contained in `renaming_conventions.txt` (shown below) are iteratively applied to the input name (only), and then the modified input name and all of the output names are checked for equality.
+If `--inference_use_naming_conventions` is enabled, matches can be refined based on the naming conventions of the inputs and outputs in the CWL CommandLineTools. Specifically, first `input_` is removed from the input name and `output_` is removed from all output names. Then, the default renamings contained in `renaming_conventions` tag of `config.json` (shown below) are iteratively applied to the input name (only), and then the modified input name and all of the output names are checked for equality.
 
 ```
-# The biobb CWL files do not always use consistent naming
-# conventions, so we need to perform some renamings here.
-# Eventually, the CWL files themselves should be fixed.
-
-energy_     edr_
-structure_  tpr_
-traj_       trr_
+"renaming_conventions": [
+        [
+            "energy_",
+            "edr_"
+        ],
+        [
+            "structure_",
+            "tpr_"
+        ],
+        [
+            "traj_",
+            "trr_"
+        ]
+    ]
 ```
 
 If there is now a unique match, then great! If there are still multiple matches, it chooses the first (i.e. most recent) match. If there are now no matches, it ignores the naming conventions and chooses the first (i.e. most recent) match based on types and formats only. If there are still multiple matches, it again chooses the first (i.e. most recent) match. Note that there are cases (i.e. file format conversions) where using naming conventions may not yield the desired behavior, so again ***`users should always check that edge inference actually produces the intended DAG`***.
 
 ### Inference Rules
 
-Users can customize the inference algorithm using inference rules. The default inference rules stored in inference_rules.txt are shown below:
+Users can customize the inference algorithm using inference rules. The default inference rules stored in `inference_rules` tag of `config.json` are shown below:
 
 ```
-# Amber, gromacs (zipped 3880) topology
-edam:format_3881 continue
-edam:format_3987 continue
-
-# Amber, gromacs coordinates
-edam:format_3878 break
-edam:format_2033 break
+"inference_rules": {
+        "edam:format_3881": "continue",
+        "edam:format_3987": "continue",
+        "edam:format_3878": "break",
+        "edam:format_2033": "break"
+    }
 ```
 
 Currently, the only inference rule implemented is `break`, which stops the inference algorithm from considering any further outputs beyond the current output from matching the current input. (The current output is allowed, i.e. break is inclusive.) This is useful when the most recent output file is desired, but the inference algorithm for some reason doesn't match it and chooses a subsequent / earlier file. This can happen when converting from one file format, performing a workflow step, and converting back to the original format, where in some cases the inference algorithm may choose the original file, thus accidentally skipping the workflow step.
@@ -229,7 +235,7 @@ wic:
 
 ## Namespaces
 
-Namespaces can be used to distinguish two different tools / workflows with the same name from different sources. For example, suppose a collaborator has shared an alternative minimization protocol, which we have downloaded to `bar/min.yml`. We can use their protocol by adding the line `foo    bar/` to `yml_dirs.txt` and annotating the call site in `basic.yml` with `namespace: foo` as shown below.
+Namespaces can be used to distinguish two different tools / workflows with the same name from different sources. For example, suppose a collaborator has shared an alternative minimization protocol, which we have downloaded to `bar/min.yml`. We can use their protocol by adding the namespace tag `foo` to `search_paths_yml` tag of `config.json` and annotating the call site in `basic.yml` with `namespace: foo` as shown below.
 
 ```yaml
 ...

docs/tutorials/config_ci.json

@@ -1,4 +1,5 @@
-// NOTE: This file should be in one of the directories listed in yml_dirs.txt
+// NOTE: This file should be in one of the directories listed in 'search_paths_yml'
+// tag of config.json
 // (Technically comments are not allowed in JSON, but we are manually
 // stripping out all lines that start with // before parsing.)
 {

docs/userguide.md

@@ -10,24 +10,36 @@ See [overview](overview.md)
 
 Many software packages have a way of automatically discovering files which they can use. (examples: [pytest](https://docs.pytest.org/en/latest/explanation/goodpractices.html#conventions-for-python-test-discovery) [pylint](https://pylint.pycqa.org/en/latest/user_guide/usage/run.html))
 
-By default, wic will recursively search for tools / workflows within the directories (and subdirectories) listed in `~/wic/cwl_dirs.txt` and `~/wic/yml_dirs.txt`. The default cwl_dirs.txt is shown.
+By default, wic will recursively search for tools / workflows within the directories (and subdirectories) listed in the config file's json tags `search_paths_cwl` and `search_paths_yml`. The paths listed can be absolute or relative. The default `config.json` is shown.
 
 ***`We strongly recommend placing all repositories of tools / workflows in the same parent directory.`***
 
 (All your repos should be side-by-side in sibling directories, as shown.)
 
 ```
-# Namespace Directory
-global      ../workflow-inference-compiler/cwl_adapters/
-global      ../image-workflows/cwl_adapters/
-global      ../biobb_adapters/biobb_adapters/
-global      ../mm-workflows/cwl_adapters/
-gpu         ../mm-workflows/gpu/
-# foo       a/relative/path/
-# bar       /an/absolute/path/
+......
+"search_paths_cwl": {
+        "global": [
+            "../workflow-inference-compiler/cwl_adapters",
+            "../image-workflows/cwl_adapters",
+            "../biobb_adapters/biobb_adapters",
+            "../mm-workflows/cwl_adapters"
+        ],
+        "gpu": [
+            "../mm-workflows/gpu"
+        ]
+    },
+"search_paths_yml": {
+        "global": [
+            "./workflow-inference-compiler/docs/tutorials",
+            "../image-workflows/workflows",
+            "../mm-workflows/examples"
+        ]
+    }
+.....
 ```
 
-If you do not have these files in your `~/wic/` directory, they will be automatically created for you the first time you run wic. (Because of this, the first time you run wic you should be in the root directory of any one of your repos.) Then you can manually edit these files with additional sources of tools / workflows.
+If you do not specify config file using the command line argument `--config`, it will be automatically created for you the first time you run wic in `~/wic/global_config.json`. (Because of this, the first time you run wic you should be in the root directory of any one of your repos.) Then you can manually edit this file with additional sources of tools / workflows.
 
 To avoid dealing with relative file paths in YAML files, by default
 

pyproject.toml

@@ -128,14 +128,15 @@ version = {attr = "wic.__version__"}
 
 [tool.setuptools]
 package-dir = {"" = "src"}
-include-package-data = false
+include-package-data = true
 
 [tool.setuptools.packages.find]
 where = ["src"]
-namespaces = false
+namespaces = true
 
 [tool.setuptools.package-data]
 "*" = ["*.txt"]
+"wic" = ["*.json"]
 
 [tool.aliases]
 test = "pytest --workers 8"

src/wic/ast.py

@@ -93,7 +93,7 @@ def read_ast_from_disk(homedir: str,
             if paths_ns_i == {}:
                 wicdir = Path(homedir) / 'wic'
                 raise Exception(
-                    f'Error! namespace {plugin_ns} not found in yaml paths. Check {wicdir / "yml_dirs.txt"}')
+                    f"Error! namespace {plugin_ns} not found in yaml paths. Check 'search_paths_yml' in your config file")
             if stem not in paths_ns_i:
                 msg = f'Error! {stem} not found in namespace {plugin_ns} when attempting to read {step_id.stem}.yml'
                 if stem == 'in':

src/wic/cli.py

@@ -7,11 +7,13 @@
 parser = argparse.ArgumentParser(prog='main', description='Convert a high-level yaml workflow file to CWL.')
 parser.add_argument('--yaml', type=str, required=('--generate_schemas_only' not in sys.argv),
                     help='Yaml workflow file')
+parser.add_argument('--config_file', type=str, required=False, default=str(Path().home()/'wic'/'global_config.json'),
+                    help='User provided (JSON) config file')
 # version action exits the parser Ref : https://github.com/python/cpython/blob/1f515e8a109204f7399d85b7fd806135166422d9/Lib/argparse.py#L1167
 parser.add_argument('--version', action='version', version=__version__,
                     default='==SUPPRESS==', help='Current version of the Workflow Inference Compiler')
 parser.add_argument('--generate_schemas_only', default=False, action="store_true",
-                    help='Generate schemas for the files in ~/wic/cwl_dirs.txt and ~/wic/yml_dirs.txt')
+                    help='Generate schemas for the files in config.json (search_paths_yml and search_paths_cwl)')
 parser.add_argument('--homedir', type=str, required=False, default=str(Path().home()),
                     help='The users home directory. This is necessary because CWL clears environment variables (e.g. HOME)')
 # Change default to True for now. See comment in compiler.py

src/wic/config.json

@@ -0,0 +1,31 @@
+{
+    "search_paths_cwl": {
+        "global": [
+            "../workflow-inference-compiler/cwl_adapters",
+            "../image-workflows/cwl_adapters",
+            "../biobb_adapters/biobb_adapters",
+            "../mm-workflows/cwl_adapters"
+        ],
+        "gpu": [
+            "../mm-workflows/gpu"
+        ]
+    },
+    "search_paths_yml": {
+        "global": [
+            "../workflow-inference-compiler/docs/tutorials",
+            "../image-workflows/workflows",
+            "../mm-workflows/examples"
+        ]
+    },
+    "renaming_conventions": [
+        ["energy_", "edr_"],
+        ["structure_", "tpr_"],
+        ["traj_", "trr_"]
+    ],
+    "inference_rules": {
+        "edam:format_3881": "continue",
+        "edam:format_3987": "continue",
+        "edam:format_3878": "break",
+        "edam:format_2033": "break"
+    }
+}

src/wic/cwl_dirs.txt

@@ -258,14 +258,32 @@ def main() -> None:
     logfile.touch()
 
     args_vals = json.loads(args.config)
+    # In CWL all env variables are hidden by default so Path().home() doesn't work
+    # Also User may specify a different homedir
+    default_config_file = Path(args.homedir)/'wic'/'global_config.json'
+    global_config: Json = {}
+    if not Path(args.config_file).exists():
+        if Path(args.config_file) == default_config_file:
+            global_config = io.get_default_config()
+            # write the default config object to the 'global_config.json' file in user's ~/wic directory
+            # for user to inspect and or modify the config json file
+            io.write_config_to_disk(global_config, default_config_file)
+            print(f'default config file : {default_config_file} generated')
+        else:
+            print(f"Error user specified config file {args.config_file} doesn't exist")
+            sys.exit()
+    else:
+        # reading user specified config file only if it exists
+        # never overwrite user's config file or generate another file in user's non-default directory
+        # TODO : Validate the json inside 'read_config_from_disk' function
+        global_config = io.read_config_from_disk(Path(args.config_file))
 
-    tools_cwl = get_tools_cwl(args.homedir, quiet=args.quiet)
-    yml_paths = get_yml_paths(args.homedir)
+    tools_cwl = get_tools_cwl(global_config, quiet=args.quiet)
+    yml_paths = get_yml_paths(global_config)
 
     # Perform initialization via mutating global variables (This is not ideal)
-    wicdir = Path(args.homedir) / 'wic'
-    compiler.inference_rules = dict(io.read_lines_pairs(wicdir / 'inference_rules.txt'))
-    inference.renaming_conventions = io.read_lines_pairs(wicdir / 'renaming_conventions.txt')
+    compiler.inference_rules = global_config.get('inference_rules', {})
+    inference.renaming_conventions = global_config.get('renaming_conventions', [])
 
     # Generate schemas for validation
     yaml_stems = utils.flatten([list(p) for p in yml_paths.values()])

src/wic/cwl_watcher.py

@@ -1,14 +1,13 @@
 import argparse
-import logging
+import copy
 import json
 from pathlib import Path
-import subprocess as sub
 from typing import Any, List, Tuple
 
 import yaml
 
 from . import auto_gen_header
-from .wic_types import (Namespaces, NodeData, RoseTree, Yaml, ExplicitEdgeCalls)
+from .wic_types import (Namespaces, NodeData, RoseTree, Yaml, ExplicitEdgeCalls, Json)
 
 
 def read_lines_pairs(filename: Path) -> List[Tuple[str, str]]:
@@ -112,43 +111,69 @@ def write_to_disk(rose_tree: RoseTree, path: Path, relative_run_path: bool) -> N
         write_to_disk(sub_rose_tree, subpath, relative_run_path)
 
 
-logger_wicad = logging.getLogger("wicautodiscovery")
+def write_config_to_disk(config: Json, config_file: Path) -> None:
+    """Writes config json object to config_file
 
+    Args:
+        config (Json): The json object that is to be written to disk
+        config_file (Path): The file path where it is to be written
+    """
+    config_dir = Path(config_file).parent
+    # make the full path if it doesn't exist
+    config_dir.mkdir(parents=True, exist_ok=True)
+    with open(config_file, 'w', encoding='utf-8') as f:
+        json.dump(config, f)
 
-def copy_config_files(homedir: str) -> None:
-    """Copies the following configuration files to ~/wic/\n
-    cwl_dirs.txt, yml_dirs.txt, renaming_conventions.txt, inference_rules.txt
+
+def read_config_from_disk(config_file: Path) -> Json:
+    """Returns the config json object from config_file with absolute paths
 
     Args:
-        homedir (str): The users home directory
+        config_file (Path): The path of json file where it is to be read from
+
+    Returns:
+        Json: The config json object with absolute filepaths
     """
-    files = ['cwl_dirs.txt', 'yml_dirs.txt', 'renaming_conventions.txt', 'inference_rules.txt']
-    src_dir = Path(__file__).parent
-    wicdir = Path(homedir) / 'wic'
-    wicdir.mkdir(exist_ok=True)
+    config: Json = {}
+    # config_file can contain absolute or relative paths
+    with open(config_file, 'r', encoding='utf-8') as f:
+        config = json.load(f)
+    conf_tags = ['search_paths_cwl', 'search_paths_yml']
+    for tag in conf_tags:
+        config[tag] = get_absolute_paths(config[tag])
+    return config
 
-    for file in files:
-        if not (wicdir / file).exists():
-            logger_wicad.warning(f'Writing {str(wicdir / file)}')
-            logger_wicad.warning('Please check this file and make sure that the paths in it are correct.')
-            cmd = ['cp', str(src_dir / file), str(wicdir / file)]
-            sub.run(cmd, check=True)
 
-    write_absolute_config_files(wicdir / 'cwl_dirs.txt')
-    write_absolute_config_files(wicdir / 'yml_dirs.txt')
+def get_default_config() -> Json:
+    """Returns the default config with absolute paths
+
+    Returns:
+        Json: The config json object with absolute filepaths
+    """
+    src_dir = Path(__file__).parent
+    conf_tags = ['search_paths_cwl', 'search_paths_yml']
+    default_config: Json = {}
+    # config.json can contain absolute or relative paths
+    default_config = read_config_from_disk(src_dir/'config.json')
+    for tag in conf_tags:
+        default_config[tag] = get_absolute_paths(default_config[tag])
+    return default_config
 
 
-def write_absolute_config_files(dirs_file: Path) -> None:
-    """Makes the paths within the \*_dirs.txt files absolute
+def get_absolute_paths(sub_config: Json) -> Json:
+    """Makes the paths within the dirs_file file absolute and write them into sub_config object.
 
     Args:
-        dirs_file (Path): The path to the \*_dirs.txt file
-        dirs_file_abs (str): The path to the absolute \*_dirs.txt file
+        sub_config (dict): The json (sub)object where filepaths are stored
+
+    Returns:
+        Json: The json (sub)object with absolute filepaths
     """
-    ns_paths = read_lines_pairs(dirs_file)
-    pairs_abs = [ns + ' ' + str(Path(path).absolute()) for ns, path in ns_paths]
-    with open(dirs_file, mode='w', encoding='utf-8') as f:
-        f.write('\n'.join(pairs_abs))
+    abs_sub_config = copy.deepcopy(sub_config)
+    for ns in abs_sub_config:
+        abs_paths = [str(Path(path).absolute()) for path in abs_sub_config[ns]]
+        abs_sub_config[ns] = abs_paths
+    return abs_sub_config
 
 
 def write_absolute_yaml_tags(args: argparse.Namespace, in_dict_in: Yaml, namespaces: Namespaces,
@@ -178,6 +203,3 @@ def write_absolute_yaml_tags(args: argparse.Namespace, in_dict_in: Yaml, namespa
     for arg_key_ in arg_keys_:
         in_name_ = f'{step_name_i}___{arg_key_}'  # {step_name_i}_input___{arg_key}
         explicit_edge_calls_copy.update({in_name_: (namespaces + [step_name_i], arg_key_)})
-
-    write_absolute_config_files(Path(args.homedir) / 'wic' / 'cwl_dirs.txt')
-    write_absolute_config_files(Path(args.homedir) / 'wic' / 'yml_dirs.txt')