CLS/chplcheck: Support editor agnostic configuration files (#26978)

This PR expands the existing support for using ConfigArgParse to allow
for editor-agnostic configuration files. This allows users to specify
options for their project in an editor-agnostic way, and then allow
CLS/chplcheck to auto-pick them up.

This also integrates nicely with Mason, allowing users to specify
CLS/chplcheck options in a Mason.toml file. For example

```toml
[brick]
...

[tool.chplcheck]
disable-rule = ["UnusedLoopIndex"]

```

This PR also expands the options for chplcheck, allowing the following

```toml
...

[tool.chplcheck]
file = ["src/*.chpl"]
add-rules = ["lint/rules.py"]
disable-rule = ["IncorrectIndentation"]

```

This lets users specify all the options needed in 1 file, and then from
the project root running just `chplcheck` does the right thing

This PR also adds the -M/--module-dir options to CLS, allowing users to
add a few paths to the module search path instead of needing a full
.cls-commands file. This also includes adding an internal, developer
only, ability to set the root directory for standard modules.

- [x] paratest with/without gasnet

[Reviewed by @DanilaFe]

Author: jabraham17

compiler/main/driver.cpp

@@ -2346,6 +2346,7 @@ static void dynoConfigureContext(std::string chpl_module_path) {
 
   chpl::parsing::setupModuleSearchPaths(gContext,
                                         CHPL_HOME,
+                                        "", //moduleRoot
                                         fMinimalModules,
                                         CHPL_LOCALE_MODEL,
                                         fEnableTaskTracking,

doc/rst/tools/chpl-language-server/chpl-language-server.rst

@@ -141,6 +141,35 @@ language server with linting enabled various ``chplcheck`` flags:
      --chplcheck-disable-rule UnusedLoopIndex \
      --chplcheck-add-rules path/to/my/myrules.py
 
+Configuration Files
+^^^^^^^^^^^^^^^^^^^
+
+``chpl-language-server`` can be configured without passing command line flags using a
+configuration file. This file can be specified using the ``--config`` (also
+``-c``) flag. The configuration file can either be a YAML file or a specific
+TOML file. For example, running ``chpl-language-server -c config.yaml`` will load the
+configuration from ``config.yaml``. Additionally, ``chpl-language-server`` will look for a
+configuration files in the current directory named ``chpl-language-server.cfg`` or
+``.chpl-language-server.cfg`` (in that order).
+
+Most command line options can be specified in the configuration file. For
+example, the following YAML configuration file will enable end of block markers for loops and declarations.
+
+.. code-block:: yaml
+
+   end-markers: "loop,decl"
+
+TOML configuration files can also be used, for example the following is the same configuration as above:
+
+.. code-block:: toml
+
+   [tool.chpl-language-server]
+   end-markers = ["loop", "decl"]
+
+This configuration can also be added to a :ref:`Mason <readme-mason>`
+configuration file. ``CLS`` will automatically use a configuration file
+contained in a ``Mason.toml`` file in the current directory.
+
 Configuring Chapel Projects
 ---------------------------
 

doc/rst/tools/chplcheck/chplcheck.rst

@@ -148,6 +148,39 @@ as at least 1 rule opts in to using them. For example, custom rules could confor
 to the setting ``MyIgnoreList``, which could be set as
 ``--setting MyIgnoreList="foo,bar"``.
 
+Configuration Files
+-------------------
+
+``chplcheck`` can be configured without passing command line flags using a
+configuration file. This file can be specified using the ``--config`` (also
+``-c``) flag. The configuration file can either be a YAML file or a specific
+TOML file. For example, running ``chplcheck -c config.yaml`` will load the
+configuration from ``config.yaml``. Additionally, ``chplcheck`` will look for a
+configuration files in the current directory named ``chplcheck.cfg`` or
+``.chplcheck.cfg`` (in that order).
+
+Most command line options can be specified in the configuration file. For
+example, the following YAML configuration file will explicitly enable the
+``UseExplicitModules`` rule and disable the ``UnusedLoopIndex`` and
+``UnusedFormal`` rules:
+
+.. code-block:: yaml
+
+   enable-rule: ["UseExplicitModules"]
+   disable-rule: ["UnusedLoopIndex", "UnusedFormal"]
+
+TOML configuration files can also be used, for example the following is the same configuration as above:
+
+.. code-block:: toml
+
+   [tool.chplcheck]
+   enable-rule = ["UseExplicitModules"]
+   disable-rule = ["UnusedLoopIndex", "UnusedFormal"]
+
+This configuration can also be added to a :ref:`Mason <readme-mason>`
+configuration file. ``chplcheck`` will automatically use a configuration file
+contained in a ``Mason.toml`` file in the current directory.
+
 Setting Up In Your Editor
 -------------------------
 

frontend/include/chpl/parsing/parsing-queries.h

@@ -299,6 +299,11 @@ void setBundledModulePath(Context* context, UniqueString path);
     chplSysModulesSubdir -- CHPL_SYS_MODULES_SUBDIR
     chplModulePath       -- CHPL_MODULE_PATH
 
+  The 'moduleRoot' argument allows overriding the default module root. If
+  'moduleRoot' is the empty string, then the default of 'CHPL_HOME/modules' is
+  used. Regardless, if 'minimalModules' is true '/minimal' is appended to the
+  'moduleRoot'.
+
   The arguments 'prependInternalModulePaths' and 'prependStandardModulePaths',
   if non-empty, allow one to override where the context will search for
   internal and standard modules, respectively. It will search each successive
@@ -312,6 +317,7 @@ void setBundledModulePath(Context* context, UniqueString path);
 void setupModuleSearchPaths(
                   Context* context,
                   const std::string& chplHome,
+                  const std::string& moduleRoot,
                   bool minimalModules,
                   const std::string& chplLocaleModel,
                   bool enableTaskTracking,
@@ -324,6 +330,18 @@ void setupModuleSearchPaths(
                   const std::vector<std::string>& cmdLinePaths,
                   const std::vector<std::string>& inputFilenames);
 
+/**
+  Overload of the more general setupModuleSearchPaths that uses the
+  context's stored chplHome and chplEnv to determine the values of most
+  arguments.
+*/
+void setupModuleSearchPaths(Context* context,
+                            const std::string& moduleRoot,
+                            bool minimalModules,
+                            bool enableTaskTracking,
+                            const std::vector<std::string>& cmdLinePaths,
+                            const std::vector<std::string>& inputFilenames);
+
 /**
   Overload of the more general setupModuleSearchPaths that uses the
   context's stored chplHome and chplEnv to determine the values of most

frontend/lib/parsing/parsing-queries.cpp

@@ -652,6 +652,7 @@ addCommandLineFileDirectories(std::vector<std::string>& searchPath,
 void setupModuleSearchPaths(
                   Context* context,
                   const std::string& chplHome,
+                  const std::string& moduleRoot,
                   bool minimalModules,
                   const std::string& chplLocaleModel,
                   bool enableTaskTracking,
@@ -668,10 +669,13 @@ void setupModuleSearchPaths(
       "setupModuleSearchPaths should be called before any queries are run");
 
   std::string modRoot;
-  if (!minimalModules) {
+  if (moduleRoot.empty()) {
     modRoot = chplHome + "/modules";
   } else {
-    modRoot = chplHome + "/modules/minimal";
+    modRoot = moduleRoot;
+  }
+  if (minimalModules) {
+    modRoot += "/minimal";
   }
 
   std::string internal = modRoot + "/internal";
@@ -756,6 +760,7 @@ void setupModuleSearchPaths(
 
 
 void setupModuleSearchPaths(Context* context,
+                            const std::string& moduleRoot,
                             bool minimalModules,
                             bool enableTaskTracking,
                             const std::vector<std::string>& cmdLinePaths,
@@ -770,6 +775,7 @@ void setupModuleSearchPaths(Context* context,
   auto chplModulePath = (it != chplEnv->end()) ? it->second : "";
   setupModuleSearchPaths(context,
                          chplHomeStr,
+                         moduleRoot,
                          minimalModules,
                          chplEnv->at("CHPL_LOCALE_MODEL"),
                          false,
@@ -783,6 +789,15 @@ void setupModuleSearchPaths(Context* context,
                          inputFilenames);
 }
 
+void setupModuleSearchPaths(Context* context,
+                            bool minimalModules,
+                            bool enableTaskTracking,
+                            const std::vector<std::string>& cmdLinePaths,
+                            const std::vector<std::string>& inputFilenames) {
+  setupModuleSearchPaths(context, "", minimalModules, enableTaskTracking,
+                                      cmdLinePaths, inputFilenames);
+}
+
 bool
 filePathIsInInternalModule(Context* context, UniqueString filePath) {
   // check for prepended internal module paths that may have come in from

test/chplcheck/configfile-toml.chpl

@@ -0,0 +1 @@
+configfile.chpl

test/chplcheck/configfile-toml.chplcheckopts

@@ -0,0 +1 @@
+--config configfile.toml

test/chplcheck/configfile-toml.good

@@ -0,0 +1,3 @@
+module configfile {
+  for i in 1..10 {}
+}

test/chplcheck/configfile.chpl

@@ -0,0 +1 @@
+-c configfile.yaml

test/chplcheck/configfile.chplcheckopts

@@ -0,0 +1,2 @@
+[tool.chplcheck]
+disable-rule = ["UnusedLoopIndex", "PascalCaseModules"]

test/chplcheck/configfile.good

@@ -0,0 +1 @@
+disable-rule: ["UnusedLoopIndex", "PascalCaseModules"]

test/chplcheck/configfile.toml

@@ -4,5 +4,7 @@ lsprotocol==2023.0.1
 pygls==1.3.1
 typeguard==4.3.0
 ConfigArgParse==1.7
+PyYAML==6.0.2 # required by ConfigArgParse
+toml==0.10.2 # required by ConfigArgParse
 # needed for <python3.11
 exceptiongroup==1.2.2

test/chplcheck/configfile.yaml

@@ -53,6 +53,16 @@ CLASS_BEGIN(Context)
          if (auto autoUseScope = resolution::scopeForAutoModule(node)) {
            std::ignore = resolution::resolveVisibilityStmts(node, autoUseScope, false);
          })
+  METHOD(Context, _set_module_paths, "Set the module path arguments to the given lists of module paths and filenames, using a potentially different module root",
+          void(std::string, std::vector<std::string>, std::vector<std::string>),
+
+          auto& modRoot = std::get<0>(args);
+          auto& paths = std::get<1>(args);
+          auto& filenames = std::get<2>(args);
+          parsing::setupModuleSearchPaths(node, modRoot, false, false, paths, filenames);
+          if (auto autoUseScope = resolution::scopeForAutoModule(node)) {
+            std::ignore = resolution::resolveVisibilityStmts(node, autoUseScope, false);
+          })
   METHOD(Context, is_bundled_path, "Check if the given file path is within the bundled (built-in) Chapel files",
          bool(chpl::UniqueString),
 

third-party/chpl-venv/chapel-py-requirements.txt

@@ -42,6 +42,7 @@
 import re
 import sys
 import importlib.util
+import copy
 
 import chapel
 from chapel.lsp import location_to_range, error_to_diagnostic
@@ -140,7 +141,6 @@
     CallHierarchyIncomingCall,
 )
 
-import argparse
 import configargparse
 import sys
 import functools
@@ -671,9 +671,16 @@ def all(cls) -> Dict[str, "EndMarkerPattern"]:
 
 
 class ContextContainer:
-    def __init__(self, file: str, config: Optional["WorkspaceConfig"]):
+    def __init__(
+        self,
+        file: str,
+        cls_config: "CLSConfig",
+        config: Optional["WorkspaceConfig"],
+    ):
+        self.cls_config: CLSConfig = cls_config
         self.config: Optional["WorkspaceConfig"] = config
         self.file_paths: List[str] = []
+        self.std_module_root: str = ""
         self.module_paths: List[str] = [os.path.dirname(os.path.abspath(file))]
         self.context: chapel.Context = chapel.Context()
         self.file_infos: List["FileInfo"] = []
@@ -687,7 +694,12 @@ def __init__(self, file: str, config: Optional["WorkspaceConfig"]):
                 self.module_paths = file_config["module_dirs"]
                 self.file_paths = file_config["files"]
 
-        self.context.set_module_paths(self.module_paths, self.file_paths)
+        self.std_module_root = self.cls_config.get("std_module_root")
+        self.module_paths.extend(self.cls_config.get("module_dir"))
+
+        self.context._set_module_paths(
+            self.std_module_root, self.module_paths, self.file_paths
+        )
 
     def register_signature(self, sig: chapel.TypedSignature) -> str:
         """
@@ -734,7 +746,9 @@ def advance(self) -> List[Any]:
         """
 
         self.context.advance_to_next_revision(False)
-        self.context.set_module_paths(self.module_paths, self.file_paths)
+        self.context._set_module_paths(
+            self.std_module_root, self.module_paths, self.file_paths
+        )
 
         with self.context.track_errors() as errors:
             for fi in self.file_infos:
@@ -1365,8 +1379,20 @@ def __init__(self):
 
     def _construct_parser(self):
         self.parser = configargparse.ArgParser(
-            default_config_files=[],  # Empty for now because cwd() is odd with VSCode etc.
-            config_file_parser_class=configargparse.YAMLConfigFileParser,
+            default_config_files=[
+                os.path.join(os.getcwd(), "chpl-language-server.cfg"),
+                os.path.join(os.getcwd(), ".chpl-language-server.cfg"),
+                os.path.join(os.getcwd(), "Mason.toml"),
+            ],
+            config_file_parser_class=configargparse.CompositeConfigParser(
+                [
+                    configargparse.YAMLConfigFileParser,
+                    configargparse.TomlConfigParser(
+                        ["tool.chpl-language-server"]
+                    ),
+                ]
+            ),
+            args_for_setting_config_path=["--config", "-c"],
         )
 
         def add_bool_flag(name: str, dest: str, default: bool):
@@ -1379,6 +1405,12 @@ def add_bool_flag(name: str, dest: str, default: bool):
             self.parser.set_defaults(**{dest: default})
 
         add_bool_flag("resolver", "resolver", False)
+        self.parser.add_argument(
+            "--std-module-root", default="", help=configargparse.SUPPRESS
+        )
+        self.parser.add_argument(
+            "--module-dir", "-M", action="append", default=[]
+        )
         add_bool_flag("type-inlays", "type_inlays", True)
         add_bool_flag("param-inlays", "param_inlays", True)
         add_bool_flag("literal-arg-inlays", "literal_arg_inlays", True)
@@ -1401,25 +1433,25 @@ def _validate_end_markers(self):
         valid_choices = ["all", "none"] + list(EndMarkerPattern.all().keys())
         for marker in self.args["end_markers"]:
             if marker not in valid_choices:
-                raise argparse.ArgumentError(
+                raise configargparse.ArgumentError(
                     None, f"Invalid end marker choice: {marker}"
                 )
         n_markers = len(self.args["end_markers"])
         if n_markers != len(set(self.args["end_markers"])):
-            raise argparse.ArgumentError(
+            raise configargparse.ArgumentError(
                 None, "Cannot specify the same end marker multiple times"
             )
         if "none" in self.args["end_markers"] and n_markers > 1:
-            raise argparse.ArgumentError(
+            raise configargparse.ArgumentError(
                 None, "Cannot specify 'none' with other end marker choices"
             )
         if "all" in self.args["end_markers"] and n_markers > 1:
-            raise argparse.ArgumentError(
+            raise configargparse.ArgumentError(
                 None, "Cannot specify 'all' with other end marker choices"
             )
 
     def parse_args(self):
-        self.args = vars(self.parser.parse_args())
+        self.args = copy.deepcopy(vars(self.parser.parse_args()))
         self._parse_end_markers()
         self._validate_end_markers()
 
@@ -1431,6 +1463,7 @@ class ChapelLanguageServer(LanguageServer):
     def __init__(self, config: CLSConfig):
         super().__init__("chpl-language-server", "v0.1")
 
+        self.config: CLSConfig = config
         self.contexts: Dict[str, ContextContainer] = {}
         self.context_ids: Dict[ContextContainer, str] = {}
         self.context_id_counter = 0
@@ -1530,7 +1563,7 @@ def get_context(self, uri: str) -> ContextContainer:
         if path in self.contexts:
             return self.contexts[path]
 
-        context = ContextContainer(path, workspace_config)
+        context = ContextContainer(path, self.config, workspace_config)
         for file in context.file_paths:
             self.contexts[file] = context
         self.contexts[path] = context