Merge pull request #7 from bcdev/forman-important_features

Important features

Author: forman

.github/workflows/publish.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
 
     steps:
     - uses: actions/checkout@v3

.github/workflows/tests.yml

@@ -17,7 +17,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
 
     steps:
     - uses: actions/checkout@v3

.gitignore

@@ -1,7 +1,5 @@
 # xrlint
-xrlint.config.py
-xrlint.config.yaml
-xrlint.config.json
+/xrlint_config.*
 
 # Logs
 *.log

CHANGES.md

@@ -2,14 +2,24 @@
 ## Early development snapshots
 
 - Version 0.0.2 (in development) 
-   - more rules
-   - more tests
-   - config list item can be config name
-   - xcube is no longer default plugin
-   - no rules by default
-   - no rules configured reports error
-   - CLI fails with max warnings exceeded 
-   - CLI fails with no files given
-   - new CLI options `--plugin` and `--rule`
-
+  - more rules
+  - more tests
+  - config list item can be config name
+  - xcube is no longer default plugin
+  - no rules by default
+  - no rules configured is an error
+  - CLI exit code 1 with max warnings exceeded 
+  - CLI exit code 1 with no files given
+  - new CLI options `--plugin`, `--rule`, `--init`
+  - markdown rule reference in docs via `mkruleref.py` tool
+  - using `files` config option to specify valid filename extensions, see
+    [here](https://eslint.org/docs/latest/use/configure/configuration-files#specifying-files-with-arbitrary-extensions)
+  - using default filename extensions `["**/*.zarr", "**/*.nc"]`
+  - using subset of [minimatch](https://github.com/isaacs/minimatch) 
+    syntax instead of the simple globs used by Python's `fnmatch`
+  - allow the CLI's `FILES` args to contain directories, which are 
+    automatically recursively traversed
+  - rule `coords-for-dims` works on dataset level
+  - support for custom processors
+  
 - Version 0.0.1 (30.12.2024) - Initial version. 

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2024 Brockmann Consult Development
+Copyright (c) 2025 Brockmann Consult Development
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

docs/about.md

@@ -27,13 +27,18 @@ corresponding issue.
 
 ## Development
 
-To set up development environment, with repository root as current
-working directory:
+To install the XRLint development environment into an existing Python environment
 
 ```bash
 pip install .[dev,doc]
 ```
 
+or create a new environment using
+
+```bash
+mamba env create 
+```
+
 ### Testing and Coverage
 
 XRLint uses [pytest](https://docs.pytest.org/) for unit-level testing 
@@ -45,10 +50,12 @@ pytest --cov=xrlint --cov-report html
 
 ### Code Style
 
-XRLint source code is formatted using the [black](https://black.readthedocs.io/) tool.
+XRLint source code is formatted using the [black](https://black.readthedocs.io/) tool and 
+quality-controlled using [flake8](https://flake8.pycqa.org/).
 
 ```bash
 black .
+flake8 --max-line-length=88 . 
 ```
 
 ### Documentation
@@ -65,6 +72,13 @@ mkdocs serve
 mkdocs gh-deploy
 ```
 
+The rule reference page is generated by a script called `mkruleref.py`.
+After changing or adding a rule, make sure you recreate the page:
+
+```bash
+python -m mkruleref
+```
+
 ## License
 
 XRLint is open source made available under the terms and conditions of the 

docs/api.md

@@ -18,6 +18,26 @@ All described objects can be imported from the `xrlint.all` module.
 
 ::: xrlint.config.ConfigList
 
+## Class `Plugin`
+
+::: xrlint.plugin.Plugin
+
+## Class `PluginMeta`
+
+::: xrlint.plugin.PluginMeta
+
+## Class `Processor`
+
+::: xrlint.processor.Processor
+
+## Class `ProcessorMeta`
+
+::: xrlint.processor.ProcessorMeta
+
+## Class `ProcessorOp`
+
+::: xrlint.processor.ProcessorOp
+
 ## Class `RuleConfig`
 
 ::: xrlint.rule.RuleConfig
@@ -38,6 +58,14 @@ All described objects can be imported from the `xrlint.all` module.
 
 ::: xrlint.rule.RuleContext
 
+## Class `RuleTester`
+
+::: xrlint.testing.RuleTester
+
+## Class `RuleTest`
+
+::: xrlint.testing.RuleTest
+
 ## Class `Result`
 
 ::: xrlint.result.Result

docs/cli.md

@@ -17,13 +17,18 @@ Usage: xrlint [OPTIONS] [FILES]...
 
 Options:
   --no-default-config     Disable use of default configuration from
-                          xrlint.config.*
-  -c, --config PATH       Use this configuration, overriding xrlint.config.*
+                          xrlint_config.*
+  -c, --config PATH       Use this configuration, overriding xrlint_config.*
                           config options if present
+  --plugin MODULE         Specify plugins. MODULE is the name of Python module
+                          that defines an 'export_plugin()' function.
+  --rule SPEC             Specify rules. SPEC must have format '<rule-name>:
+                          <rule-config>' (note the space character).
   -f, --format NAME       Use a specific output format - default: simple
   -o, --output-file PATH  Specify file to write report to
   --max-warnings COUNT    Number of warnings to trigger nonzero exit code -
-                          default: -1
+                          default: 5
+  --init                  Write initial configuration file and exit.
   --version               Show the version and exit.
   --help                  Show this message and exit.
 

docs/config.md

@@ -0,0 +1,92 @@
+# Configure XRLint
+
+_**Note**: this chapter's material is based on the documentation of how to [configure ESLint](https://eslint.org/docs/latest/use/configure/).
+Many parts have been copied and adjusted as it applies in many similar ways to XRLint._ 
+
+## Configuration File
+
+The XRLint configuration file may be named any of the following:
+
+* `xrlint_config.yaml`
+* `xrlint_config.json`
+* `xrlint_config.py`
+
+It should be placed in the root directory of your project and export 
+an array of [configuration objects](#configuration-objects) or 
+references to [predefined configuration objects](#predefined-configuration-objects). 
+
+Here’s a YAML example:
+
+```yaml
+- files: ["**/*.zarr", "**/*.nc"]
+- plugins:
+    xcube: xrlint.plugins.xcube
+- recommended
+- xcube/recommended
+```
+
+Same using JSON:
+
+```json
+[
+  {"files": ["**/*.zarr", "**/*.nc"]},
+  {
+    "plugins": {
+      "xcube": "xrlint.plugins.xcube"
+    }
+  },
+  "recommended",
+  "xcube/recommended"
+]
+```
+
+And as Python script:
+
+```python
+def export_configs():
+    return [
+      {"files": ["**/*.zarr", "**/*.nc"]},
+      {
+        "plugins": {
+          "xcube": "xrlint.plugins.xcube"
+        }
+      },
+      "recommended",
+      "xcube/recommended"
+    ]
+```
+
+
+## Configuration Objects
+
+Each configuration object contains all of the information XRLint needs 
+to execute on a set of files. Each configuration object is made up of 
+these properties:
+
+* `name` - A name for the configuration object. 
+  This is used in error messages and config inspector to help identify which 
+  configuration object is being used.
+* `files` - A list of glob patterns indicating the files that the 
+  configuration object should apply to. If not specified, the configuration 
+  object applies to all files matched by any other configuration object.
+* `ignores` - A list of glob patterns indicating the files that the 
+  configuration object should not apply to. If not specified, the configuration 
+  object applies to all files matched by files. If ignores is used without any 
+  other keys in the configuration object, then the patterns act as _global ignores_.
+* `opener_options` - A dictionary specifying keyword-arguments that are passed 
+  directly to the `xarray.open_dataset()` function. The available options are 
+  dependent on the xarray backend selected by the `engine` option.
+* `linter_options` - A dictionary containing settings related to 
+  the linting process. (Currently not used.)
+* `processor` - A string indicating the name of a processor inside of a plugin, 
+  i.e., `"<plugin-name>/<processor-name>"`. In Python configurations 
+  it can also be an object of type `ProcessorOp` containing
+  `preprocess()` and `postprocess()` methods.
+* `plugins` - A dictionary containing a name-value mapping of plugin names 
+  to either plugin module names or `Plugin` objects. When `files` is specified, 
+  these plugins are only available to the matching files.
+* `rules` - An object containing the configured rules. 
+  When `files` or `ignores` are specified, these rule configurations are only 
+  available to the matching files.
+* `settings` - An object containing name-value pairs of information that should 
+  be available to all rules.

docs/rule-ref.md

@@ -0,0 +1,66 @@
+# Rule Reference
+
+## Core Rules
+
+### :material-bug: `coords-for-dims`
+
+Dimensions of data variables should have corresponding coordinates.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+
+### :material-lightbulb: `dataset-title-attr`
+
+Datasets should be given a non-empty title.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-alert:
+
+### :material-bug: `grid-mappings`
+
+Grid mappings, if any, shall have valid grid mapping coordinate variables.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+
+### :material-lightbulb: `no-empty-attrs`
+
+Every dataset element should have metadata that describes it.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-alert:
+
+### :material-lightbulb: `var-units-attr`
+
+Every variable should have a valid 'units' attribute.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-alert:
+
+## xcube Rules
+
+### :material-bug: `any-spatial-data-var`
+
+A datacube should have spatial data variables.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+
+### :material-bug: `cube-dims-order`
+
+Order of dimensions in spatio-temporal datacube variables should be [time, ..., y, x].
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+
+### :material-lightbulb: `grid-mapping-naming`
+
+Grid mapping variables should be called 'spatial_ref' or 'crs' for compatibility with rioxarray and other packages.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-alert:
+
+### :material-bug: `lat-lon-naming`
+
+Latitude and longitude coordinates and dimensions should be called 'lat' and 'lon'.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+
+### :material-bug: `single-grid-mapping`
+
+A single grid mapping shall be used for all spatial data variables of a datacube.
+
+Contained in:  `all`-:material-lightning-bolt: `recommended`-:material-lightning-bolt:
+