Add JSON schema (#1426)

* Add JSON schema for openPMD

Written as .toml files for ease of documentation, maintailability and
readability.

* Add convert-json-toml tool

Needed for "compiling" the schema to JSON
Also add a Makefile to further simplify this

* Add script for checking openPMD file against the schema

Workflow documented in README.md

* Don't use spaces in SerialIOTest attribute names

The JSON schema verification package does not like that

* Fix bugs detected by this verifier

Both of the form "data not found in places where data was expected"

* Add GitHub workflow

Verify all JSON-openPMD files written by testing against the schema

* Shorthand attributes

* Add dataset template mode

* Fix path

* Fix reading from stdin

* toml11 4.0 compatibility

* Only check for existing Iterations in writeOnly mode

* Some additions to schema

1. Support UNDEFINED datasets in template mode
2. gridUnitSI may now be a vector

* Remove deprecated jsonschema.validators.RefResolver

Apparently it's better to make everything 100 times more complicated

* Use most recent version of jsonschema

* Allow empty variable-based series

* Use if-then-else for better-steered parsing

anyOf and oneOf now only used for trivial distinctions, this makes
schemas much more robust since errors can be caught early and error
messages become actually useful.

* hmm

* Remove json cfg after test

Otherwise CI thinks this is an openPMD file

* Update documentation, rename convert-toml-json tool

* Apply suggestions from code review

Co-authored-by: Axel Huebl <[email protected]>

* Add reference to openPMD-validator

* Update README.md

---------

Co-authored-by: Axel Huebl <[email protected]>

Author: franzpoeschel

.github/workflows/linux.yml

@@ -260,7 +260,7 @@ jobs:
     - name: Install
       run: |
         sudo apt-get update
-        sudo apt-get install g++ libopenmpi-dev libhdf5-openmpi-dev python3 python3-numpy python3-mpi4py python3-pandas python3-h5py-mpi
+        sudo apt-get install g++ libopenmpi-dev libhdf5-openmpi-dev python3 python3-numpy python3-mpi4py python3-pandas python3-h5py-mpi python3-pip
 #       TODO ADIOS2
     - name: Build
       env: {CXXFLAGS: -Werror, PKG_CONFIG_PATH: /usr/lib/x86_64-linux-gnu/pkgconfig}
@@ -275,6 +275,22 @@ jobs:
         cmake --build build --parallel 4
         ctest --test-dir build --output-on-failure
 
+        python3 -m pip install jsonschema==4.* referencing
+        cd share/openPMD/json_schema
+        PATH="../../../build/bin:$PATH" make -j 2
+        # We need to exclude the thetaMode example since that has a different
+        # meshesPath and the JSON schema needs to hardcode that.
+        find ../../../build/samples/      \
+            ! -path '*thetaMode*'         \
+            ! -path '/*many_iterations/*' \
+            ! -name 'profiling.json'      \
+            ! -name '*config.json'        \
+            -iname '*.json'               \
+          | while read i; do
+            echo "Checking $i"
+            ./check.py "$i"
+        done
+
   musllinux_py10:
     runs-on: ubuntu-22.04
     if: github.event.pull_request.draft == false

CMakeLists.txt

@@ -691,11 +691,12 @@ set(openPMD_TEST_NAMES
 # command line tools
 set(openPMD_CLI_TOOL_NAMES
     ls
+    convert-toml-json
 )
 set(openPMD_PYTHON_CLI_TOOL_NAMES
     pipe
 )
-set(openPMD_PYTHON_CLI_MODULE_NAMES ${openPMD_CLI_TOOL_NAMES})
+set(openPMD_PYTHON_CLI_MODULE_NAMES ls)
 # examples
 set(openPMD_EXAMPLE_NAMES
     1_structure
@@ -900,6 +901,9 @@ if(openPMD_BUILD_CLI_TOOLS)
         endif()
 
         target_link_libraries(openpmd-${toolname} PRIVATE openPMD)
+        target_include_directories(openpmd-${toolname} SYSTEM PRIVATE
+            $<TARGET_PROPERTY:openPMD::thirdparty::nlohmann_json,INTERFACE_INCLUDE_DIRECTORIES>
+            $<TARGET_PROPERTY:openPMD::thirdparty::toml11,INTERFACE_INCLUDE_DIRECTORIES>)
     endforeach()
 endif()
 

include/openPMD/auxiliary/JSON_internal.hpp

@@ -219,16 +219,25 @@ namespace json
      * @param options as a parsed JSON object.
      * @param considerFiles If yes, check if `options` refers to a file and read
      *        from there.
+     * @param convertLowercase If yes, lowercase conversion is applied
+     *        recursively to keys and values, except for some hardcoded places
+     *        that should be left untouched.
      */
-    ParsedConfig parseOptions(std::string const &options, bool considerFiles);
+    ParsedConfig parseOptions(
+        std::string const &options,
+        bool considerFiles,
+        bool convertLowercase = true);
 
 #if openPMD_HAVE_MPI
 
     /**
      * Parallel version of parseOptions(). MPI-collective.
      */
-    ParsedConfig
-    parseOptions(std::string const &options, MPI_Comm comm, bool considerFiles);
+    ParsedConfig parseOptions(
+        std::string const &options,
+        MPI_Comm comm,
+        bool considerFiles,
+        bool convertLowercase = true);
 
 #endif
 

share/openPMD/json_schema/Makefile

@@ -0,0 +1,15 @@
+convert := openpmd-convert-toml-json
+
+json_files = attribute_defs.json attributes.json dataset_defs.json iteration.json mesh.json mesh_record_component.json particle_patches.json particle_species.json patch_record.json record.json record_component.json series.json
+
+.PHONY: all
+all: $(json_files)
+
+# The target file should only be created if the conversion succeeded
+$(json_files): %.json: %.toml
+	$(convert) @$^ > $@.tmp
+	mv $@.tmp $@
+
+.PHONY: clean
+clean:
+	for file in $(json_files); do rm -f "$$file" "$$file.tmp"; done

share/openPMD/json_schema/README.md

@@ -0,0 +1,47 @@
+# JSON Validation
+
+This folder contains a JSON schema for validation of openPMD files written as `.json` files.
+
+## Usage
+
+### Generating the JSON schema
+
+For improved readability, maintainability and documentation purposes, the JSON schema is written in `.toml` format and needs to be "compiled" to `.json` files first before usage.
+To do this, the openPMD-api installs a tool named `openpmd-convert-toml-json` which can be used to convert between JSON and TOML files in both directions, e.g.:
+
+```bash
+openpmd-convert-toml-json @series.toml > series.json
+```
+
+A `Makefile` is provided in this folder to automate generating the needed JSON files from the TOML files.
+
+### Verifying a file against the JSON schema
+
+In theory, the JSON schema should be applicable by any JSON validator. This JSON schema is written in terms of multiple files however, and most validators require special care to properly set up the links between the single files. A Python script `check.py` is provided in this folder which sets up the [Python jsonschema](https://python-jsonschema.readthedocs.io) library and verifies a file against it, e.g.:
+
+```bash
+./check.py path/to/my/dataset.json
+```
+
+For further usage notes check the documentation of the script itself `./check.py --help`.
+
+## Caveats
+
+The openPMD standard is not entirely expressible in terms of a JSON schema:
+
+* Many semantic dependencies, e.g., that the `position/x` and `position/y` vectors of a particle species need to be of the same size, or that the `axisLabels` have the same dimensionality as the dataset itself, will go unchecked.
+* The `meshesPath` is assumed to be `meshes/` and the `particlesPath` is assumed to be `particles/`. This dependency cannot be expressed.
+
+While a large part of the openPMD standard can indeed be verified by checking against a static JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages may become unexpectedly verbose and not very informative, especially when parsing disjunctive statements such as "A Record is either a scalar Record Component or a vector of non-scalar Record Components". We have taken care to decide disjunctive statements early on, e.g. with json-schema's support for `if` statements, but error messages may in general become unwieldy even due to tiny mistakes far down in the parse tree.
+
+The layout of attributes is assumed to be that which is created by the JSON backend of the openPMD-api. Both the longhand and shorthand forms are recognized:
+
+```json
+"meshesPath": {
+  "datatype": "STRING",
+  "value": "meshes/"
+},
+"particlesPath": "particles/"
+```
+
+For a custom-written verification of openPMD datasets, also consider using the [openPMD-validator](https://github.com/openPMD/openPMD-validator).