Add JSON schema #1426

franzpoeschel · 2023-04-18T09:55:21Z

TODO:

Define particlePatches
Fill in things like "description", "type", "title", ...
~~Maybe add support for abbreviated JSON representations~~ Not now
Dataset layout

I have added a CLI tool openpmd-convert_json_toml, exposing internally existing functionality to convert between TOML and JSON. I think it's better to define and maintain the schema in terms of TOML (it can be commented and is more legible), and just compile it to JSON for verification.

x-ref #1436

ax3l

Great work, thank you!

Wondering a bit if validation of an XML backend would be easier.... not sure.

.github/workflows/linux.yml

ax3l · 2023-08-17T03:38:27Z

.github/workflows/linux.yml

+        # We need to exclude the thetaMode example since that has a different
+        # meshesPath and the JSON schema needs to hardcode that.


Are we able to patch this in check.py?

Not very easily. The JSON schema is on the file system and the single .json files refer to each other by their file names. Changing this would require (1) traversing the entire JSON schema and overriding the meshes path, the particles path and the references and (2) somehow setting up python-jsonschema to cross-reference in-memory schemas which I don't even know if it supports that, both at runtime of check.py.

franzpoeschel · 2023-08-17T13:53:24Z

Wondering a bit if validation of an XML backend would be easier.... not sure.

The major issue of the JSON schema approach is that it is not a predictive parser (https://en.wikipedia.org/wiki/Recursive_descent_parser), meaning that it deals with {"anyOf": [<option1>, <option2>, ...]}-like statements very naively. If there is an error somewhere deep down the hierarchy, the JSON schema just sees that all options have failed and cannot give any more details where the failure was.

If we want to seriously consider XML as an alternative, my opinion is that it must at least solve this particular issue .. which I doubt it can properly, since this involves a bit of parser theory and grammar transformation.

This job is better done by handwritten parsers, which the openPMD-api and the openPMD-validator are.

However, if we add an XML backend at some point, an XML schema is definitely at least worth a consideration.

Written as .toml files for ease of documentation, maintailability and readability.

Needed for "compiling" the schema to JSON Also add a Makefile to further simplify this

Workflow documented in README.md

The JSON schema verification package does not like that

Both of the form "data not found in places where data was expected"

Verify all JSON-openPMD files written by testing against the schema

1. Support UNDEFINED datasets in template mode 2. gridUnitSI may now be a vector

Apparently it's better to make everything 100 times more complicated

anyOf and oneOf now only used for trivial distinctions, this makes schemas much more robust since errors can be caught early and error messages become actually useful.

Otherwise CI thinks this is an openPMD file

ax3l · 2025-07-14T16:45:17Z

share/openPMD/json_schema/README.md

+openpmd_convert-json-toml @series.toml > series.json
+```
+
+A `Makefile` is provided in this folder to simplify the application of this conversion tool.


Makefile? Isn't this just built with CMake as one of the CLI tools?

Oh, I see. The makefile does the conversion of the schema toml files

Suggested change

A `Makefile` is provided in this folder to simplify the application of this conversion tool.

A `Makefile` is provided in this folder to simplify the application of this conversion tool to the `.toml` files in this folder.

Maybe a bit clearer: A Makefile is provided in this folder to automate generating the needed JSON files from the TOML files.
?

ax3l · 2025-07-14T16:47:36Z

CMakeLists.txt

    pipe
 )
-set(openPMD_PYTHON_CLI_MODULE_NAMES ${openPMD_CLI_TOOL_NAMES})
+set(openPMD_PYTHON_CLI_MODULE_NAMES ls)


This line change looks like a hack?

Not really. Until now, both openPMD_CLI_TOOL_NAMES and openPMD_PYTHON_CLI_MODULE_NAMES were identical since both contained only openpmd-ls.
But they're not identical in general; and now, we additionally have openpmd-convert-json-toml as a CLI tool, but it is not written in Python.

ax3l · 2025-07-14T16:48:16Z

CMakeLists.txt

 # command line tools
 set(openPMD_CLI_TOOL_NAMES
    ls
+    convert-json-toml


Also add this to setup.py -> entry_points -> console_scripts?

This is not a Python module, but written in C++, so I doubt that will work?

ax3l · 2025-07-14T16:49:27Z

share/openPMD/json_schema/Makefile

@@ -0,0 +1,15 @@
+convert := openpmd-convert-json-toml


Naming it openpmd-convert-toml-json seems to be more intuitive for what we use it here, but you mentioned anyway that it works bi-directional?

It is bi-directional, but we can flip the name, since that's the intended use here.

share/openPMD/json_schema/README.md

ax3l · 2025-07-14T16:51:18Z

share/openPMD/json_schema/README.md

+* Many semantic dependencies, e.g. that the `position/x` and `position/y` vector of a particle species be of the same size, or that the `axisLabels` have the same dimensionality as the dataset itself, will go unchecked.
+* The `meshesPath` is assumed to be `meshes/` and the `particlesPath` is assumed to be `particles/`. This dependency cannot be expressed.
+
+While a large part of the openPMD standard can indeed be verified by checking against a JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages will often be unexpectedly verbose and not very informative.


Might clarify a bit:

Suggested change

While a large part of the openPMD standard can indeed be verified by checking against a JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages will often be unexpectedly verbose and not very informative.

While a large part of the openPMD standard can indeed be verified by checking against a static JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages will often be unexpectedly verbose and not very informative.

ax3l · 2025-07-14T16:51:36Z

share/openPMD/json_schema/README.md

+}
+```
+
+Support for an abbreviated notation such as `"meshesPath": "meshes/"` is currently not (yet) available.


In this section, you might want to advertise & link openPMD-validator again.

Will do. We might additionally advertise the JSON schema in the validator. Adding the JSON validation to our CI brought up a number of bugs after all.

Co-authored-by: Axel Huebl <[email protected]>

ax3l

LGTM 👍

franzpoeschel added discussion tools documentation labels Apr 18, 2023

franzpoeschel force-pushed the json-schema branch from 46fcfcd to 78fc678 Compare April 18, 2023 13:24

ax3l self-requested a review April 24, 2023 20:36

ax3l self-assigned this Apr 24, 2023

franzpoeschel force-pushed the json-schema branch from 638c575 to 4f4be4b Compare May 11, 2023 09:19

franzpoeschel force-pushed the json-schema branch 3 times, most recently from 199a223 to 5b09f81 Compare May 31, 2023 09:53

franzpoeschel force-pushed the json-schema branch 3 times, most recently from b6b057b to e99b0fb Compare June 26, 2023 14:35

franzpoeschel force-pushed the json-schema branch 3 times, most recently from 3fb1ea9 to e866604 Compare August 10, 2023 09:10

ax3l reviewed Aug 17, 2023

View reviewed changes

franzpoeschel force-pushed the json-schema branch from 7a5fee0 to e66de14 Compare September 5, 2023 11:00

franzpoeschel force-pushed the json-schema branch 3 times, most recently from c9fb0c1 to 4df3d58 Compare December 22, 2023 18:31

franzpoeschel force-pushed the json-schema branch from 4df3d58 to 54a2850 Compare January 23, 2024 15:48

franzpoeschel force-pushed the json-schema branch from 54a2850 to 2a36992 Compare February 14, 2024 10:13

franzpoeschel force-pushed the json-schema branch from 2a36992 to 43b7bea Compare February 27, 2024 10:44

franzpoeschel force-pushed the json-schema branch from 43b7bea to 785dae7 Compare March 26, 2024 10:56

ax3l force-pushed the dev branch from b80dfee to 805c760 Compare May 7, 2024 17:28

franzpoeschel force-pushed the json-schema branch from 785dae7 to 321130a Compare May 21, 2024 14:50

franzpoeschel force-pushed the json-schema branch 2 times, most recently from 686d5af to 58497f1 Compare July 17, 2024 09:18

franzpoeschel force-pushed the json-schema branch from 4193a3c to cbecd5b Compare July 11, 2025 13:56

franzpoeschel added 19 commits July 14, 2025 16:12

Add JSON schema for openPMD

245e19d

Written as .toml files for ease of documentation, maintailability and readability.

Add convert-json-toml tool

a860269

Needed for "compiling" the schema to JSON Also add a Makefile to further simplify this

Add script for checking openPMD file against the schema

4c957b4

Workflow documented in README.md

Don't use spaces in SerialIOTest attribute names

a510957

The JSON schema verification package does not like that

Fix bugs detected by this verifier

facfa15

Both of the form "data not found in places where data was expected"

Add GitHub workflow

3749056

Verify all JSON-openPMD files written by testing against the schema

Shorthand attributes

ba3f5b1

Add dataset template mode

a719bb6

Fix path

3a5fc19

Fix reading from stdin

334f2d5

toml11 4.0 compatibility

90ce201

Only check for existing Iterations in writeOnly mode

779cf12

Some additions to schema

f6b1f24

1. Support UNDEFINED datasets in template mode 2. gridUnitSI may now be a vector

Remove deprecated jsonschema.validators.RefResolver

f18a88b

Apparently it's better to make everything 100 times more complicated

Use most recent version of jsonschema

5e4a870

Allow empty variable-based series

252c0d4

Use if-then-else for better-steered parsing

65aa12d

anyOf and oneOf now only used for trivial distinctions, this makes schemas much more robust since errors can be caught early and error messages become actually useful.

hmm

83ed23a

Remove json cfg after test

579e7b0

Otherwise CI thinks this is an openPMD file

franzpoeschel force-pushed the json-schema branch from cbecd5b to 579e7b0 Compare July 14, 2025 15:26

ax3l reviewed Jul 14, 2025

View reviewed changes

franzpoeschel and others added 4 commits July 15, 2025 15:15

Update documentation, rename convert-toml-json tool

81533b0

Apply suggestions from code review

985d505

Co-authored-by: Axel Huebl <[email protected]>

Add reference to openPMD-validator

3a2929f

Update README.md

d578167

ax3l approved these changes Aug 6, 2025

View reviewed changes

ax3l merged commit d4f3e2a into openPMD:dev Aug 6, 2025
30 checks passed

franzpoeschel mentioned this pull request Aug 8, 2025

Fix missing check for constant components #1776

Merged

		# We need to exclude the thetaMode example since that has a different
		# meshesPath and the JSON schema needs to hardcode that.

	A `Makefile` is provided in this folder to simplify the application of this conversion tool.
	A `Makefile` is provided in this folder to simplify the application of this conversion tool to the `.toml` files in this folder.

	While a large part of the openPMD standard can indeed be verified by checking against a JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages will often be unexpectedly verbose and not very informative.
	While a large part of the openPMD standard can indeed be verified by checking against a static JSON schema, the standard is generally large enough to make this approach come to its limits. Verification of a JSON schema is similar to the use of a naive recursive-descent parser. Error messages will often be unexpectedly verbose and not very informative.

Add JSON schema #1426

Add JSON schema #1426

Uh oh!

Conversation

franzpoeschel commented Apr 18, 2023 • edited by ax3l Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franzpoeschel commented Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ax3l Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

franzpoeschel commented Apr 18, 2023 •

edited by ax3l

Loading

franzpoeschel commented Aug 17, 2023 •

edited

Loading

ax3l Jul 14, 2025 •

edited

Loading