From e91293bcc95cab01e5f88cd64a96c3985c961843 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Franz=20P=C3=B6schel?= Date: Mon, 7 Aug 2023 17:46:39 +0200 Subject: [PATCH 1/4] Document extended meshesPath functionality --- STANDARD.md | 73 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 67 insertions(+), 6 deletions(-) diff --git a/STANDARD.md b/STANDARD.md index 9fc0b05..2702ca0 100644 --- a/STANDARD.md +++ b/STANDARD.md @@ -123,12 +123,73 @@ The following attributes are *optional* in each each file's *root* group *required* to set them if one wants to store mesh and/or particle records. - `meshesPath` - - type: *(string)* - - description: path *relative* from the `basePath` to the mesh records - - example: `meshes/` - - note: if this attribute is missing, the file is interpreted as if it - contains *no mesh records*! If the attribute is set, the group behind - it *must* exist! + - type: 1-dimensional array containing N *(string)* + - description: List of regular expressions (regex) that specify the location + of mesh records *relative* from the `basePath`. + Let the path to a mesh record be denoted by + `//`, + relative to the `basePath` and starting + with a leading slash `/`. + The mesh can be specified in three different ways: + + 1. **Full path to the group containing meshes**: + The `meshesPath` contains + `//` + This mode is recognized by both the leading and trailing + slash `/`. + Any group found in this group will be interpreted as a mesh record. + 2. **Full path to the mesh itself**: + The `meshesPath` contains + `//` + This mode is recognized by a leading, but no trailing slash `/`. + This specific path will be interpreted as a mesh record. + 3. **Shorthand notation: Name of groups that contain meshes**: + The `meshesPath` contains `/`. + This mode is recognized by no leading, but a trailing slash. + No further slashes than the trailing slash must be used. + Any group with the specified name will be treated as containing only + meshes. + + - Examples for the 3 different notations: + + 1. `.*/meshes/` is equivalent to `meshes/` (see format 3). + Specifying `/meshes/` refers only to the `meshes` group found + directly in the base path, e.g. in a group-based file: + + - `/data/0/meshes/E` will be recognized as a mesh, + - but `/data/100/refinement_levels/2/meshes/E` will not. + + In this example, `/refinement_levels/[[:num:]]+/meshes/` can be used + to refer to the meshes found at different mesh refinement levels. + 2. `/meshes/E` refers only to this particular mesh relative from the + `basePath`. E.g., in a group-based file: + + - `/data/100/meshes/E` will be recognized as a mesh, + - but neither will `/data/50/meshes/B`, + - nor will `/data/0/refinement_levels/2/meshes/E`. + + 3. `meshes/`: Any group with name `meshes` contains only mesh records. + In a group-based file, this will recognize the following paths + as meshes: + + - `/data/0/meshes/E` + - `/data/100/refinement_levels/2/meshes/E` + + - Notes: + + A single regex is technically sufficient since regexes can express + multiple options. However, using a list improves legibility and can + help tooling interpret user intent better (e.g. if the first list + entry is `fields/`, then an implementation can choose to use that + name by default instead of `meshes/`). + + The shorthand notation (format 3) corresponds with the openPMD 1.0 + notation of meshes paths. + + No specific dialect of regular expressions is specified, but the + `egrep` style is recommended. + @TODO: Should we use a recommendation? Should we encode the style + as an attribute? - `particlesPath` - type: *(string)* From f86801877588f1d235c444d01502612c9e3a1f5c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Franz=20P=C3=B6schel?= Date: Tue, 12 Sep 2023 10:54:08 +0200 Subject: [PATCH 2/4] Same for particlesPath --- STANDARD.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 68 insertions(+), 7 deletions(-) diff --git a/STANDARD.md b/STANDARD.md index 2702ca0..39cee00 100644 --- a/STANDARD.md +++ b/STANDARD.md @@ -192,13 +192,74 @@ The following attributes are *optional* in each each file's *root* group as an attribute? - `particlesPath` - - type: *(string)* - - description: path *relative* from the `basePath` to the groups for each - particle group and the records they include - - example: `particles/` - - note: if this attribute is missing, the file is interpreted as if it - contains *no particle records*! If the attribute is set, the group behind - it *must* exist! + - type: 1-dimensional array containing N *(string)* + - description: List of regular expressions (regex) that specify the location + of particle species *relative* from the `basePath`. + Let the path to a particle species be denoted by + `//`, + relative to the `basePath` and starting + with a leading slash `/`. + The particle can be specified in three different ways: + + 1. **Full path to the group containing particles**: + The `particlesPath` contains + `//` + This mode is recognized by both the leading and trailing + slash `/`. + Any group found in this group will be interpreted as a particle species. + 2. **Full path to the particle species itself**: + The `particlesPath` contains + `//` + This mode is recognized by a leading, but no trailing slash `/`. + This specific path will be interpreted as a particle record. + 3. **Shorthand notation: Name of groups that contain particles**: + The `particlesPath` contains `/`. + This mode is recognized by no leading, but a trailing slash. + No further slashes than the trailing slash must be used. + Any group with the specified name will be treated as containing only + particles. + + - Examples for the 3 different notations: + + 1. `.*/particles/` is equivalent to `particles/` (see format 3). + Specifying `/particles/` refers only to the `particles` group found + directly in the base path, e.g. in a group-based file: + + - `/data/0/particles/e` will be recognized as a particle species, + - but `/data/100/generations/2/particles/e` will not. + + In this example, `/generations/[[:num:]]+/particles/` can be used + to refer to the particles found at different particle generations. + 2. `/particles/e` refers only to this particular particle species relative + from the `basePath`. E.g., in a group-based file: + + - `/data/100/particles/e` will be recognized as a particle species, + - but neither will `/data/50/particles/i`, + - nor will `/data/0/generations/2/particles/e`. + + 3. `particles/`: Any group with name `particles` contains only + particle species. + In a group-based file, this will recognize the following paths + as particle species: + + - `/data/0/particles/e` + - `/data/100/generations/2/particles/e` + + - Notes: + + A single regex is technically sufficient since regexes can express + multiple options. However, using a list improves legibility and can + help tooling interpret user intent better (e.g. if the first list + entry is `fields/`, then an implementation can choose to use that + name by default instead of `particles/`). + + The shorthand notation (format 3) corresponds with the openPMD 1.0 + notation of particles paths. + + No specific dialect of regular expressions is specified, but the + `egrep` style is recommended. + @TODO: Should we use a recommendation? Should we encode the style + as an attribute? It is *recommended* that each file's *root* group (path `/`) further contains the attributes: From 543c374a5366b653a2783ccb3f74fbf9c465dee1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Franz=20P=C3=B6schel?= Date: Thu, 21 Dec 2023 18:30:18 +0100 Subject: [PATCH 3/4] Globbing for meshes --- STANDARD.md | 73 +++++++++++++++++++---------------------------------- 1 file changed, 26 insertions(+), 47 deletions(-) diff --git a/STANDARD.md b/STANDARD.md index 39cee00..0ac47e4 100644 --- a/STANDARD.md +++ b/STANDARD.md @@ -124,73 +124,52 @@ The following attributes are *optional* in each each file's *root* group - `meshesPath` - type: 1-dimensional array containing N *(string)* - - description: List of regular expressions (regex) that specify the location - of mesh records *relative* from the `basePath`. - Let the path to a mesh record be denoted by - `//`, - relative to the `basePath` and starting - with a leading slash `/`. - The mesh can be specified in three different ways: + - description: List of globbing expressions that specify the location + of mesh containers *relative* from the `basePath`. + Mesh containers are openPMD groups that contain meshes. + Each group in a mesh container is treated as a mesh; each dataset in a mesh container is treated as a (scalar) mesh. + The entries in the `meshesPath` can be specified in two different ways: - 1. **Full path to the group containing meshes**: - The `meshesPath` contains - `//` - This mode is recognized by both the leading and trailing - slash `/`. - Any group found in this group will be interpreted as a mesh record. - 2. **Full path to the mesh itself**: - The `meshesPath` contains - `//` - This mode is recognized by a leading, but no trailing slash `/`. - This specific path will be interpreted as a mesh record. - 3. **Shorthand notation: Name of groups that contain meshes**: - The `meshesPath` contains `/`. - This mode is recognized by no leading, but a trailing slash. - No further slashes than the trailing slash must be used. - Any group with the specified name will be treated as containing only - meshes. + 1. **Full path to mesh container**: + An entry in the `meshesPath` that starts and ends with a slash `/`. + This can also be a globbing expression: - - Examples for the 3 different notations: + a. A single `%` expands to any legal group name. + b. A double `%%` expands to any legal path. Unlike a single `%`, its expansion may contain slashes. The full path may start with a double `%%` instead of the leading slash `/`. + + As a regular expression: `(/|%%)[[:alnum:]_%/]+/`. + + 2. **Shorthand notation: Name of mesh containers**: + An entry in the `meshesPath` that ends with a slash `/` and contains no other slashes. + Any group with the specified name will be treated as a mesh container. + There is no globbing support. + + As a regular expression: `[[:alnum:]_]+/`. - 1. `.*/meshes/` is equivalent to `meshes/` (see format 3). + - Examples for the 2 different notations: + + 1. `%%/meshes/` is equivalent to `meshes/` (see format 3). Specifying `/meshes/` refers only to the `meshes` group found directly in the base path, e.g. in a group-based file: - `/data/0/meshes/E` will be recognized as a mesh, - but `/data/100/refinement_levels/2/meshes/E` will not. - In this example, `/refinement_levels/[[:num:]]+/meshes/` can be used + In this example, `/refinement_levels/%/meshes/` can be used to refer to the meshes found at different mesh refinement levels. - 2. `/meshes/E` refers only to this particular mesh relative from the - `basePath`. E.g., in a group-based file: - - - `/data/100/meshes/E` will be recognized as a mesh, - - but neither will `/data/50/meshes/B`, - - nor will `/data/0/refinement_levels/2/meshes/E`. - 3. `meshes/`: Any group with name `meshes` contains only mesh records. + 2. `meshes/`: Any group with name `meshes` contains only mesh records. In a group-based file, this will recognize the following paths as meshes: - `/data/0/meshes/E` - `/data/100/refinement_levels/2/meshes/E` - - Notes: - - A single regex is technically sufficient since regexes can express - multiple options. However, using a list improves legibility and can - help tooling interpret user intent better (e.g. if the first list - entry is `fields/`, then an implementation can choose to use that - name by default instead of `meshes/`). + - Note: - The shorthand notation (format 3) corresponds with the openPMD 1.0 + The shorthand notation (format 2) corresponds with the openPMD 1.0 notation of meshes paths. - No specific dialect of regular expressions is specified, but the - `egrep` style is recommended. - @TODO: Should we use a recommendation? Should we encode the style - as an attribute? - - `particlesPath` - type: 1-dimensional array containing N *(string)* - description: List of regular expressions (regex) that specify the location From 35f75fc498e64dcb2dfa5bc9a8d5279ff97a7c93 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Franz=20P=C3=B6schel?= Date: Thu, 21 Dec 2023 18:37:15 +0100 Subject: [PATCH 4/4] Globbing for particles --- STANDARD.md | 77 +++++++++++++++++++---------------------------------- 1 file changed, 28 insertions(+), 49 deletions(-) diff --git a/STANDARD.md b/STANDARD.md index 0ac47e4..2ae9ebe 100644 --- a/STANDARD.md +++ b/STANDARD.md @@ -172,51 +172,41 @@ The following attributes are *optional* in each each file's *root* group - `particlesPath` - type: 1-dimensional array containing N *(string)* - - description: List of regular expressions (regex) that specify the location - of particle species *relative* from the `basePath`. - Let the path to a particle species be denoted by - `//`, - relative to the `basePath` and starting - with a leading slash `/`. - The particle can be specified in three different ways: - - 1. **Full path to the group containing particles**: - The `particlesPath` contains - `//` - This mode is recognized by both the leading and trailing - slash `/`. - Any group found in this group will be interpreted as a particle species. - 2. **Full path to the particle species itself**: - The `particlesPath` contains - `//` - This mode is recognized by a leading, but no trailing slash `/`. - This specific path will be interpreted as a particle record. - 3. **Shorthand notation: Name of groups that contain particles**: - The `particlesPath` contains `/`. - This mode is recognized by no leading, but a trailing slash. - No further slashes than the trailing slash must be used. - Any group with the specified name will be treated as containing only - particles. - - - Examples for the 3 different notations: - - 1. `.*/particles/` is equivalent to `particles/` (see format 3). + - description: List of globbing expressions that specify the location + of particle containers *relative* from the `basePath`. + Particle containers are openPMD groups that contain (one or more) particle species. + Each group in a particle container is treated as a particle species. + The entries in the `particlesPath` can be specified in two different ways: + + 1. **Full path to particle container**: + An entry in the `particlesPath` that starts and ends with a slash `/`. + This can also be a globbing expression: + + a. A single `%` expands to any legal group name. + b. A double `%%` expands to any legal path. Unlike a single `%`, its expansion may contain slashes. The full path may start with a double `%%` instead of the leading slash `/`. + + As a regular expression: `(/|%%)[[:alnum:]_%/]+/`. + + 2. **Shorthand notation: Name of mesh containers**: + An entry in the `particlesPath` that ends with a slash `/` and contains no other slashes. + Any group with the specified name will be treated as a particles container. + There is no globbing support. + + As a regular expression: `[[:alnum:]_]+/`. + + - Examples for the 2 different notations: + + 1. `%%/particles/` is equivalent to `particles/` (see format 2). Specifying `/particles/` refers only to the `particles` group found directly in the base path, e.g. in a group-based file: - `/data/0/particles/e` will be recognized as a particle species, - but `/data/100/generations/2/particles/e` will not. - In this example, `/generations/[[:num:]]+/particles/` can be used + In this example, `/generations/%/particles/` can be used to refer to the particles found at different particle generations. - 2. `/particles/e` refers only to this particular particle species relative - from the `basePath`. E.g., in a group-based file: - - - `/data/100/particles/e` will be recognized as a particle species, - - but neither will `/data/50/particles/i`, - - nor will `/data/0/generations/2/particles/e`. - 3. `particles/`: Any group with name `particles` contains only + 2. `particles/`: Any group with name `particles` contains only particle species. In a group-based file, this will recognize the following paths as particle species: @@ -226,20 +216,9 @@ The following attributes are *optional* in each each file's *root* group - Notes: - A single regex is technically sufficient since regexes can express - multiple options. However, using a list improves legibility and can - help tooling interpret user intent better (e.g. if the first list - entry is `fields/`, then an implementation can choose to use that - name by default instead of `particles/`). - - The shorthand notation (format 3) corresponds with the openPMD 1.0 + The shorthand notation (format 2) corresponds with the openPMD 1.0 notation of particles paths. - No specific dialect of regular expressions is specified, but the - `egrep` style is recommended. - @TODO: Should we use a recommendation? Should we encode the style - as an attribute? - It is *recommended* that each file's *root* group (path `/`) further contains the attributes: