Skip to content

Commit da3bcd7

Browse files
rlyrly
authored
Merge pull request #285 from NeurodataWithoutBorders/doc/object_id
Document object_id
2 parents ddbb568 + f39446c commit da3bcd7

File tree

4 files changed

+44
-26
lines changed

4 files changed

+44
-26
lines changed

‎docs/format/source/format_description.rst‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,23 @@ Extending *NWBContainer* works in the same way, e.g., to create more specific ty
203203
data processing.
204204

205205

206+
Common attributes
207+
-----------------
208+
209+
All NWB:N Groups and Datasets with an assigned neurodata_type have three required attributes: `neurodata_type`,
210+
`namespace`, and `object_id`.
211+
212+
- ``neurodata_type`` (variable-length string) is the name of the NWB:N primitive that this group or dataset maps onto
213+
- ``namespace`` (variable-length string) is the namespace where ``neurodata_type`` is defined, e.g. "core" or the
214+
namespace of an extension
215+
- ``object_id`` (variable-length string) is a universally unique identifier for this object within its hierarchy.
216+
It should be set to the string representation of a random UUID version 4 value
217+
(see `RFC 4122 <https://tools.ietf.org/html/rfc4122>`_) upon first creation. It is **not** a hash of the data. Files
218+
that contain the exact same data but were generated in different instances will have different ``object_id`` values.
219+
Currently, modification of an object does not require its ``object_id`` to be changed.
220+
221+
222+
206223
Comments and Definitions
207224
========================
208225

@@ -313,4 +330,3 @@ The timestamps\_link and data\_link fields refer to links made between
313330
time series, such as if timeseries A and timeseries B, each having
314331
different data (or time) share time (or data). This is much more
315332
important information as it shows structural associations in the data.
316-

‎docs/storage/source/storage_description.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ basic primitives, e.g., Files, Groups, Datasets, Attributes, and Links to descri
1515
The role of the data storage then is to store large collections of neuroscience data. In other words,
1616
the role of the storage is to map NWB:N primitives (and types, i.e., neurodata_types) to persistent storage.
1717
For an overview of the various components of the NWB:N project
18-
see `here <http://nwb-overview.readthedocs.io/en/latest/nwbintro.html>`_ .
18+
see `here <https://neurodatawithoutborders.github.io/overview>`_ .
1919

2020
How are NWB:N files stored?
2121
===========================

‎docs/storage/source/storage_hdf5.rst‎

Lines changed: 20 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ HDF5
55
====
66

77
The NWB:N format currently uses the `Hierarchical Data Format (HDF5) <https://www.hdfgroup.org/HDF5/>`_
8-
as primary mechanism for data storage. HDF5 was selected for the
8+
as the primary mechanism for data storage. HDF5 was selected for the
99
NWB format because it met several of the project's
1010
requirements. First, it is a mature data format standard with libraries
1111
available in multiple programming languages. Second, the format's
@@ -27,7 +27,7 @@ technology.
2727
Format Mapping
2828
==============
2929

30-
Here we describe the mapping of NWB primitives (e.g,. Groups, Datasets, Attributes, Links etc.) used by
30+
Here we describe the mapping of NWB primitives (e.g,. Groups, Datasets, Attributes, Links, etc.) used by
3131
the NWB format and specification to HDF5 storage primitives. As the NWB:N format was designed with HDF5
3232
in mind, the high-level mapping between the format specification and HDF5 is quite simple:
3333

@@ -42,20 +42,21 @@ in mind, the high-level mapping between the format specification and HDF5 is qui
4242
Group Group
4343
Dataset Dataset
4444
Attribute Attribute
45-
Link Soft Link (or External Link)
45+
Link Soft Link or External Link
4646
============= ===============================================
4747

4848

4949
.. note::
5050

51-
In HDF5 Links are stored as HDF5 Soft Links (or External Links). Hard Links are not used in NWB as the primary location
52-
and, hence, primary ownership and link path for secondary locations, cannot be determined for Hard Links.
51+
Using HDF5, NWB links are stored as HDF5 Soft Links or External Links. Hard Links are not used in NWB because
52+
the primary location and, hence, primary ownership and link path for secondary locations, cannot be determined
53+
for Hard Links.
5354

5455

5556
Key Mapping
5657
===========
5758

58-
Here we describe the mapping of keys from the specifcation language to HDF5 storage objects:
59+
Here we describe the mapping of keys from the specification language to HDF5 storage objects:
5960

6061
Groups
6162
------
@@ -77,7 +78,8 @@ Groups
7778
linkable Not mapped; Stored in schema only
7879
quantity Not mapped; Number of appearances of the dataset.
7980
neurodata_type Attribute ``neurodata_type``
80-
namespace ID Attribute ``neurodata_namespace``
81+
namespace ID Attribute ``namespace``
82+
object ID Attribute ``object_id``
8183
============================ ======================================================================================
8284

8385

@@ -89,7 +91,6 @@ Datasets
8991
.. table:: Mapping of datasets
9092
:class: longtable
9193

92-
9394
============================ ======================================================================================
9495
NWB Key HDF5
9596
============================ ======================================================================================
@@ -102,14 +103,15 @@ Datasets
102103
linkable Not mapped; Stored in schema only
103104
quantity Not mapped; Number of appearances of the dataset.
104105
neurodata_type Attribute ``neurodata_type``
105-
namespace ID Attribute ``neurodata_namespace``
106+
namespace ID Attribute ``namespace``
107+
object ID Attribute ``object_id``
106108
============================ ======================================================================================
107109

108110
.. note::
109111

110-
* TODO Update mapping of namespace ID
111112
* TODO Update mapping of dims
112113

114+
113115
Attributes
114116
----------
115117

@@ -127,7 +129,6 @@ Attributes
127129
shape Shape of the HDF5 dataset if the shape is fixed, otherwise shape defines the maxshape
128130
dims Not mapped; Reflected by the shape of the attribute data
129131
required Not mapped; Stored in schema only
130-
parent Not mapped; In HDF5 all attributes are explicitly tied to the parent.
131132
value Data value of the attribute
132133
============================ ======================================================================================
133134

@@ -157,7 +158,7 @@ The mappings of data types is as follows
157158
+--------------------------+----------------------------------+----------------+
158159
| ``dtype`` **spec value** | **storage type** | **size** |
159160
+--------------------------+----------------------------------+----------------+
160-
| * "float" | single precision floating point | 32 bit |
161+
| * "float" | single precision floating point | 32 bit |
161162
| * "float32" | | |
162163
+--------------------------+----------------------------------+----------------+
163164
| * "double" | double precision floating point | 64 bit |
@@ -173,13 +174,13 @@ The mappings of data types is as follows
173174
+--------------------------+----------------------------------+----------------+
174175
| * "int8" | signed 8 bit integer | 8 bit |
175176
+--------------------------+----------------------------------+----------------+
176-
| * "uint32" | unsigned 32 bit integer | 32 bit |
177+
| * "uint32" | unsigned 32 bit integer | 32 bit |
177178
+--------------------------+----------------------------------+----------------+
178-
| * "uint16" | unsigned 16 bit integer | 16 bit |
179+
| * "uint16" | unsigned 16 bit integer | 16 bit |
179180
+--------------------------+----------------------------------+----------------+
180-
| * "uint8" | unsigned 8 bit integer | 8 bit |
181+
| * "uint8" | unsigned 8 bit integer | 8 bit |
181182
+--------------------------+----------------------------------+----------------+
182-
| * "bool" | boolean | 8 bit |
183+
| * "bool" | boolean | 8 bit |
183184
+--------------------------+----------------------------------+----------------+
184185
| * "text" | unicode | variable |
185186
| * "utf" | | |
@@ -196,9 +197,9 @@ The mappings of data types is as follows
196197
| * region | Reference to a region | |
197198
| | of another dataset | |
198199
+--------------------------+----------------------------------+----------------+
199-
| compound dtype + HDF5 compound data type | |
200+
| * compound dtype | HDF5 compound data type | |
200201
+--------------------------+----------------------------------+----------------+
201-
| * "isodatetime" | ASCII ISO8061 datetime string. | variable |
202+
| * "isodatetime" | ASCII ISO8061 datetime string. | variable |
202203
| | For example | |
203204
| | ``2018-09-28T14:43:54.123+02:00``| |
204205
+--------------------------+----------------------------------+----------------+
@@ -222,4 +223,4 @@ data type (e.g., ``dtype=special_dtype(vlen=binary_type)`` in Python). The speci
222223
``/specifications/<namespace-name>/<version>/namespace`` while additional source files are stored in
223224
``/specifications/<namespace-name>/<version>/<source-filename>``. Here ``<source-filename>`` refers to the main name
224225
of the source-file without file extension (e.g,. the core namespace defines ``nwb.ephys.yaml`` as source which would
225-
be stored in ``/specifications/core/2.0.1/nwb.ecephys``).
226+
be stored in ``/specifications/core/2.0.1/nwb.ecephys``).

‎docs/storage/source/storage_release_notes.rst‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,24 @@
22
Release Notes
33
=============
44

5+
NWB:N - v2.1.0
6+
--------------
7+
Added documentation for new NWB key 'object_id' (see also format release notes for NWB 2.1.0: https://nwb-schema.readthedocs.io/en/latest/format_release_notes.html#september-2019).
8+
59
NWB:N - v2.0.1
610
--------------
711
Added missing documentation on how format specification are cached in HDF5.
812

913
NWB:N - v2.0.0
1014
---------------
1115

12-
Created seperate reStructuredText documentation (i.e., this document) discuss and govern
16+
Created separate reStructuredText documentation (i.e., this document) discuss and govern
1317
storage-related concerns. In particular this documents describes how primitives and keys
1418
described via the specification language are mapped to storage, in particular HDF5.
1519

1620
NWB:N - v1.0.x and earlier
1721
--------------------------
1822

19-
For version 1.0.x and earlier, there was no official seperate document governing NWB:N storage concerns as
23+
For version 1.0.x and earlier, there was no official separate document governing NWB:N storage concerns as
2024
HDF5 was the only supported storage backend with implicit mapping between HDF5 types and NWB:N
2125
language primitives.
22-
23-
24-

0 commit comments

Comments
 (0)