Clarify nesting, external index location

Author: mkitti

docs/v3/codecs/sharding-indexed/index.rst

@@ -5,17 +5,18 @@ Sharding codec
 ==============
 
 Version:
-    1.0
+    1.1
 Specification URI:
     https://zarr-specs.readthedocs.io/en/latest/v3/codecs/sharding-indexed/
 Editors:
     * Jonathan Striebel (`@jstriebel <https://github.com/jstriebel>`_), Scalable Minds
     * Norman Rzepka (`@normanrz <https://github.com/normanrz>`_), Scalable Minds
     * Jeremy Maitin-Shepard (`@jbms <https://github.com/jbms>`_), Google
+    * Mark Kittisopikul (`@mkitti <https://github.com/mkitti>`_), Howard Hughes Medical Institute
 Corresponding ZEP:
     `ZEP0002 — Sharding codec <https://zarr.dev/zeps/accepted/ZEP0002.html>`_
 Issue tracking:
-    `GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/sharding-indexed-codec-v1.0>`_
+    `GitHub issues <https://github.com/zarr-developers/zarr-specs/labels/sharding-indexed-codec-v1.1>`_
 Suggest an edit for this spec:
     `GitHub editor <https://github.com/zarr-developers/zarr-specs/blob/main/docs/codecs/sharding-indexed/index.rst>`_
 
@@ -140,25 +141,33 @@ Sharding can be configured per array in the :ref:`array-metadata` as follows::
 ``codecs``
 
     Specifies a list of codecs to be used for encoding and decoding inner chunks. 
-    The value must be an array of objects, as specified in the 
+    The value must be an array of objects, as specified in the
     :ref:`array-metadata`. The ``codecs`` member is required and needs to contain
     exactly one ``array -> bytes`` codec.
 
 ``index_codecs``
 
-    Specifies a list of codecs to be used for encoding and decoding shard index. 
-    The value must be an array of objects, as specified in the 
+    Specifies a list of codecs to be used for encoding and decoding a shard index.
+    The shard index is an array with a ``shape`` of ``[N,2]`` and a ``data_type`` of
+    ``uint64`` where ``N`` is the number of chunks to be indexed in the shard.
+    The ``index_codecs`` value must be an array of objects, as specified in the
     :ref:`array-metadata`. The ``index_codecs`` member is required and needs to 
-    contain exactly one ``array -> bytes`` codec. Codecs that produce 
-    variable-sized encoded representation, such as compression codecs, MUST NOT
-    be used for index codecs. It is RECOMMENDED to use a little-endian codec 
-    followed by a crc32c checksum as index codecs.
+    contain exactly one ``array -> bytes`` codec. That codec MAY be preceded by
+    ``array -> array`` codecs that modify either the ``shape`` or ``data_type``
+    of the array. Codecs that produce variable-sized encoded representation,
+    such as compression codecs, MUST NOT be used for index codecs. It is
+    RECOMMENDED to use a little-endian codec followed by a crc32c checksum as
+    index codecs.
 
 ``index_location``
 
-    Specifies whether the shard index is located at the beginning or end of the 
-    file. The parameter value must be either the string ``start`` or ``end``. 
-    If the parameter is not present, the value defaults to ``end``.
+    Specifies whether the shard index is located at the beginning of the file,
+    the end of the file, or external to the file in its own key. The parameter
+    value must be either the string ``start``, ``end``, or ``external``. A value
+    of external indicates the shard index has been written to an external file
+    referred to by a key constructed by appending ".shard_index" to the key of
+    the sharded chunk.  If the parameter is not present, the value defaults to
+    ``end``.
 
 Definitions
 ===========
@@ -180,8 +189,8 @@ Binary shard format
 This is an ``array -> bytes`` codec.
 
 In the ``sharding_indexed`` binary format, inner chunks are written successively in a 
-shard, where unused space between them is allowed, followed by an index 
-referencing them.
+shard, where unused space between them is allowed. An index referencing them may
+precede, follow, or exist external to the shard.
 
 The index is an array with 64-bit unsigned integers with a shape that matches the
 chunks per shard tuple with an appended dimension of size 2.
@@ -199,11 +208,11 @@ Empty inner chunks are interpreted as being filled with the fill value. The inde
 always has the full shape of all possible inner chunks per shard, even if they extend
 beyond the array shape.
 
-The index is either placed at the end of the file or at the beginning of the file,
-as configured by the ``index_location`` parameter. The index is encoded into binary 
-representations using the specified index codecs. The byte size of the index is 
-determined by the number of inner chunks in the shard ``n``, i.e. the product of 
-chunks per shard, and the choice of index codecs.
+The index is either placed at the end of the file or, at the beginning of the file,
+or under its own key, as configured by the ``index_location`` parameter. The index
+is encoded into binary representations using the specified index codecs. The byte
+size of the index is determined by the number of inner chunks in the shard ``n``,
+i.e. the product of chunks per shard, and the choice of index codecs.
 
 For an example, consider a shard shape of ``[64, 64]``, an inner chunk shape of 
 ``[32, 32]`` and an index codec combination of a little-endian codec followed by 
@@ -250,12 +259,12 @@ common optimizations.
 * **Decoding**: A simple implementation to decode inner chunks in a shard would (a) 
   read the entire value from the store into a byte buffer, (b) parse the shard
   index as specified above from the beginning or end (according to the 
-  ``index_location``) of the buffer and (c) cut out the relevant bytes that belong 
-  to the requested chunk. The relevant bytes are determined by the 
-  ``offset,nbytes`` pair in the shard index. This bytestream then needs to be 
-  decoded with the inner codecs as specified in the sharding configuration applying 
-  the :ref:`decoding_procedure`. This is similar to how an implementation would 
-  access a sub-slice of a chunk.
+  ``index_location``) of the buffer or from an external index and (c) cut out
+  the relevant bytes that belong to the requested chunk. The relevant bytes are
+  determined by the ``offset,nbytes`` pair in the shard index. This bytestream
+  then needs to be decoded with the inner codecs as specified in the sharding
+  configuration applying the :ref:`decoding_procedure`. This is similar to how
+  an implementation would access a sub-slice of a chunk.
 
   The size of the index can be determined by applying ``c.compute_encoded_size``
   for each index codec recursively. The initial size is the byte size of the index 
@@ -277,24 +286,35 @@ common optimizations.
   encode the new chunk per :ref:`encoding_procedure` in a byte buffer using the 
   shard's inner codecs, (b) read an existing shard from the store, (c) create a 
   new bytestream with all encoded inner chunks of that shard including the overwritten 
-  chunk, (d) generate a new shard index that is prepended or appended (according 
-  to the ``index_location``) to the chunk bytestream and (e) writes the shard to 
-  the store. If there was no existing shard, an empty shard is assumed. When 
-  writing entire inner chunks, reading the existing shard first may be skipped.
+  chunk, (d) generate a new shard index that is prepended, appended, or
+  externally written (according to the ``index_location``) to the chunk
+  bytestream and (e) writes the shard to the store. If there was no existing
+  shard, an empty shard is assumed. When writing entire inner chunks, reading
+  the existing shard first may be skipped.
 
   When working with inner chunks that have a fixed byte size (e.g., uncompressed) and 
   a store that supports partial writes, a optimization would be to replace the
   new chunk by writing to the store at the specified byte range.
 
   On stores with random-write capabilities, it may be useful to (a) place the shard 
-  index at the beginning of the file, (b) write out inner chunks in 
-  application-specific order, and (c) update the shard index accordingly. 
+  index at the beginning of the file or in a separate file, (b) write out inner
+  chunks in application-specific order, and (c) update the shard index accordingly.
   Synchronization of parallelly written inner chunks needs to be handled by the
   application.
 
   Other use case-specific optimizations may be available, e.g., for append-only
   workloads.
 
+* **Nesting**: The ``sharding_indexed`` codec can be used as part of a codec
+  chain of another ``sharding_indexed`` codec. This means that an inner chunk
+  MAY itself be a shard nested within an outer chunk, creating a hierarchical
+  index and multiple levels of partitioning. While the number of nested levels
+  of shards is not restricted, some implementations MAY support a limited
+  number of nested shards or MAY NOT support nesting. Primary shards that
+  are not contained within other shards MAY have an ``index_location`` value of
+  ``start``, ``end``, or ``external``.  Nested shards MAY have an
+  ``index_location`` value of ``start`` or ``end``. Nested shards MUST NOT have
+  an ``index_location`` value of ``external``.
 
 References
 ==========
@@ -306,6 +326,8 @@ References
 Change log
 ==========
 
+* Add ``external`` as a parameter value for ``index_location`` to Version 1.1 and clarified nesting. `PR ABC <https://github.com/zarr-developers/zarr-specs/pull/ABC>`_
+
 * Adds ``index_location`` parameter. `PR 280 <https://github.com/zarr-developers/zarr-specs/pull/280>`_
 
 * ZEP0002 was accepted. `Issue 254 <https://github.com/zarr-developers/zarr-specs/pull/254>`_