Update documentation

Author: actions-user

explanation/type-system/index.html

@@ -554,6 +554,15 @@
 </a>
 </li>
 <li class="md-nav__item">
+<a class="md-nav__link" href="#plugin-codecs">
+<span class="md-ellipsis">
+      
+        Plugin Codecs
+      
+    </span>
+</a>
+</li>
+<li class="md-nav__item">
 <a class="md-nav__link" href="#blob-serialized-python-objects">
 <span class="md-ellipsis">
 
@@ -572,6 +581,15 @@
 </a>
 </li>
 <li class="md-nav__item">
+<a class="md-nav__link" href="#npy-numpy-arrays-as-npy-files">
+<span class="md-ellipsis">
+      
+        &lt;npy@&gt; — NumPy Arrays as .npy Files
+      
+    </span>
+</a>
+</li>
+<li class="md-nav__item">
 <a class="md-nav__link" href="#object-path-addressed-storage">
 <span class="md-ellipsis">
 
@@ -2394,6 +2412,15 @@
 </a>
 </li>
 <li class="md-nav__item">
+<a class="md-nav__link" href="#plugin-codecs">
+<span class="md-ellipsis">
+      
+        Plugin Codecs
+      
+    </span>
+</a>
+</li>
+<li class="md-nav__item">
 <a class="md-nav__link" href="#blob-serialized-python-objects">
 <span class="md-ellipsis">
 
@@ -2412,6 +2439,15 @@
 </a>
 </li>
 <li class="md-nav__item">
+<a class="md-nav__link" href="#npy-numpy-arrays-as-npy-files">
+<span class="md-ellipsis">
+      
+        &lt;npy@&gt; — NumPy Arrays as .npy Files
+      
+    </span>
+</a>
+</li>
+<li class="md-nav__item">
 <a class="md-nav__link" href="#object-path-addressed-storage">
 <span class="md-ellipsis">
 
@@ -2487,6 +2523,7 @@ <h2 id="three-layer-architecture">Three-Layer Architecture<a class="headerlink"
     subgraph "Layer 3: Codecs"
         blob["‹blob›"]
         attach["‹attach›"]
+        npy["‹npy@›"]
         object["‹object@›"]
         hash["‹hash@›"]
         custom["‹custom›"]
@@ -2508,6 +2545,7 @@ <h2 id="three-layer-architecture">Three-Layer Architecture<a class="headerlink"
 
     blob --&gt; bytes
     attach --&gt; bytes
+    npy --&gt; json
     object --&gt; json
     hash --&gt; json
     bytes --&gt; BLOB
@@ -2663,6 +2701,12 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
 <td>Local file path</td>
 </tr>
 <tr>
+<td><code>&lt;npy@&gt;</code></td>
+<td>❌</td>
+<td>✅</td>
+<td>NpyRef (lazy)</td>
+</tr>
+<tr>
 <td><code>&lt;object@&gt;</code></td>
 <td>❌</td>
 <td>✅</td>
@@ -2682,6 +2726,58 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
 </tr>
 </tbody>
 </table>
+<h3 id="plugin-codecs">Plugin Codecs<a class="headerlink" href="#plugin-codecs" title="Permanent link">¶</a></h3>
+<p>Additional codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.</p>
+<table>
+<thead>
+<tr>
+<th>Package</th>
+<th>Codec</th>
+<th>Description</th>
+<th>Repository</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>dj-zarr-codecs</code></td>
+<td><code>&lt;zarr@&gt;</code></td>
+<td>Zarr arrays with lazy chunked access</td>
+<td><a href="https://github.com/datajoint/dj-zarr-codecs">datajoint/dj-zarr-codecs</a></td>
+</tr>
+<tr>
+<td><code>dj-figpack-codecs</code></td>
+<td><code>&lt;figpack@&gt;</code></td>
+<td>Interactive browser visualizations</td>
+<td><a href="https://github.com/datajoint/dj-figpack-codecs">datajoint/dj-figpack-codecs</a></td>
+</tr>
+<tr>
+<td><code>dj-photon-codecs</code></td>
+<td><code>&lt;photon@&gt;</code></td>
+<td>Photon imaging data formats</td>
+<td><a href="https://github.com/datajoint/dj-photon-codecs">datajoint/dj-photon-codecs</a></td>
+</tr>
+</tbody>
+</table>
+<p><strong>Installation and discovery:</strong></p>
+<p>Plugin codecs use Python's entry point mechanism for automatic registration. Install the package and DataJoint discovers the codec automatically:</p>
+<div class="highlight"><pre><span></span><code>pip<span class="w"> </span>install<span class="w"> </span>dj-zarr-codecs
+</code></pre></div>
+<div class="highlight"><pre><span></span><code><span class="kn">import</span><span class="w"> </span><span class="nn">datajoint</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">dj</span>
+
+<span class="c1"># Codec is available immediately after install</span>
+<span class="nd">@schema</span>
+<span class="k">class</span><span class="w"> </span><span class="nc">Analysis</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Computed</span><span class="p">):</span>
+    <span class="n">definition</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2">    -&gt; Recording</span>
+<span class="s2">    ---</span>
+<span class="s2">    data : &lt;zarr@store&gt;</span>
+<span class="s2">    """</span>
+</code></pre></div>
+<p>Packages declare their codecs in <code>pyproject.toml</code> under the <code>datajoint.codecs</code> entry point group:</p>
+<div class="highlight"><pre><span></span><code><span class="k">[project.entry-points.</span><span class="s2">"datajoint.codecs"</span><span class="k">]</span>
+<span class="n">zarr</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"dj_zarr_codecs:ZarrCodec"</span>
+</code></pre></div>
+<p>DataJoint loads these entry points on first use, making third-party codecs indistinguishable from built-ins.</p>
 <h3 id="blob-serialized-python-objects"><code>&lt;blob&gt;</code> — Serialized Python Objects<a class="headerlink" href="#blob-serialized-python-objects" title="Permanent link">¶</a></h3>
 <p>Stores NumPy arrays, dicts, lists, and other Python objects using DataJoint's custom binary serialization format.</p>
 <p><strong>Serialization format:</strong></p>
@@ -2727,6 +2823,34 @@ <h3 id="attach-file-attachments"><code>&lt;attach&gt;</code> — File Attachment
 <span class="s2">    data_file : &lt;attach@&gt;       # Large file in object store</span>
 <span class="s2">    """</span>
 </code></pre></div>
+<h3 id="npy-numpy-arrays-as-npy-files"><code>&lt;npy@&gt;</code> — NumPy Arrays as .npy Files<a class="headerlink" href="#npy-numpy-arrays-as-npy-files" title="Permanent link">¶</a></h3>
+<p>Stores NumPy arrays as standard <code>.npy</code> files with lazy loading. Returns <code>NpyRef</code> which provides metadata access (shape, dtype) without downloading.</p>
+<div class="highlight"><pre><span></span><code><span class="k">class</span><span class="w"> </span><span class="nc">Recording</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Computed</span><span class="p">):</span>
+    <span class="n">definition</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2">    -&gt; Session</span>
+<span class="s2">    ---</span>
+<span class="s2">    waveform : &lt;npy@&gt;           # Default store</span>
+<span class="s2">    spectrogram : &lt;npy@archive&gt; # Named store</span>
+<span class="s2">    """</span>
+</code></pre></div>
+<p><strong>Lazy access:</strong></p>
+<div class="highlight"><pre><span></span><code><span class="n">ref</span> <span class="o">=</span> <span class="p">(</span><span class="n">Recording</span> <span class="o">&amp;</span> <span class="n">key</span><span class="p">)</span><span class="o">.</span><span class="n">fetch1</span><span class="p">(</span><span class="s1">'waveform'</span><span class="p">)</span>
+<span class="n">ref</span><span class="o">.</span><span class="n">shape</span>   <span class="c1"># (1000, 32) — no download</span>
+<span class="n">ref</span><span class="o">.</span><span class="n">dtype</span>   <span class="c1"># float64 — no download</span>
+
+<span class="c1"># Explicit load</span>
+<span class="n">arr</span> <span class="o">=</span> <span class="n">ref</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
+
+<span class="c1"># Transparent numpy integration</span>
+<span class="n">result</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">ref</span><span class="p">)</span>  <span class="c1"># Downloads automatically</span>
+</code></pre></div>
+<p><strong>Key features:</strong></p>
+<ul>
+<li><strong>Portable format</strong>: Standard <code>.npy</code> readable by NumPy, MATLAB, etc.</li>
+<li><strong>Lazy loading</strong>: Shape/dtype available without I/O</li>
+<li><strong>Safe bulk fetch</strong>: Fetching many rows doesn't download until needed</li>
+<li><strong>Memory mapping</strong>: <code>ref.load(mmap_mode='r')</code> for random access to large arrays</li>
+</ul>
 <h3 id="object-path-addressed-storage"><code>&lt;object@&gt;</code> — Path-Addressed Storage<a class="headerlink" href="#object-path-addressed-storage" title="Permanent link">¶</a></h3>
 <p>For large/complex file structures (Zarr, HDF5). Path derived from primary key.</p>
 <div class="highlight"><pre><span></span><code><span class="k">class</span><span class="w"> </span><span class="nc">ProcessedData</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Computed</span><span class="p">):</span>
@@ -2829,17 +2953,25 @@ <h2 id="choosing-types">Choosing Types<a class="headerlink" href="#choosing-type
 </tr>
 <tr>
 <td>NumPy arrays (large)</td>
-<td><code>&lt;blob@&gt;</code></td>
+<td><code>&lt;npy@&gt;</code> or <code>&lt;blob@&gt;</code></td>
 </tr>
 <tr>
 <td>Files to attach</td>
 <td><code>&lt;attach&gt;</code> or <code>&lt;attach@&gt;</code></td>
 </tr>
 <tr>
-<td>Zarr/HDF5</td>
+<td>Zarr arrays</td>
+<td><code>&lt;zarr@&gt;</code> (plugin)</td>
+</tr>
+<tr>
+<td>Complex file structures</td>
 <td><code>&lt;object@&gt;</code></td>
 </tr>
 <tr>
+<td>Interactive visualizations</td>
+<td><code>&lt;figpack@&gt;</code> (plugin)</td>
+</tr>
+<tr>
 <td>File references (in-store)</td>
 <td><code>&lt;filepath@store&gt;</code></td>
 </tr>

llms-full.txt

@@ -1,6 +1,6 @@
 # DataJoint Documentation (Full)
 
-Generated: 2026-01-21 16:12:15 UTC
+Generated: 2026-01-21 19:27:05 UTC
 Commit: unknown
 Branch: unknown
 
@@ -2644,6 +2644,7 @@ graph TB
     subgraph "Layer 3: Codecs"
         blob["‹blob›"]
         attach["‹attach›"]
+        npy["‹npy@›"]
         object["‹object@›"]
         hash["‹hash@›"]
         custom["‹custom›"]
@@ -2665,6 +2666,7 @@ graph TB
 
     blob --> bytes
     attach --> bytes
+    npy --> json
     object --> json
     hash --> json
     bytes --> BLOB
@@ -2739,10 +2741,51 @@ Codec types use angle bracket notation:
 |-------|----------|--------------|---------|
 | `<blob>` | ✅ | ✅ `<blob@>` | Python object |
 | `<attach>` | ✅ | ✅ `<attach@>` | Local file path |
+| `<npy@>` | ❌ | ✅ | NpyRef (lazy) |
 | `<object@>` | ❌ | ✅ | ObjectRef |
 | `<hash@>` | ❌ | ✅ | bytes |
 | `<filepath@>` | ❌ | ✅ | ObjectRef |
 
+### Plugin Codecs
+
+Additional codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.
+
+| Package | Codec | Description | Repository |
+|---------|-------|-------------|------------|
+| `dj-zarr-codecs` | `<zarr@>` | Zarr arrays with lazy chunked access | [datajoint/dj-zarr-codecs](https://github.com/datajoint/dj-zarr-codecs) |
+| `dj-figpack-codecs` | `<figpack@>` | Interactive browser visualizations | [datajoint/dj-figpack-codecs](https://github.com/datajoint/dj-figpack-codecs) |
+| `dj-photon-codecs` | `<photon@>` | Photon imaging data formats | [datajoint/dj-photon-codecs](https://github.com/datajoint/dj-photon-codecs) |
+
+**Installation and discovery:**
+
+Plugin codecs use Python's entry point mechanism for automatic registration. Install the package and DataJoint discovers the codec automatically:
+
+```bash
+pip install dj-zarr-codecs
+```
+
+```python
+import datajoint as dj
+
+# Codec is available immediately after install
+@schema
+class Analysis(dj.Computed):
+    definition = """
+    -> Recording
+    ---
+    data : <zarr@store>
+    """
+```
+
+Packages declare their codecs in `pyproject.toml` under the `datajoint.codecs` entry point group:
+
+```toml
+[project.entry-points."datajoint.codecs"]
+zarr = "dj_zarr_codecs:ZarrCodec"
+```
+
+DataJoint loads these entry points on first use, making third-party codecs indistinguishable from built-ins.
+
 ### `<blob>` — Serialized Python Objects
 
 Stores NumPy arrays, dicts, lists, and other Python objects using DataJoint's custom binary serialization format.
@@ -2794,6 +2837,41 @@ class Config(dj.Manual):
     """
 ```
 
+### `<npy@>` — NumPy Arrays as .npy Files
+
+Stores NumPy arrays as standard `.npy` files with lazy loading. Returns `NpyRef` which provides metadata access (shape, dtype) without downloading.
+
+```python
+class Recording(dj.Computed):
+    definition = """
+    -> Session
+    ---
+    waveform : <npy@>           # Default store
+    spectrogram : <npy@archive> # Named store
+    """
+```
+
+**Lazy access:**
+
+```python
+ref = (Recording & key).fetch1('waveform')
+ref.shape   # (1000, 32) — no download
+ref.dtype   # float64 — no download
+
+# Explicit load
+arr = ref.load()
+
+# Transparent numpy integration
+result = np.mean(ref)  # Downloads automatically
+```
+
+**Key features:**
+
+- **Portable format**: Standard `.npy` readable by NumPy, MATLAB, etc.
+- **Lazy loading**: Shape/dtype available without I/O
+- **Safe bulk fetch**: Fetching many rows doesn't download until needed
+- **Memory mapping**: `ref.load(mmap_mode='r')` for random access to large arrays
+
 ### `<object@>` — Path-Addressed Storage
 
 For large/complex file structures (Zarr, HDF5). Path derived from primary key.
@@ -2872,9 +2950,11 @@ class Network(dj.Computed):
 | Small scalars | Core types (`int32`, `float64`) |
 | Short strings | `varchar(n)` |
 | NumPy arrays (small) | `<blob>` |
-| NumPy arrays (large) | `<blob@>` |
+| NumPy arrays (large) | `<npy@>` or `<blob@>` |
 | Files to attach | `<attach>` or `<attach@>` |
-| Zarr/HDF5 | `<object@>` |
+| Zarr arrays | `<zarr@>` (plugin) |
+| Complex file structures | `<object@>` |
+| Interactive visualizations | `<figpack@>` (plugin) |
 | File references (in-store) | `<filepath@store>` |
 | Custom objects | Custom codec |