Skip to content

Commit 649f831

Browse files
committed
Update documentation
1 parent dcfaf63 commit 649f831

4 files changed

Lines changed: 153 additions & 67 deletions

File tree

explanation/type-system/index.html

Lines changed: 87 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -590,10 +590,10 @@
590590
</a>
591591
</li>
592592
<li class="md-nav__item">
593-
<a class="md-nav__link" href="#object-path-addressed-storage">
593+
<a class="md-nav__link" href="#object-schema-addressed-storage">
594594
<span class="md-ellipsis">
595595

596-
&lt;object@&gt; — Path-Addressed Storage
596+
&lt;object@&gt; — Schema-Addressed Storage
597597

598598
</span>
599599
</a>
@@ -2448,10 +2448,10 @@
24482448
</a>
24492449
</li>
24502450
<li class="md-nav__item">
2451-
<a class="md-nav__link" href="#object-path-addressed-storage">
2451+
<a class="md-nav__link" href="#object-schema-addressed-storage">
24522452
<span class="md-ellipsis">
24532453

2454-
&lt;object@&gt; — Path-Addressed Storage
2454+
&lt;object@&gt; — Schema-Addressed Storage
24552455

24562456
</span>
24572457
</a>
@@ -2522,44 +2522,87 @@ <h2 id="three-layer-architecture">Three-Layer Architecture<a class="headerlink"
25222522
<pre class="mermaid"><code>graph TB
25232523
subgraph "Layer 3: Codecs"
25242524
blob["‹blob›"]
2525+
blob_at["‹blob@›"]
25252526
attach["‹attach›"]
2527+
attach_at["‹attach@›"]
25262528
npy["‹npy@›"]
25272529
object["‹object@›"]
2530+
filepath["‹filepath@›"]
25282531
hash["‹hash@›"]
2529-
custom["‹custom›"]
2532+
plugin["‹plugin›"]
25302533
end
25312534
subgraph "Layer 2: Core Types"
25322535
int32
25332536
float64
25342537
varchar
25352538
json
25362539
bytes
2540+
uuid
25372541
end
2538-
subgraph "Layer 1: Native"
2539-
INT["INT"]
2540-
DOUBLE["DOUBLE"]
2541-
VARCHAR["VARCHAR"]
2542+
subgraph "Layer 1: Native Types (MySQL / PostgreSQL)"
2543+
INT["INT / INTEGER"]
2544+
DOUBLE["DOUBLE / DOUBLE PRECISION"]
2545+
VARCHAR_N["VARCHAR"]
25422546
JSON_N["JSON"]
2543-
BLOB["LONGBLOB"]
2547+
BYTES_N["LONGBLOB / BYTEA"]
2548+
UUID_N["BINARY(16) / UUID"]
25442549
end
25452550

25462551
blob --&gt; bytes
2552+
blob_at --&gt; hash
25472553
attach --&gt; bytes
2554+
attach_at --&gt; hash
2555+
hash --&gt; json
25482556
npy --&gt; json
25492557
object --&gt; json
2550-
hash --&gt; json
2551-
bytes --&gt; BLOB
2558+
filepath --&gt; json
2559+
2560+
bytes --&gt; BYTES_N
25522561
json --&gt; JSON_N
25532562
int32 --&gt; INT
25542563
float64 --&gt; DOUBLE
2555-
varchar --&gt; VARCHAR
2564+
varchar --&gt; VARCHAR_N
2565+
uuid --&gt; UUID_N
25562566
</code></pre>
2567+
<p>Core types provide <strong>portability</strong> — the same table definition works on both MySQL and PostgreSQL. For example, <code>bytes</code> maps to <code>LONGBLOB</code> on MySQL but <code>BYTEA</code> on PostgreSQL; <code>uuid</code> maps to <code>BINARY(16)</code> on MySQL but native <code>UUID</code> on PostgreSQL. Native types can be used directly but sacrifice cross-backend compatibility.</p>
25572568
<h2 id="layer-1-native-database-types">Layer 1: Native Database Types<a class="headerlink" href="#layer-1-native-database-types" title="Permanent link"></a></h2>
2558-
<p>Backend-specific types (MySQL, PostgreSQL). <strong>Discouraged for direct use.</strong></p>
2559-
<div class="highlight"><pre><span></span><code><span class="c1"># Native types (avoid)</span>
2560-
<span class="n">column</span> <span class="p">:</span> <span class="n">TINYINT</span> <span class="n">UNSIGNED</span>
2561-
<span class="n">column</span> <span class="p">:</span> <span class="n">MEDIUMBLOB</span>
2569+
<p>Backend-specific types. <strong>Can be used directly at the cost of portability.</strong></p>
2570+
<div class="highlight"><pre><span></span><code><span class="c1"># Native types — work but not portable</span>
2571+
<span class="n">column</span> <span class="p">:</span> <span class="n">TINYINT</span> <span class="n">UNSIGNED</span> <span class="c1"># MySQL only</span>
2572+
<span class="n">column</span> <span class="p">:</span> <span class="n">MEDIUMBLOB</span> <span class="c1"># MySQL only (use BYTEA on PostgreSQL)</span>
2573+
<span class="n">column</span> <span class="p">:</span> <span class="n">SERIAL</span> <span class="c1"># PostgreSQL only</span>
25622574
</code></pre></div>
2575+
<table>
2576+
<thead>
2577+
<tr>
2578+
<th>MySQL</th>
2579+
<th>PostgreSQL</th>
2580+
<th>Portable Alternative</th>
2581+
</tr>
2582+
</thead>
2583+
<tbody>
2584+
<tr>
2585+
<td><code>LONGBLOB</code></td>
2586+
<td><code>BYTEA</code></td>
2587+
<td><code>bytes</code></td>
2588+
</tr>
2589+
<tr>
2590+
<td><code>BINARY(16)</code></td>
2591+
<td><code>UUID</code></td>
2592+
<td><code>uuid</code></td>
2593+
</tr>
2594+
<tr>
2595+
<td><code>SMALLINT</code></td>
2596+
<td><code>SMALLINT</code></td>
2597+
<td><code>int16</code></td>
2598+
</tr>
2599+
<tr>
2600+
<td><code>DOUBLE</code></td>
2601+
<td><code>DOUBLE PRECISION</code></td>
2602+
<td><code>float64</code></td>
2603+
</tr>
2604+
</tbody>
2605+
</table>
25632606
<h2 id="layer-2-core-datajoint-types">Layer 2: Core DataJoint Types<a class="headerlink" href="#layer-2-core-datajoint-types" title="Permanent link"></a></h2>
25642607
<p>Standardized, scientist-friendly types that work identically across backends.</p>
25652608
<h3 id="numeric-types">Numeric Types<a class="headerlink" href="#numeric-types" title="Permanent link"></a></h3>
@@ -2684,6 +2727,7 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
26842727
<th>Codec</th>
26852728
<th>Database</th>
26862729
<th>Object Store</th>
2730+
<th>Addressing</th>
26872731
<th>Returns</th>
26882732
</tr>
26892733
</thead>
@@ -2692,42 +2736,48 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
26922736
<td><code>&lt;blob&gt;</code></td>
26932737
<td></td>
26942738
<td><code>&lt;blob@&gt;</code></td>
2739+
<td>Hash</td>
26952740
<td>Python object</td>
26962741
</tr>
26972742
<tr>
26982743
<td><code>&lt;attach&gt;</code></td>
26992744
<td></td>
27002745
<td><code>&lt;attach@&gt;</code></td>
2746+
<td>Hash</td>
27012747
<td>Local file path</td>
27022748
</tr>
27032749
<tr>
27042750
<td><code>&lt;npy@&gt;</code></td>
27052751
<td></td>
27062752
<td></td>
2753+
<td>Schema</td>
27072754
<td>NpyRef (lazy)</td>
27082755
</tr>
27092756
<tr>
27102757
<td><code>&lt;object@&gt;</code></td>
27112758
<td></td>
27122759
<td></td>
2760+
<td>Schema</td>
27132761
<td>ObjectRef</td>
27142762
</tr>
27152763
<tr>
27162764
<td><code>&lt;hash@&gt;</code></td>
27172765
<td></td>
27182766
<td></td>
2767+
<td>Hash</td>
27192768
<td>bytes</td>
27202769
</tr>
27212770
<tr>
27222771
<td><code>&lt;filepath@&gt;</code></td>
27232772
<td></td>
27242773
<td></td>
2774+
<td></td>
27252775
<td>ObjectRef</td>
27262776
</tr>
27272777
</tbody>
27282778
</table>
27292779
<h3 id="plugin-codecs">Plugin Codecs<a class="headerlink" href="#plugin-codecs" title="Permanent link"></a></h3>
2730-
<p>Additional codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.</p>
2780+
<p>Additional schema-addressed codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.</p>
27312781
<table>
27322782
<thead>
27332783
<tr>
@@ -2741,19 +2791,19 @@ <h3 id="plugin-codecs">Plugin Codecs<a class="headerlink" href="#plugin-codecs"
27412791
<tr>
27422792
<td><code>dj-zarr-codecs</code></td>
27432793
<td><code>&lt;zarr@&gt;</code></td>
2744-
<td>Zarr arrays with lazy chunked access</td>
2794+
<td>Schema-addressed Zarr arrays with lazy chunked access</td>
27452795
<td><a href="https://github.com/datajoint/dj-zarr-codecs">datajoint/dj-zarr-codecs</a></td>
27462796
</tr>
27472797
<tr>
27482798
<td><code>dj-figpack-codecs</code></td>
27492799
<td><code>&lt;figpack@&gt;</code></td>
2750-
<td>Interactive browser visualizations</td>
2800+
<td>Schema-addressed interactive browser visualizations</td>
27512801
<td><a href="https://github.com/datajoint/dj-figpack-codecs">datajoint/dj-figpack-codecs</a></td>
27522802
</tr>
27532803
<tr>
27542804
<td><code>dj-photon-codecs</code></td>
27552805
<td><code>&lt;photon@&gt;</code></td>
2756-
<td>Photon imaging data formats</td>
2806+
<td>Schema-addressed photon imaging data formats</td>
27572807
<td><a href="https://github.com/datajoint/dj-photon-codecs">datajoint/dj-photon-codecs</a></td>
27582808
</tr>
27592809
</tbody>
@@ -2824,7 +2874,7 @@ <h3 id="attach-file-attachments"><code>&lt;attach&gt;</code> — File Attachment
28242874
<span class="s2"> """</span>
28252875
</code></pre></div>
28262876
<h3 id="npy-numpy-arrays-as-npy-files"><code>&lt;npy@&gt;</code> — NumPy Arrays as .npy Files<a class="headerlink" href="#npy-numpy-arrays-as-npy-files" title="Permanent link"></a></h3>
2827-
<p>Stores NumPy arrays as standard <code>.npy</code> files with lazy loading. Returns <code>NpyRef</code> which provides metadata access (shape, dtype) without downloading.</p>
2877+
<p>Schema-addressed storage for NumPy arrays as standard <code>.npy</code> files. Returns <code>NpyRef</code> which provides metadata access (shape, dtype) without downloading.</p>
28282878
<div class="highlight"><pre><span></span><code><span class="k">class</span><span class="w"> </span><span class="nc">Recording</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Computed</span><span class="p">):</span>
28292879
<span class="n">definition</span> <span class="o">=</span> <span class="s2">"""</span>
28302880
<span class="s2"> -&gt; Session</span>
@@ -2851,15 +2901,16 @@ <h3 id="npy-numpy-arrays-as-npy-files"><code>&lt;npy@&gt;</code> — NumPy Array
28512901
<li><strong>Safe bulk fetch</strong>: Fetching many rows doesn't download until needed</li>
28522902
<li><strong>Memory mapping</strong>: <code>ref.load(mmap_mode='r')</code> for random access to large arrays</li>
28532903
</ul>
2854-
<h3 id="object-path-addressed-storage"><code>&lt;object@&gt;</code>Path-Addressed Storage<a class="headerlink" href="#object-path-addressed-storage" title="Permanent link"></a></h3>
2855-
<p>For large/complex file structures (Zarr, HDF5). Path derived from primary key.</p>
2904+
<h3 id="object-schema-addressed-storage"><code>&lt;object@&gt;</code>Schema-Addressed Storage<a class="headerlink" href="#object-schema-addressed-storage" title="Permanent link"></a></h3>
2905+
<p>Schema-addressed storage for files and folders. Path mirrors the database structure: <code>{schema}/{table}/{pk}/{attribute}</code>.</p>
28562906
<div class="highlight"><pre><span></span><code><span class="k">class</span><span class="w"> </span><span class="nc">ProcessedData</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Computed</span><span class="p">):</span>
28572907
<span class="n">definition</span> <span class="o">=</span> <span class="s2">"""</span>
28582908
<span class="s2"> -&gt; Recording</span>
28592909
<span class="s2"> ---</span>
2860-
<span class="s2"> zarr_data : &lt;object@&gt; # Stored at </span><span class="si">{schema}</span><span class="s2">/</span><span class="si">{table}</span><span class="s2">/</span><span class="si">{pk}</span><span class="s2">/</span>
2910+
<span class="s2"> results : &lt;object@&gt; # Stored at </span><span class="si">{schema}</span><span class="s2">/</span><span class="si">{table}</span><span class="s2">/</span><span class="si">{pk}</span><span class="s2">/results/</span>
28612911
<span class="s2"> """</span>
28622912
</code></pre></div>
2913+
<p>Accepts files, folders, or bytes. Returns <code>ObjectRef</code> for lazy access.</p>
28632914
<h3 id="filepathstore-portable-references"><code>&lt;filepath@store&gt;</code> — Portable References<a class="headerlink" href="#filepathstore-portable-references" title="Permanent link"></a></h3>
28642915
<p>References to independently-managed files with portable paths.</p>
28652916
<div class="highlight"><pre><span></span><code><span class="k">class</span><span class="w"> </span><span class="nc">RawData</span><span class="p">(</span><span class="n">dj</span><span class="o">.</span><span class="n">Manual</span><span class="p">):</span>
@@ -2870,12 +2921,16 @@ <h3 id="filepathstore-portable-references"><code>&lt;filepath@store&gt;</code>
28702921
<span class="s2"> """</span>
28712922
</code></pre></div>
28722923
<h2 id="storage-modes">Storage Modes<a class="headerlink" href="#storage-modes" title="Permanent link"></a></h2>
2924+
<p>Object store codecs use one of two addressing schemes:</p>
2925+
<p><strong>Hash-addressed</strong> — Path derived from content hash (e.g., <code>_hash/ab/cd/abcd1234...</code>). Provides automatic deduplication—identical content stored once. Used by <code>&lt;blob@&gt;</code>, <code>&lt;attach@&gt;</code>, <code>&lt;hash@&gt;</code>.</p>
2926+
<p><strong>Schema-addressed</strong> — Path mirrors database structure: <code>{schema}/{table}/{pk}/{attribute}</code>. Human-readable, browsable paths that reflect your data organization. No deduplication. Used by <code>&lt;object@&gt;</code>, <code>&lt;npy@&gt;</code>, and plugin codecs (<code>&lt;zarr@&gt;</code>, <code>&lt;figpack@&gt;</code>, <code>&lt;photon@&gt;</code>).</p>
28732927
<table>
28742928
<thead>
28752929
<tr>
28762930
<th>Mode</th>
28772931
<th>Database</th>
28782932
<th>Object Store</th>
2933+
<th>Deduplication</th>
28792934
<th>Use Case</th>
28802935
</tr>
28812936
</thead>
@@ -2884,19 +2939,22 @@ <h2 id="storage-modes">Storage Modes<a class="headerlink" href="#storage-modes"
28842939
<td>Database</td>
28852940
<td>Data</td>
28862941
<td></td>
2942+
<td></td>
28872943
<td>Small data</td>
28882944
</tr>
28892945
<tr>
28902946
<td>Hash-addressed</td>
28912947
<td>Metadata</td>
2892-
<td>Deduplicated</td>
2948+
<td>Content hash path</td>
2949+
<td>✅ Automatic</td>
28932950
<td>Large/repeated data</td>
28942951
</tr>
28952952
<tr>
2896-
<td>Path-addressed</td>
2953+
<td>Schema-addressed</td>
28972954
<td>Metadata</td>
2898-
<td>PK-based path</td>
2899-
<td>Complex files</td>
2955+
<td>Schema-mirrored path</td>
2956+
<td>❌ None</td>
2957+
<td>Complex files, browsable storage</td>
29002958
</tr>
29012959
</tbody>
29022960
</table>

0 commit comments

Comments
 (0)