590590</ a >
591591</ li >
592592< li class ="md-nav__item ">
593- < a class ="md-nav__link " href ="#object-path -addressed-storage ">
593+ < a class ="md-nav__link " href ="#object-schema -addressed-storage ">
594594< span class ="md-ellipsis ">
595595
596- <object@> — Path -Addressed Storage
596+ <object@> — Schema -Addressed Storage
597597
598598 </ span >
599599</ a >
24482448</ a >
24492449</ li >
24502450< li class ="md-nav__item ">
2451- < a class ="md-nav__link " href ="#object-path -addressed-storage ">
2451+ < a class ="md-nav__link " href ="#object-schema -addressed-storage ">
24522452< span class ="md-ellipsis ">
24532453
2454- <object@> — Path -Addressed Storage
2454+ <object@> — Schema -Addressed Storage
24552455
24562456 </ span >
24572457</ a >
@@ -2522,44 +2522,87 @@ <h2 id="three-layer-architecture">Three-Layer Architecture<a class="headerlink"
25222522< pre class ="mermaid "> < code > graph TB
25232523 subgraph "Layer 3: Codecs"
25242524 blob["‹blob›"]
2525+ blob_at["‹blob@›"]
25252526 attach["‹attach›"]
2527+ attach_at["‹attach@›"]
25262528 npy["‹npy@›"]
25272529 object["‹object@›"]
2530+ filepath["‹filepath@›"]
25282531 hash["‹hash@›"]
2529- custom ["‹custom ›"]
2532+ plugin ["‹plugin ›"]
25302533 end
25312534 subgraph "Layer 2: Core Types"
25322535 int32
25332536 float64
25342537 varchar
25352538 json
25362539 bytes
2540+ uuid
25372541 end
2538- subgraph "Layer 1: Native"
2539- INT["INT"]
2540- DOUBLE["DOUBLE"]
2541- VARCHAR ["VARCHAR"]
2542+ subgraph "Layer 1: Native Types (MySQL / PostgreSQL) "
2543+ INT["INT / INTEGER "]
2544+ DOUBLE["DOUBLE / DOUBLE PRECISION "]
2545+ VARCHAR_N ["VARCHAR"]
25422546 JSON_N["JSON"]
2543- BLOB["LONGBLOB"]
2547+ BYTES_N["LONGBLOB / BYTEA"]
2548+ UUID_N["BINARY(16) / UUID"]
25442549 end
25452550
25462551 blob --> bytes
2552+ blob_at --> hash
25472553 attach --> bytes
2554+ attach_at --> hash
2555+ hash --> json
25482556 npy --> json
25492557 object --> json
2550- hash --> json
2551- bytes --> BLOB
2558+ filepath --> json
2559+
2560+ bytes --> BYTES_N
25522561 json --> JSON_N
25532562 int32 --> INT
25542563 float64 --> DOUBLE
2555- varchar --> VARCHAR
2564+ varchar --> VARCHAR_N
2565+ uuid --> UUID_N
25562566</ code > </ pre >
2567+ < p > Core types provide < strong > portability</ strong > — the same table definition works on both MySQL and PostgreSQL. For example, < code > bytes</ code > maps to < code > LONGBLOB</ code > on MySQL but < code > BYTEA</ code > on PostgreSQL; < code > uuid</ code > maps to < code > BINARY(16)</ code > on MySQL but native < code > UUID</ code > on PostgreSQL. Native types can be used directly but sacrifice cross-backend compatibility.</ p >
25572568< h2 id ="layer-1-native-database-types "> Layer 1: Native Database Types< a class ="headerlink " href ="#layer-1-native-database-types " title ="Permanent link "> ¶</ a > </ h2 >
2558- < p > Backend-specific types (MySQL, PostgreSQL). < strong > Discouraged for direct use.</ strong > </ p >
2559- < div class ="highlight "> < pre > < span > </ span > < code > < span class ="c1 "> # Native types (avoid)</ span >
2560- < span class ="n "> column</ span > < span class ="p "> :</ span > < span class ="n "> TINYINT</ span > < span class ="n "> UNSIGNED</ span >
2561- < span class ="n "> column</ span > < span class ="p "> :</ span > < span class ="n "> MEDIUMBLOB</ span >
2569+ < p > Backend-specific types. < strong > Can be used directly at the cost of portability.</ strong > </ p >
2570+ < div class ="highlight "> < pre > < span > </ span > < code > < span class ="c1 "> # Native types — work but not portable</ span >
2571+ < span class ="n "> column</ span > < span class ="p "> :</ span > < span class ="n "> TINYINT</ span > < span class ="n "> UNSIGNED</ span > < span class ="c1 "> # MySQL only</ span >
2572+ < span class ="n "> column</ span > < span class ="p "> :</ span > < span class ="n "> MEDIUMBLOB</ span > < span class ="c1 "> # MySQL only (use BYTEA on PostgreSQL)</ span >
2573+ < span class ="n "> column</ span > < span class ="p "> :</ span > < span class ="n "> SERIAL</ span > < span class ="c1 "> # PostgreSQL only</ span >
25622574</ code > </ pre > </ div >
2575+ < table >
2576+ < thead >
2577+ < tr >
2578+ < th > MySQL</ th >
2579+ < th > PostgreSQL</ th >
2580+ < th > Portable Alternative</ th >
2581+ </ tr >
2582+ </ thead >
2583+ < tbody >
2584+ < tr >
2585+ < td > < code > LONGBLOB</ code > </ td >
2586+ < td > < code > BYTEA</ code > </ td >
2587+ < td > < code > bytes</ code > </ td >
2588+ </ tr >
2589+ < tr >
2590+ < td > < code > BINARY(16)</ code > </ td >
2591+ < td > < code > UUID</ code > </ td >
2592+ < td > < code > uuid</ code > </ td >
2593+ </ tr >
2594+ < tr >
2595+ < td > < code > SMALLINT</ code > </ td >
2596+ < td > < code > SMALLINT</ code > </ td >
2597+ < td > < code > int16</ code > </ td >
2598+ </ tr >
2599+ < tr >
2600+ < td > < code > DOUBLE</ code > </ td >
2601+ < td > < code > DOUBLE PRECISION</ code > </ td >
2602+ < td > < code > float64</ code > </ td >
2603+ </ tr >
2604+ </ tbody >
2605+ </ table >
25632606< h2 id ="layer-2-core-datajoint-types "> Layer 2: Core DataJoint Types< a class ="headerlink " href ="#layer-2-core-datajoint-types " title ="Permanent link "> ¶</ a > </ h2 >
25642607< p > Standardized, scientist-friendly types that work identically across backends.</ p >
25652608< h3 id ="numeric-types "> Numeric Types< a class ="headerlink " href ="#numeric-types " title ="Permanent link "> ¶</ a > </ h3 >
@@ -2684,6 +2727,7 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
26842727< th > Codec</ th >
26852728< th > Database</ th >
26862729< th > Object Store</ th >
2730+ < th > Addressing</ th >
26872731< th > Returns</ th >
26882732</ tr >
26892733</ thead >
@@ -2692,42 +2736,48 @@ <h3 id="built-in-codecs">Built-in Codecs<a class="headerlink" href="#built-in-co
26922736< td > < code > <blob></ code > </ td >
26932737< td > ✅</ td >
26942738< td > ✅ < code > <blob@></ code > </ td >
2739+ < td > Hash</ td >
26952740< td > Python object</ td >
26962741</ tr >
26972742< tr >
26982743< td > < code > <attach></ code > </ td >
26992744< td > ✅</ td >
27002745< td > ✅ < code > <attach@></ code > </ td >
2746+ < td > Hash</ td >
27012747< td > Local file path</ td >
27022748</ tr >
27032749< tr >
27042750< td > < code > <npy@></ code > </ td >
27052751< td > ❌</ td >
27062752< td > ✅</ td >
2753+ < td > Schema</ td >
27072754< td > NpyRef (lazy)</ td >
27082755</ tr >
27092756< tr >
27102757< td > < code > <object@></ code > </ td >
27112758< td > ❌</ td >
27122759< td > ✅</ td >
2760+ < td > Schema</ td >
27132761< td > ObjectRef</ td >
27142762</ tr >
27152763< tr >
27162764< td > < code > <hash@></ code > </ td >
27172765< td > ❌</ td >
27182766< td > ✅</ td >
2767+ < td > Hash</ td >
27192768< td > bytes</ td >
27202769</ tr >
27212770< tr >
27222771< td > < code > <filepath@></ code > </ td >
27232772< td > ❌</ td >
27242773< td > ✅</ td >
2774+ < td > —</ td >
27252775< td > ObjectRef</ td >
27262776</ tr >
27272777</ tbody >
27282778</ table >
27292779< h3 id ="plugin-codecs "> Plugin Codecs< a class ="headerlink " href ="#plugin-codecs " title ="Permanent link "> ¶</ a > </ h3 >
2730- < p > Additional codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.</ p >
2780+ < p > Additional schema-addressed codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.</ p >
27312781< table >
27322782< thead >
27332783< tr >
@@ -2741,19 +2791,19 @@ <h3 id="plugin-codecs">Plugin Codecs<a class="headerlink" href="#plugin-codecs"
27412791< tr >
27422792< td > < code > dj-zarr-codecs</ code > </ td >
27432793< td > < code > <zarr@></ code > </ td >
2744- < td > Zarr arrays with lazy chunked access</ td >
2794+ < td > Schema-addressed Zarr arrays with lazy chunked access</ td >
27452795< td > < a href ="https://github.com/datajoint/dj-zarr-codecs "> datajoint/dj-zarr-codecs</ a > </ td >
27462796</ tr >
27472797< tr >
27482798< td > < code > dj-figpack-codecs</ code > </ td >
27492799< td > < code > <figpack@></ code > </ td >
2750- < td > Interactive browser visualizations</ td >
2800+ < td > Schema-addressed interactive browser visualizations</ td >
27512801< td > < a href ="https://github.com/datajoint/dj-figpack-codecs "> datajoint/dj-figpack-codecs</ a > </ td >
27522802</ tr >
27532803< tr >
27542804< td > < code > dj-photon-codecs</ code > </ td >
27552805< td > < code > <photon@></ code > </ td >
2756- < td > Photon imaging data formats</ td >
2806+ < td > Schema-addressed photon imaging data formats</ td >
27572807< td > < a href ="https://github.com/datajoint/dj-photon-codecs "> datajoint/dj-photon-codecs</ a > </ td >
27582808</ tr >
27592809</ tbody >
@@ -2824,7 +2874,7 @@ <h3 id="attach-file-attachments"><code><attach></code> — File Attachment
28242874< span class ="s2 "> """</ span >
28252875</ code > </ pre > </ div >
28262876< h3 id ="npy-numpy-arrays-as-npy-files "> < code > <npy@></ code > — NumPy Arrays as .npy Files< a class ="headerlink " href ="#npy-numpy-arrays-as-npy-files " title ="Permanent link "> ¶</ a > </ h3 >
2827- < p > Stores NumPy arrays as standard < code > .npy</ code > files with lazy loading . Returns < code > NpyRef</ code > which provides metadata access (shape, dtype) without downloading.</ p >
2877+ < p > Schema-addressed storage for NumPy arrays as standard < code > .npy</ code > files. Returns < code > NpyRef</ code > which provides metadata access (shape, dtype) without downloading.</ p >
28282878< div class ="highlight "> < pre > < span > </ span > < code > < span class ="k "> class</ span > < span class ="w "> </ span > < span class ="nc "> Recording</ span > < span class ="p "> (</ span > < span class ="n "> dj</ span > < span class ="o "> .</ span > < span class ="n "> Computed</ span > < span class ="p "> ):</ span >
28292879 < span class ="n "> definition</ span > < span class ="o "> =</ span > < span class ="s2 "> """</ span >
28302880< span class ="s2 "> -> Session</ span >
@@ -2851,15 +2901,16 @@ <h3 id="npy-numpy-arrays-as-npy-files"><code><npy@></code> — NumPy Array
28512901< li > < strong > Safe bulk fetch</ strong > : Fetching many rows doesn't download until needed</ li >
28522902< li > < strong > Memory mapping</ strong > : < code > ref.load(mmap_mode='r')</ code > for random access to large arrays</ li >
28532903</ ul >
2854- < h3 id ="object-path -addressed-storage "> < code > <object@></ code > — Path -Addressed Storage< a class ="headerlink " href ="#object-path -addressed-storage " title ="Permanent link "> ¶</ a > </ h3 >
2855- < p > For large/complex file structures (Zarr, HDF5) . Path derived from primary key .</ p >
2904+ < h3 id ="object-schema -addressed-storage "> < code > <object@></ code > — Schema -Addressed Storage< a class ="headerlink " href ="#object-schema -addressed-storage " title ="Permanent link "> ¶</ a > </ h3 >
2905+ < p > Schema-addressed storage for files and folders . Path mirrors the database structure: < code > {schema}/{table}/{pk}/{attribute} </ code > .</ p >
28562906< div class ="highlight "> < pre > < span > </ span > < code > < span class ="k "> class</ span > < span class ="w "> </ span > < span class ="nc "> ProcessedData</ span > < span class ="p "> (</ span > < span class ="n "> dj</ span > < span class ="o "> .</ span > < span class ="n "> Computed</ span > < span class ="p "> ):</ span >
28572907 < span class ="n "> definition</ span > < span class ="o "> =</ span > < span class ="s2 "> """</ span >
28582908< span class ="s2 "> -> Recording</ span >
28592909< span class ="s2 "> ---</ span >
2860- < span class ="s2 "> zarr_data : <object@> # Stored at </ span > < span class ="si "> {schema}</ span > < span class ="s2 "> /</ span > < span class ="si "> {table}</ span > < span class ="s2 "> /</ span > < span class ="si "> {pk}</ span > < span class ="s2 "> /</ span >
2910+ < span class ="s2 "> results : <object@> # Stored at </ span > < span class ="si "> {schema}</ span > < span class ="s2 "> /</ span > < span class ="si "> {table}</ span > < span class ="s2 "> /</ span > < span class ="si "> {pk}</ span > < span class ="s2 "> /results /</ span >
28612911< span class ="s2 "> """</ span >
28622912</ code > </ pre > </ div >
2913+ < p > Accepts files, folders, or bytes. Returns < code > ObjectRef</ code > for lazy access.</ p >
28632914< h3 id ="filepathstore-portable-references "> < code > <filepath@store></ code > — Portable References< a class ="headerlink " href ="#filepathstore-portable-references " title ="Permanent link "> ¶</ a > </ h3 >
28642915< p > References to independently-managed files with portable paths.</ p >
28652916< div class ="highlight "> < pre > < span > </ span > < code > < span class ="k "> class</ span > < span class ="w "> </ span > < span class ="nc "> RawData</ span > < span class ="p "> (</ span > < span class ="n "> dj</ span > < span class ="o "> .</ span > < span class ="n "> Manual</ span > < span class ="p "> ):</ span >
@@ -2870,12 +2921,16 @@ <h3 id="filepathstore-portable-references"><code><filepath@store></code>
28702921< span class ="s2 "> """</ span >
28712922</ code > </ pre > </ div >
28722923< h2 id ="storage-modes "> Storage Modes< a class ="headerlink " href ="#storage-modes " title ="Permanent link "> ¶</ a > </ h2 >
2924+ < p > Object store codecs use one of two addressing schemes:</ p >
2925+ < p > < strong > Hash-addressed</ strong > — Path derived from content hash (e.g., < code > _hash/ab/cd/abcd1234...</ code > ). Provides automatic deduplication—identical content stored once. Used by < code > <blob@></ code > , < code > <attach@></ code > , < code > <hash@></ code > .</ p >
2926+ < p > < strong > Schema-addressed</ strong > — Path mirrors database structure: < code > {schema}/{table}/{pk}/{attribute}</ code > . Human-readable, browsable paths that reflect your data organization. No deduplication. Used by < code > <object@></ code > , < code > <npy@></ code > , and plugin codecs (< code > <zarr@></ code > , < code > <figpack@></ code > , < code > <photon@></ code > ).</ p >
28732927< table >
28742928< thead >
28752929< tr >
28762930< th > Mode</ th >
28772931< th > Database</ th >
28782932< th > Object Store</ th >
2933+ < th > Deduplication</ th >
28792934< th > Use Case</ th >
28802935</ tr >
28812936</ thead >
@@ -2884,19 +2939,22 @@ <h2 id="storage-modes">Storage Modes<a class="headerlink" href="#storage-modes"
28842939< td > Database</ td >
28852940< td > Data</ td >
28862941< td > —</ td >
2942+ < td > —</ td >
28872943< td > Small data</ td >
28882944</ tr >
28892945< tr >
28902946< td > Hash-addressed</ td >
28912947< td > Metadata</ td >
2892- < td > Deduplicated</ td >
2948+ < td > Content hash path</ td >
2949+ < td > ✅ Automatic</ td >
28932950< td > Large/repeated data</ td >
28942951</ tr >
28952952< tr >
2896- < td > Path -addressed</ td >
2953+ < td > Schema -addressed</ td >
28972954< td > Metadata</ td >
2898- < td > PK-based path</ td >
2899- < td > Complex files</ td >
2955+ < td > Schema-mirrored path</ td >
2956+ < td > ❌ None</ td >
2957+ < td > Complex files, browsable storage</ td >
29002958</ tr >
29012959</ tbody >
29022960</ table >
0 commit comments