You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<p>These components work together: code defines the schema and computations, the database tracks all metadata and relationships, and object storage holds the large scientific data files.</p>
6533
6533
<h2id="pipeline-as-a-dag">Pipeline as a DAG<aclass="headerlink" href="#pipeline-as-a-dag" title="Permanent link">¶</a></h2>
6534
-
<p>A DataJoint pipeline forms a <strong>Directed Acyclic Graph (DAG)</strong> at two levels:</p>
6535
-
<p><imgalt="Pipeline DAG Structure" src="../../images/pipeline-modules.svg" /></p>
6536
-
<p><strong>Nodes</strong> represent Python modules, which correspond to database schemas — the dashed clusters in the diagram above.</p>
6537
-
<p><strong>Edges</strong> represent:</p>
6538
-
<ul>
6539
-
<li>Python import dependencies between modules</li>
6540
-
<li>Bundles of foreign key references between schemas</li>
6541
-
</ul>
6542
-
<p>This dual structure ensures that both code dependencies and data dependencies flow in the same direction.</p>
6534
+
<p>A DataJoint pipeline forms a <strong>Directed Acyclic Graph (DAG)</strong> at two levels, and the same
6535
+
pipeline can be viewed at either one.</p>
6536
+
<p><strong>Module level — the collapsed view.</strong> Each node is a Python module, which corresponds to a
6537
+
database schema. There is one edge per pair of schemas, standing for a bundle of dependencies:
6538
+
every foreign key reference between the two schemas' tables, together with the Python import
6539
+
dependency between their modules. A collapsed edge records only that the bundle exists — line
6540
+
weight and line style describe an individual foreign key, so they are read at the table level,
6541
+
not here. Below, <code>lab → session</code> and <code>session → imaging</code> each bundle two foreign keys,
6542
+
<code>reference</code> supplies both <code>session</code> and <code>imaging</code>, and fourteen tables (including parts)
6543
+
collapse to four nodes.</p>
6544
+
<p><imgalt="Pipeline DAG at the module level" src="../../images/pipeline-modules-collapsed.svg" /></p>
6545
+
<p><strong>Table level — the expanded view.</strong> Each node is a table; the dashed clusters group the
6546
+
tables of each module. Each edge is an individual foreign key constraint — the two foreign
6547
+
keys that cross the <code>session → imaging</code> boundary (<code>Scan → ScanQuality</code> and
6548
+
<code>ScanInfo → MotionCorrection</code>) appear separately here, having collapsed into the single
6549
+
bundled edge above; likewise <code>Subject → Session</code> and <code>User → Session</code> across
6550
+
<code>lab → session</code>. Each edge here carries the full notation — line weight for cardinality, line
6551
+
style for whether the reference is part of the child's primary key — as specified in
<p>This model treats the database schema as an <strong>executable workflow specification</strong>—defining not just what data exists but when and how it comes into existence.</p>
<p>Each schema corresponds to a dedicated Python module. The module import structure mirrors the foreign key dependencies between schemas:</p>
6566
-
<p>Within a schema, tables of different tiers form their own DAG — here, a <code>scan</code> schema with lookup, manual, and imported tables, including a master table with its part tables:</p>
<p>Each schema corresponds to a dedicated Python module, and the module import structure mirrors the foreign key dependencies between schemas. The pipeline above is organized as:</p>
<p>Within a schema, tables form their own DAG. Drilling into the <code>imaging</code> module: computed tables, including two masters with their part tables — <code>Segmentation</code> with <code>Segmentation.Roi</code>, and <code>Fluorescence</code> with <code>Fluorescence.Trace</code>, whose rows reference individual ROIs:</p>
<p>For practical guidance on organizing multi-schema pipelines, configuring repositories, and managing team access, see <ahref="../../how-to/manage-pipeline-project/">Manage a Pipeline Project</a>.</p>
<p>Scientific data often includes large objects—images, recordings, time series, instrument outputs—that don't fit efficiently in relational tables. DataJoint addresses this through <strong>Object-Augmented Schemas (OAS)</strong>, a hybrid storage architecture that preserves relational semantics while handling arbitrarily large data.</p>
<p>When a database row is deleted, its associated stored objects are garbage-collected. Foreign key cascades work correctly—delete upstream data and downstream results (including their objects) disappear. The database and object store remain synchronized without manual cleanup.</p>
6620
6633
<p><strong>5. Multiple storage tiers support diverse access patterns.</strong></p>
6621
6634
<p>Different attributes can route to different stores:</p>
<spanclass="s2"> raw_data : <blob@fast> # Hot storage for active analysis</span>
6639
+
<spanclass="s2"> nframes : int32</span>
6640
+
<spanclass="s2"> fps : float32</span>
6641
+
<spanclass="s2"> raw_movie : <blob@fast> # Hot storage for active analysis</span>
6627
6642
<spanclass="s2"> archive : <blob@cold> # Cold storage for long-term retention</span>
6628
6643
<spanclass="s2"> """</span>
6629
6644
</code></pre></div>
6645
+
<p>Here the pipeline's <code>ScanInfo</code> table keeps its scalar metadata (<code>nframes</code>, <code>fps</code>) in the database while routing its payloads to two different stores.</p>
6630
6646
<p>This architecture lets teams work with terabyte-scale datasets while retaining the query power, integrity guarantees, and reproducibility of the relational model.</p>
<p><strong>Acquisition</strong> — Data is collected from instruments, experiments, or external sources. Raw files land in object storage; metadata populates Manual tables.</p>
6651
+
<p><strong>Acquisition</strong> — Data is collected from instruments, experiments, or external sources. Raw files land in object storage; metadata populates Manual tables (<code>Session</code>, <code>Scan</code>).</p>
6636
6652
</li>
6637
6653
<li>
6638
-
<p><strong>Import</strong> — Automated processes parse raw data, extract signals, and populate Imported tables with structured results.</p>
6654
+
<p><strong>Import</strong> — Automated processes parse raw data, extract signals, and populate Imported tables with structured results (<code>ScanInfo</code>).</p>
6639
6655
</li>
6640
6656
<li>
6641
-
<p><strong>Computation</strong> — The <code>populate()</code> mechanism identifies new data and triggers downstream processing. Compute resources execute transformations and populate Computed tables.</p>
6657
+
<p><strong>Computation</strong> — The <code>populate()</code> mechanism identifies new data and triggers downstream processing. Compute resources execute transformations and populate Computed tables (<code>MotionCorrection</code>, <code>Segmentation</code>, <code>Fluorescence</code>).</p>
6642
6658
</li>
6643
6659
<li>
6644
6660
<p><strong>Query & Analysis</strong> — Users query results across the pipeline, combining data from multiple stages to generate insights, reports, or visualizations.</p>
<h2id="a-worked-example">A worked example<aclass="headerlink" href="#a-worked-example" title="Permanent link">¶</a></h2>
6609
6609
<p>Diagrams in this documentation use the same notation as <code>dj.Diagram</code> in
6610
-
<code>datajoint-python</code>: <strong>Manual</strong> tables are green rectangles, <strong>Lookup</strong>
6611
-
tables are gray rectangles, <strong>Imported</strong> tables are blue ovals, and
6612
-
<strong>Computed</strong> tables are red ovals. A <strong>Part</strong> table is a plain rectangle
6613
-
grouped with its master inside a light box. Tier is conveyed by shape and
6610
+
<code>datajoint-python</code>: <strong>Manual</strong> tables are green rounded boxes, <strong>Lookup</strong>
6611
+
tables are gray rounded boxes, <strong>Imported</strong> tables are blue ellipses, and
6612
+
<strong>Computed</strong> tables are orange ellipses. A <strong>Part</strong> table is a small plain box
6613
+
grouped with its master inside a light box (the <em>entity cluster</em>). Tier is conveyed by shape and
6614
6614
color, and <strong>edge thickness</strong> shows how a child relates to its parent — a
6615
6615
thick line means the child <strong>extends</strong> the parent (one per parent); a thin
6616
6616
line means the child is <strong>contained within</strong> the parent (many per parent).
@@ -6633,7 +6633,7 @@ <h2 id="a-worked-example">A worked example<a class="headerlink" href="#a-worked-
6633
6633
or combining several, inheriting its whole key and adding no new
6634
6634
dimension. The legend below the figure keys the full notation.</p>
6635
6635
<p><imgalt="Worked-example imaging pipeline diagram spanning two schemas: experiment (Mouse → Session → Scan) and analysis (AverageFrame → Segmentation → Fluorescence, with Lookup SegmentationParam feeding Segmentation, and the Part tables Roi on Segmentation and Trace on Fluorescence)." src="../../images/rwm-pipeline.svg" /></p>
6636
-
<p><imgalt="Legend: table tiers — Manual (green rectangle), Lookup (gray rectangle), Imported (blue oval), Computed (red oval), Part (smaller plain rectangle); an underlined name is a new entity type (a new schema dimension, many rows per parent) while a plain name is composed from existing entities (one row per parent); edge thickness — thick means the child extends the parent, thin means the child is contained within the parent; a dashed rounded box is a schema module (labeled in the corner); a gray box encloses a master with its parts; edges have no arrowheads, so direction follows the layout." src="../../images/rwm-legend.svg" /></p>
6636
+
<p><imgalt="Legend: table tiers — Manual (green rounded box), Lookup (gray rounded box), Imported (blue ellipse), Computed (orange ellipse), Part (smaller plain box); an underlined name is a new entity type (a new schema dimension, many rows per parent) while a plain name is composed from existing entities (one row per parent); edge thickness — thick means the child extends the parent, thin means the child is contained within the parent; a dashed rounded box is a schema module (labeled in the corner); a gray box encloses a master with its parts; edges have no arrowheads, so direction follows the layout." src="../../images/rwm-legend.svg" /></p>
6637
6637
<p>The notation is specified in full in the <ahref="../../reference/specs/diagram/">Diagram specification</a>. The concepts it depicts are explained in depth elsewhere: <ahref="../entity-integrity/">entity integrity</a> (keys, entity types, and schema dimensions), <ahref="../../reference/specs/master-part/">master–part tables</a> (the entity group and its all-or-nothing populate), the <ahref="../computation-model/">computation model</a> (how <code>make()</code> produces Imported and Computed tables), and <ahref="../semantic-matching/">semantic matching</a> (why a name means the same thing everywhere it appears).</p>
6638
6638
<p>The pipeline spans two schemas: <strong><code>experiment</code></strong> holds the raw, manually
6639
6639
entered tables, and <strong><code>analysis</code></strong> holds everything derived from them.
0 commit comments