Skip to content

Commit bd53c48

Browse files
dimitri-yatsenkodimitri-yatsenko
authored
Merge pull request #9 from dimitri-yatsenko/main
Revise the Operations section
2 parents a5e0bc7 + 784cd02 commit bd53c48

5 files changed

Lines changed: 239 additions & 243 deletions

File tree

‎book/40-operations/000-workflow-operations.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Workflow Operations
2+
3+
## Executing the Workflow
4+
5+
The previous sections established the **Relational Workflow Model** and schema design principles.
6+
Your schema defines *what* entities exist, *how* they depend on each other, and *when* they are created in the workflow.
7+
**Operations** are the actions that execute this workflow—populating your pipeline with actual data.
8+
9+
In DataJoint, operations fall into two categories:
10+
11+
1. **Manual operations** — Actions initiated *outside* the pipeline using `insert`, `delete`, and occasionally `update`
12+
2. **Automatic operations** — Pipeline-driven population using `populate` for Imported and Computed tables
13+
14+
The term "manual" does not imply human involvement—it means the operation originates *external to the pipeline*.
15+
A script that parses instrument files and inserts session records is performing manual operations, even though no human is involved.
16+
The key distinction is *who initiates the action*: external processes (manual) versus the pipeline's own `populate` mechanism (automatic).
17+
18+
This distinction maps directly to the table tiers introduced in the [Relational Workflow Model](../20-concepts/05-workflows.md):
19+
20+
| Table Tier | How Data Enters | Typical Operations |
21+
|------------|-----------------|-------------------|
22+
| **Lookup** | Schema definition (`contents` property) | None—predefined |
23+
| **Manual** | External to pipeline | `insert`, `delete` |
24+
| **Imported** | Pipeline-driven acquisition | `populate` |
25+
| **Computed** | Pipeline-driven computation | `populate` |
26+
27+
## Lookup Tables: Part of the Schema
28+
29+
**Lookup tables are not part of the workflow**—they are part of the schema definition itself.
30+
31+
Lookup tables contain reference data, controlled vocabularies, parameter sets, and configuration values that define the *context* in which the workflow operates.
32+
This data is:
33+
34+
- Defined in the table class using the `contents` property
35+
- Automatically present when the schema is activated
36+
- Shared across all workflow executions
37+
38+
Examples include:
39+
- Species names and codes
40+
- Experimental protocols
41+
- Processing parameter sets
42+
- Instrument configurations
43+
44+
Because lookup data defines the problem space rather than recording workflow execution, it is specified declaratively as part of the table definition:
45+
46+
```python
47+
@schema
48+
class BlobParamSet(dj.Lookup):
49+
definition = """
50+
blob_paramset : int
51+
---
52+
min_sigma : float
53+
max_sigma : float
54+
threshold : float
55+
"""
56+
contents = [
57+
(1, 1.0, 5.0, 0.1),
58+
(2, 2.0, 10.0, 0.05),
59+
]
60+
```
61+
62+
When the schema is activated, an "empty" pipeline already has its lookup tables populated.
63+
This ensures that reference data is always available and consistent across all installations of the pipeline.
64+
65+
## Manual Tables: The Workflow Entry Points
66+
67+
**Manual tables** are where new information enters the workflow from external sources.
68+
The term "manual" refers to the data's origin—*outside the pipeline*—not to how it gets there.
69+
70+
Manual tables capture information that originates external to the computational pipeline:
71+
72+
- Experimental subjects and sessions
73+
- Observations and annotations
74+
- External system identifiers
75+
- Curated selections and decisions
76+
77+
Data enters Manual tables through explicit `insert` operations from various sources:
78+
79+
- **Human entry**: Data entry forms, lab notebooks, manual curation
80+
- **Automated scripts**: Parsing instrument files, syncing from external databases
81+
- **External systems**: Laboratory information management systems (LIMS), scheduling software
82+
- **Integration pipelines**: ETL processes that import data from other sources
83+
84+
Each insert into a Manual table potentially triggers downstream computations—this is the "data enters the system" event that drives the pipeline forward.
85+
Whether a human clicks a button or a cron job runs a script, the effect is the same: new data enters the pipeline and becomes available for automatic processing.
86+
87+
## Automatic Population: The Workflow Engine
88+
89+
**Imported** and **Computed** tables are populated automatically through the `populate` mechanism.
90+
This is the core of workflow automation in DataJoint.
91+
92+
When you call `populate()` on an auto-populated table, DataJoint:
93+
94+
1. Identifies what work is missing by examining upstream dependencies
95+
2. Executes the table's `make()` method for each pending item
96+
3. Wraps each computation in a transaction for integrity
97+
4. Continues through all pending work, handling errors gracefully
98+
99+
This automation embodies the Relational Workflow Model's key principle: **the schema is an executable specification**.
100+
You don't write scripts to orchestrate computations—you define dependencies, and the system figures out what to run.
101+
102+
```python
103+
# The schema defines what should be computed
104+
# populate() executes it
105+
Detection.populate(display_progress=True)
106+
```
107+
108+
## The Three Core Operations
109+
110+
### Insert: Adding Data
111+
112+
The `insert` operation adds new entities to Manual tables, representing new information entering the workflow from external sources.
113+
114+
```python
115+
# Single row
116+
Subject.insert1({"subject_id": "M001", "species": "mouse", "sex": "M"})
117+
118+
# Multiple rows
119+
Session.insert([
120+
{"subject_id": "M001", "session_date": "2024-01-15"},
121+
{"subject_id": "M001", "session_date": "2024-01-16"},
122+
])
123+
```
124+
125+
### Delete: Removing Data with Cascade
126+
127+
The `delete` operation removes entities and **all their downstream dependents**.
128+
This cascading behavior is fundamental to maintaining **computational validity**—the guarantee that derived data remains consistent with its inputs.
129+
130+
When you delete an entity:
131+
- All entities that depend on it (via foreign keys) are also deleted
132+
- This cascades through the entire dependency graph
133+
- The result is a consistent database state
134+
135+
```python
136+
# Deleting a session removes all its downstream analysis
137+
(Session & {"subject_id": "M001", "session_date": "2024-01-15"}).delete()
138+
```
139+
140+
Cascading delete is the primary mechanism for:
141+
- **Correcting errors**: Delete incorrect upstream data; downstream results disappear automatically
142+
- **Reprocessing**: Delete computed results to regenerate them with updated code
143+
- **Data lifecycle**: Remove obsolete data and everything derived from it
144+
145+
### Update: Rare and Deliberate
146+
147+
The `update` operation modifies existing values **in place**.
148+
In DataJoint, updates are deliberately rare because they can violate computational validity.
149+
150+
Consider: if you update an upstream value, downstream computed results become inconsistent—they were derived from the old value but now coexist with the new one.
151+
The proper approach is usually **delete and reinsert**:
152+
153+
1. Delete the incorrect data (cascading removes dependent computations)
154+
2. Insert the corrected data
155+
3. Re-run `populate()` to regenerate downstream results
156+
157+
The `update1` method exists for cases where in-place correction is truly needed—typically for:
158+
- Fixing typos in descriptive fields that don't affect computations
159+
- Correcting metadata that has no downstream dependencies
160+
- Administrative changes to non-scientific attributes
161+
162+
```python
163+
# Use sparingly—only for corrections that don't affect downstream data
164+
Subject.update1({"subject_id": "M001", "notes": "Corrected housing info"})
165+
```
166+
167+
## The Workflow Execution Pattern
168+
169+
A typical DataJoint workflow follows this pattern:
170+
171+
```
172+
┌─────────────────────────────────────────────────────────────┐
173+
│ 1. SCHEMA ACTIVATION │
174+
│ - Define tables and dependencies │
175+
│ - Lookup tables are automatically populated (contents) │
176+
└─────────────────────────────────────────────────────────────┘
177+
↓
178+
┌─────────────────────────────────────────────────────────────┐
179+
│ 2. EXTERNAL DATA ENTRY │
180+
│ - Insert subjects, sessions, trials into Manual tables │
181+
│ - Each insert is a potential trigger for downstream │
182+
└─────────────────────────────────────────────────────────────┘
183+
↓
184+
┌─────────────────────────────────────────────────────────────┐
185+
│ 3. AUTOMATIC POPULATION │
186+
│ - Call populate() on Imported tables (data acquisition) │
187+
│ - Call populate() on Computed tables (analysis) │
188+
│ - System determines order from dependency graph │
189+
└─────────────────────────────────────────────────────────────┘
190+
↓
191+
┌─────────────────────────────────────────────────────────────┐
192+
│ 4. ITERATION │
193+
│ - New manual entries trigger new computations │
194+
│ - Errors corrected via delete + reinsert + repopulate │
195+
│ - Pipeline grows incrementally │
196+
└─────────────────────────────────────────────────────────────┘
197+
```
198+
199+
## Transactions and Integrity
200+
201+
All operations in DataJoint respect **ACID transactions** and **referential integrity**:
202+
203+
- **Inserts** verify that all referenced foreign keys exist
204+
- **Deletes** cascade to maintain referential integrity
205+
- **Populate** wraps each `make()` call in a transaction
206+
207+
This ensures that the database always represents a consistent state—there are no orphaned records, no dangling references, and no partially-completed computations visible to other users.
208+
209+
## Chapter Overview
210+
211+
The following chapters detail each operation:
212+
213+
- **[Insert](010-insert.ipynb)** — Adding data to Manual tables
214+
- **[Delete](020-delete.ipynb)** — Removing data with cascading dependencies
215+
- **[Updates](030-updates.ipynb)** — Rare in-place modifications
216+
- **[Transactions](040-transactions.ipynb)** — ACID semantics and consistency
217+
- **[Populate](050-populate.ipynb)** — Automatic workflow execution
218+
- **[The `make` Method](055-make.ipynb)** — Defining computational logic
219+
- **[Orchestration](060-orchestration.ipynb)** — Infrastructure for running at scale

‎book/40-operations/010-insert.ipynb‎

Lines changed: 2 additions & 103 deletions
Original file line numberDiff line numberDiff line change
@@ -3,108 +3,7 @@
33
{
44
"cell_type": "markdown",
55
"metadata": {},
6-
"source": [
7-
"# Insert \n",
8-
"\n",
9-
"(This is an AI-generated placeholder -- to be updated soon.)\n",
10-
"\n",
11-
"DataJoint provides two primary commands for adding data to tables: `insert` and `insert1`. Both commands are essential for populating tables while ensuring data integrity, but they are suited for different scenarios depending on the quantity and structure of the data being inserted.\n",
12-
"\n",
13-
"## Overview of `insert1`\n",
14-
"\n",
15-
"The `insert1` command is used for adding a single row of data to a table. It expects a dictionary where each key corresponds to a table attribute and the associated value represents the data to be inserted.\n",
16-
"\n",
17-
"### Syntax\n",
18-
"\n",
19-
"```python\n",
20-
"<Table>.insert1(data, ignore_extra_fields=False)\n",
21-
"```\n",
22-
"\n",
23-
"### Parameters\n",
24-
"\n",
25-
"1. **`data`**: A dictionary representing a single row of data, with keys matching the table's attributes.\n",
26-
"2. **`ignore_extra_fields`** *(default: False)*:\n",
27-
" - If `True`, attributes in the dictionary that are not part of the table schema are ignored.\n",
28-
" - If `False`, the presence of extra fields will result in an error.\n",
29-
"\n",
30-
"### Example\n",
31-
"\n",
32-
"```python\n",
33-
"import datajoint as dj\n",
34-
"\n",
35-
"schema = dj.Schema('example_schema')\n",
36-
"\n",
37-
"@schema\n",
38-
"class Animal(dj.Manual):\n",
39-
" definition = \"\"\"\n",
40-
" animal_id: int # Unique identifier for the animal\n",
41-
" ---\n",
42-
" species: varchar(64) # Species of the animal\n",
43-
" age: int # Age of the animal in years\n",
44-
" \"\"\"\n",
45-
"\n",
46-
"# Insert a single row into the Animal table\n",
47-
"Animal.insert1({\n",
48-
" 'animal_id': 1,\n",
49-
" 'species': 'Dog',\n",
50-
" 'age': 5\n",
51-
"})\n",
52-
"```\n",
53-
"\n",
54-
"### Key Points\n",
55-
"\n",
56-
"- `insert1` is ideal for inserting a single, well-defined record.\n",
57-
"- It ensures clarity when adding individual entries, reducing ambiguity in debugging.\n",
58-
"\n",
59-
"## Overview of `insert`\n",
60-
"\n",
61-
"The `insert` command is designed for batch insertion, allowing multiple rows to be added in a single operation. It accepts a list of dictionaries, where each dictionary represents a single row of data.\n",
62-
"\n",
63-
"### Syntax\n",
64-
"\n",
65-
"```python\n",
66-
"<Table>.insert(data, ignore_extra_fields=False, skip_duplicates=False)\n",
67-
"```\n",
68-
"\n",
69-
"### Parameters\n",
70-
"\n",
71-
"1. **`data`**: A list of dictionaries, where each dictionary corresponds to a row of data to insert.\n",
72-
"2. **`ignore_extra_fields`** *(default: False)*:\n",
73-
" - If `True`, any extra keys in the dictionaries are ignored.\n",
74-
" - If `False`, extra keys result in an error.\n",
75-
"3. **`skip_duplicates`** *(default: False)*:\n",
76-
" - If `True`, rows with duplicate primary keys are skipped.\n",
77-
" - If `False`, duplicate rows trigger an error.\n",
78-
"\n",
79-
"### Example\n",
80-
"\n",
81-
"```python\n",
82-
"# Insert multiple rows into the Animal table\n",
83-
"Animal.insert([\n",
84-
" {'animal_id': 2, 'species': 'Cat', 'age': 3},\n",
85-
" {'animal_id': 3, 'species': 'Rabbit', 'age': 2}\n",
86-
"])\n",
87-
"```\n",
88-
"\n",
89-
"### Key Points\n",
90-
"\n",
91-
"- `insert` is efficient for adding multiple records in a single operation.\n",
92-
"- Use `skip_duplicates=True` to gracefully handle re-insertions of existing data.\n",
93-
"\n",
94-
"## Best Practices\n",
95-
"\n",
96-
"1. **Use ****`insert1`**** for Single Rows**: Prefer `insert1` when working with individual entries to maintain clarity.\n",
97-
"2. **Validate Data Consistency**: Ensure the input data adheres to the schema definition.\n",
98-
"3. **Batch Insert for Performance**: Use `insert` for larger datasets to minimize database interactions.\n",
99-
"4. **Handle Extra Fields Carefully**: Use `ignore_extra_fields=False` to detect unexpected keys.\n",
100-
"5. **Avoid Duplicates**: Use `skip_duplicates=True` when re-inserting known data to avoid errors.\n",
101-
"\n",
102-
"## Summary\n",
103-
"\n",
104-
"- Use `insert1` for single-row insertions and `insert` for batch operations.\n",
105-
"- Both commands enforce schema constraints and maintain the integrity of the database.\n",
106-
"- Proper use of these commands ensures efficient, accurate, and scalable data entry into your DataJoint pi\n"
107-
]
6+
"source": "# Insert\n\nThe `insert` operation adds new entities to Manual tables.\nIn the context of the [Relational Workflow Model](../20-concepts/05-workflows.md), inserting data is how information enters the pipeline from external sources.\n\n## Insert in the Workflow\n\nThe `insert` operation applies to **Manual tables**—tables that receive data from outside the pipeline:\n\n| Table Tier | How Data Enters |\n|------------|-----------------|\n| **Lookup** | `contents` property (part of schema definition) |\n| **Manual** | `insert` from external sources |\n| **Imported/Computed** | `populate()` mechanism |\n\nFor **Manual tables**, each insert represents new information entering the workflow from an external source.\nThe term \"manual\" refers to the data's origin—*outside the pipeline*—not to how it arrives.\nInserts into Manual tables can come from human data entry, automated scripts parsing instrument files, or integrations with external systems.\nWhat matters is that the pipeline's `populate` mechanism does not create this data—it comes from outside.\n\nEach insert into a Manual table potentially triggers downstream computations: when you insert a new session, all Imported and Computed tables that depend on it become candidates for population.\n\n## The `insert1` Method\n\nUse `insert1` to add a single row:\n\n```python\n<Table>.insert1(row, ignore_extra_fields=False)\n```\n\n**Parameters:**\n- **`row`**: A dictionary with keys matching table attributes\n- **`ignore_extra_fields`**: If `True`, extra dictionary keys are silently ignored; if `False` (default), extra keys raise an error\n\n**Example:**\n```python\n# Insert a single subject into a Manual table\nSubject.insert1({\n 'subject_id': 'M001',\n 'species': 'mouse',\n 'sex': 'M',\n 'date_of_birth': '2023-06-15'\n})\n```\n\nUse `insert1` when:\n- Adding individual records interactively\n- Processing items one at a time in a loop where you need error handling per item\n- Debugging, where single-row operations provide clearer error messages\n\n## The `insert` Method\n\nUse `insert` for batch insertion of multiple rows:\n\n```python\n<Table>.insert(rows, ignore_extra_fields=False, skip_duplicates=False)\n```\n\n**Parameters:**\n- **`rows`**: A list of dictionaries (or any iterable of dict-like objects)\n- **`ignore_extra_fields`**: If `True`, extra keys are ignored\n- **`skip_duplicates`**: If `True`, rows with existing primary keys are silently skipped; if `False` (default), duplicates raise an error\n\n**Example:**\n```python\n# Batch insert multiple sessions (could be from a script parsing log files)\nSession.insert([\n {'subject_id': 'M001', 'session_date': '2024-01-15', 'session_notes': 'baseline'},\n {'subject_id': 'M001', 'session_date': '2024-01-16', 'session_notes': 'treatment'},\n {'subject_id': 'M001', 'session_date': '2024-01-17', 'session_notes': 'follow-up'},\n])\n```\n\nUse `insert` when:\n- Loading data from files or external sources\n- Importing from external databases or APIs\n- Migrating or synchronizing data between systems\n\n## Referential Integrity\n\nDataJoint enforces referential integrity on insert.\nIf a table has foreign key dependencies, the referenced entities must already exist:\n\n```python\n# This will fail if subject 'M001' doesn't exist in Subject table\nSession.insert1({\n 'subject_id': 'M001', # Must exist in Subject\n 'session_date': '2024-01-15'\n})\n```\n\nThis constraint ensures the dependency graph remains valid—you cannot create downstream entities without their upstream dependencies.\nNote that Lookup table data (defined via `contents`) is automatically available when the schema is activated, so foreign key references to Lookup tables are always satisfied.\n\n## Best Practices\n\n1. **Match insert method to use case**: Use `insert1` for single records, `insert` for batches\n2. **Keep `ignore_extra_fields=False`** (default): Helps catch data mapping errors early\n3. **Insert upstream before downstream**: Respect the dependency order defined by foreign keys\n4. **Use `skip_duplicates=True` for idempotent scripts**: When re-running import scripts, this avoids errors on existing data\n5. **Let `populate()` handle auto-populated tables**: Never insert directly into Imported or Computed tables"
1087
}
1098
],
1109
"metadata": {
@@ -114,4 +13,4 @@
11413
},
11514
"nbformat": 4,
11615
"nbformat_minor": 2
117-
}
16+
}

0 commit comments

Comments
 (0)