Merge pull request dimitri-yatsenko#29 from dimitri-yatsenko/claude/revise-operations-section-01DZM7WBqMKF7bBEGmCJSGVK

Correct Lookup table population: use contents property, not insert

Author: dimitri-yatsenko

book/40-operations/000-workflow-operations.md

@@ -19,7 +19,7 @@ This distinction maps directly to the table tiers introduced in the [Relational
 
 | Table Tier | How Data Enters | Typical Operations |
 |------------|-----------------|-------------------|
-| **Lookup** | Schema definition | `insert` (once, at setup) |
+| **Lookup** | Schema definition (`contents` property) | None—predefined |
 | **Manual** | External to pipeline | `insert`, `delete` |
 | **Imported** | Pipeline-driven acquisition | `populate` |
 | **Computed** | Pipeline-driven computation | `populate` |
@@ -29,10 +29,10 @@ This distinction maps directly to the table tiers introduced in the [Relational
 **Lookup tables are not part of the workflow**—they are part of the schema definition itself.
 
 Lookup tables contain reference data, controlled vocabularies, parameter sets, and configuration values that define the *context* in which the workflow operates.
-This data is typically:
+This data is:
 
-- Inserted once when the schema is deployed
-- Rarely modified after initial setup
+- Defined in the table class using the `contents` property
+- Automatically present when the schema is activated
 - Shared across all workflow executions
 
 Examples include:
@@ -41,16 +41,27 @@ Examples include:
 - Processing parameter sets
 - Instrument configurations
 
-Because lookup data defines the problem space rather than recording workflow execution, it should be populated as part of schema initialization—not as an ongoing workflow step.
+Because lookup data defines the problem space rather than recording workflow execution, it is specified declaratively as part of the table definition:
 
 ```python
-# Lookup tables are populated at schema setup time
-BlobParamSet.insert([
-    {"blob_paramset": 1, "min_sigma": 1, "max_sigma": 5, "threshold": 0.1},
-    {"blob_paramset": 2, "min_sigma": 2, "max_sigma": 10, "threshold": 0.05},
-], skip_duplicates=True)
+@schema
+class BlobParamSet(dj.Lookup):
+    definition = """
+    blob_paramset : int
+    ---
+    min_sigma : float
+    max_sigma : float
+    threshold : float
+    """
+    contents = [
+        (1, 1.0, 5.0, 0.1),
+        (2, 2.0, 10.0, 0.05),
+    ]
 ```
 
+When the schema is activated, an "empty" pipeline already has its lookup tables populated.
+This ensures that reference data is always available and consistent across all installations of the pipeline.
+
 ## Manual Tables: The Workflow Entry Points
 
 **Manual tables** are where new information enters the workflow from external sources.
@@ -98,9 +109,7 @@ Detection.populate(display_progress=True)
 
 ### Insert: Adding Data
 
-The `insert` operation adds new entities to the database.
-For Manual tables, this represents new information entering the workflow.
-For Lookup tables, this establishes reference data.
+The `insert` operation adds new entities to Manual tables, representing new information entering the workflow from external sources.
 
 ```python
 # Single row
@@ -161,13 +170,13 @@ A typical DataJoint workflow follows this pattern:
 
 ```
 ┌─────────────────────────────────────────────────────────────┐
-│  1. SCHEMA SETUP                                            │
+│  1. SCHEMA ACTIVATION                                       │
 │     - Define tables and dependencies                        │
-│     - Populate Lookup tables with reference data            │
+│     - Lookup tables are automatically populated (contents)  │
 └─────────────────────────────────────────────────────────────┘
                               ↓
 ┌─────────────────────────────────────────────────────────────┐
-│  2. MANUAL DATA ENTRY                                       │
+│  2. EXTERNAL DATA ENTRY                                     │
 │     - Insert subjects, sessions, trials into Manual tables  │
 │     - Each insert is a potential trigger for downstream     │
 └─────────────────────────────────────────────────────────────┘
@@ -201,7 +210,7 @@ This ensures that the database always represents a consistent state—there are
 
 The following chapters detail each operation:
 
-- **[Insert](010-insert.ipynb)** — Adding data to Manual and Lookup tables
+- **[Insert](010-insert.ipynb)** — Adding data to Manual tables
 - **[Delete](020-delete.ipynb)** — Removing data with cascading dependencies
 - **[Updates](030-updates.ipynb)** — Rare in-place modifications
 - **[Transactions](040-transactions.ipynb)** — ACID semantics and consistency

book/40-operations/010-insert.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Insert\n\nThe `insert` operation adds new entities to the database.\nIn the context of the [Relational Workflow Model](../20-concepts/05-workflows.md), inserting data is how information enters the pipeline—either as reference data in Lookup tables or as new workflow items in Manual tables.\n\n## Insert in the Workflow\n\nDifferent table tiers use insert differently:\n\n| Table Tier | When to Insert | Typical Pattern |\n|------------|----------------|-----------------|\n| **Lookup** | Schema setup | Once, with `skip_duplicates=True` |\n| **Manual** | Ongoing workflow | As new subjects, sessions, trials occur |\n| **Imported/Computed** | Never via insert | Only via `populate()` |\n\nFor **Lookup tables**, insertion happens during schema initialization—these tables define the reference data and parameter sets that configure your pipeline.\n\nFor **Manual tables**, each insert represents new information entering the workflow from an external source.\nThe term \"manual\" refers to the data's origin—*outside the pipeline*—not to how it arrives.\nInserts into Manual tables can come from human data entry, automated scripts parsing instrument files, or integrations with external systems.\nWhat matters is that the pipeline's `populate` mechanism does not create this data—it comes from outside.\n\nThis is the trigger that drives downstream computations: when you insert a new session, all Imported and Computed tables that depend on it become candidates for population.\n\n## The `insert1` Method\n\nUse `insert1` to add a single row:\n\n```python\n<Table>.insert1(row, ignore_extra_fields=False)\n```\n\n**Parameters:**\n- **`row`**: A dictionary with keys matching table attributes\n- **`ignore_extra_fields`**: If `True`, extra dictionary keys are silently ignored; if `False` (default), extra keys raise an error\n\n**Example:**\n```python\n# Insert a single subject into a Manual table\nSubject.insert1({\n    'subject_id': 'M001',\n    'species': 'mouse',\n    'sex': 'M',\n    'date_of_birth': '2023-06-15'\n})\n```\n\nUse `insert1` when:\n- Adding individual records interactively\n- Processing items one at a time in a loop where you need error handling per item\n- Debugging, where single-row operations provide clearer error messages\n\n## The `insert` Method\n\nUse `insert` for batch insertion of multiple rows:\n\n```python\n<Table>.insert(rows, ignore_extra_fields=False, skip_duplicates=False)\n```\n\n**Parameters:**\n- **`rows`**: A list of dictionaries (or any iterable of dict-like objects)\n- **`ignore_extra_fields`**: If `True`, extra keys are ignored\n- **`skip_duplicates`**: If `True`, rows with existing primary keys are silently skipped; if `False` (default), duplicates raise an error\n\n**Example:**\n```python\n# Batch insert multiple sessions (could be from a script parsing log files)\nSession.insert([\n    {'subject_id': 'M001', 'session_date': '2024-01-15', 'session_notes': 'baseline'},\n    {'subject_id': 'M001', 'session_date': '2024-01-16', 'session_notes': 'treatment'},\n    {'subject_id': 'M001', 'session_date': '2024-01-17', 'session_notes': 'follow-up'},\n])\n```\n\nUse `insert` when:\n- Loading data from files or external sources\n- Populating Lookup tables at schema setup\n- Migrating or synchronizing data between systems\n\n## Populating Lookup Tables\n\nLookup tables should be populated as part of schema initialization, not as ongoing workflow operations.\nUse `skip_duplicates=True` to make the insertion idempotent—safe to run multiple times:\n\n```python\n# Idempotent lookup table population\n# Can be run every time the pipeline starts\nSpecies.insert([\n    {'species': 'mouse', 'species_name': 'Mus musculus'},\n    {'species': 'rat', 'species_name': 'Rattus norvegicus'},\n    {'species': 'human', 'species_name': 'Homo sapiens'},\n], skip_duplicates=True)\n\n# Parameter sets for analysis\nProcessingParams.insert([\n    {'param_id': 1, 'filter_cutoff': 300, 'threshold': 0.5},\n    {'param_id': 2, 'filter_cutoff': 500, 'threshold': 0.3},\n], skip_duplicates=True)\n```\n\n## Referential Integrity\n\nDataJoint enforces referential integrity on insert.\nIf a table has foreign key dependencies, the referenced entities must already exist:\n\n```python\n# This will fail if subject 'M001' doesn't exist in Subject table\nSession.insert1({\n    'subject_id': 'M001',  # Must exist in Subject\n    'session_date': '2024-01-15'\n})\n```\n\nThis constraint ensures the dependency graph remains valid—you cannot create downstream entities without their upstream dependencies.\n\n## Best Practices\n\n1. **Match insert method to use case**: Use `insert1` for single records, `insert` for batches\n2. **Use `skip_duplicates=True` for Lookup tables**: Makes initialization scripts idempotent\n3. **Keep `ignore_extra_fields=False`** (default): Helps catch data mapping errors early\n4. **Insert upstream before downstream**: Respect the dependency order defined by foreign keys\n5. **Let `populate()` handle auto-populated tables**: Never insert directly into Imported or Computed tables"
+   "source": "# Insert\n\nThe `insert` operation adds new entities to Manual tables.\nIn the context of the [Relational Workflow Model](../20-concepts/05-workflows.md), inserting data is how information enters the pipeline from external sources.\n\n## Insert in the Workflow\n\nThe `insert` operation applies to **Manual tables**—tables that receive data from outside the pipeline:\n\n| Table Tier | How Data Enters |\n|------------|-----------------|\n| **Lookup** | `contents` property (part of schema definition) |\n| **Manual** | `insert` from external sources |\n| **Imported/Computed** | `populate()` mechanism |\n\nFor **Manual tables**, each insert represents new information entering the workflow from an external source.\nThe term \"manual\" refers to the data's origin—*outside the pipeline*—not to how it arrives.\nInserts into Manual tables can come from human data entry, automated scripts parsing instrument files, or integrations with external systems.\nWhat matters is that the pipeline's `populate` mechanism does not create this data—it comes from outside.\n\nEach insert into a Manual table potentially triggers downstream computations: when you insert a new session, all Imported and Computed tables that depend on it become candidates for population.\n\n## The `insert1` Method\n\nUse `insert1` to add a single row:\n\n```python\n<Table>.insert1(row, ignore_extra_fields=False)\n```\n\n**Parameters:**\n- **`row`**: A dictionary with keys matching table attributes\n- **`ignore_extra_fields`**: If `True`, extra dictionary keys are silently ignored; if `False` (default), extra keys raise an error\n\n**Example:**\n```python\n# Insert a single subject into a Manual table\nSubject.insert1({\n    'subject_id': 'M001',\n    'species': 'mouse',\n    'sex': 'M',\n    'date_of_birth': '2023-06-15'\n})\n```\n\nUse `insert1` when:\n- Adding individual records interactively\n- Processing items one at a time in a loop where you need error handling per item\n- Debugging, where single-row operations provide clearer error messages\n\n## The `insert` Method\n\nUse `insert` for batch insertion of multiple rows:\n\n```python\n<Table>.insert(rows, ignore_extra_fields=False, skip_duplicates=False)\n```\n\n**Parameters:**\n- **`rows`**: A list of dictionaries (or any iterable of dict-like objects)\n- **`ignore_extra_fields`**: If `True`, extra keys are ignored\n- **`skip_duplicates`**: If `True`, rows with existing primary keys are silently skipped; if `False` (default), duplicates raise an error\n\n**Example:**\n```python\n# Batch insert multiple sessions (could be from a script parsing log files)\nSession.insert([\n    {'subject_id': 'M001', 'session_date': '2024-01-15', 'session_notes': 'baseline'},\n    {'subject_id': 'M001', 'session_date': '2024-01-16', 'session_notes': 'treatment'},\n    {'subject_id': 'M001', 'session_date': '2024-01-17', 'session_notes': 'follow-up'},\n])\n```\n\nUse `insert` when:\n- Loading data from files or external sources\n- Importing from external databases or APIs\n- Migrating or synchronizing data between systems\n\n## Referential Integrity\n\nDataJoint enforces referential integrity on insert.\nIf a table has foreign key dependencies, the referenced entities must already exist:\n\n```python\n# This will fail if subject 'M001' doesn't exist in Subject table\nSession.insert1({\n    'subject_id': 'M001',  # Must exist in Subject\n    'session_date': '2024-01-15'\n})\n```\n\nThis constraint ensures the dependency graph remains valid—you cannot create downstream entities without their upstream dependencies.\nNote that Lookup table data (defined via `contents`) is automatically available when the schema is activated, so foreign key references to Lookup tables are always satisfied.\n\n## Best Practices\n\n1. **Match insert method to use case**: Use `insert1` for single records, `insert` for batches\n2. **Keep `ignore_extra_fields=False`** (default): Helps catch data mapping errors early\n3. **Insert upstream before downstream**: Respect the dependency order defined by foreign keys\n4. **Use `skip_duplicates=True` for idempotent scripts**: When re-running import scripts, this avoids errors on existing data\n5. **Let `populate()` handle auto-populated tables**: Never insert directly into Imported or Computed tables"
   }
  ],
  "metadata": {