Merge pull request dimitri-yatsenko#23 from dimitri-yatsenko/claude/merge-computations-operations-1ZJss

Merge Computations section into Operations

Author: dimitri-yatsenko

book/20-concepts/04-integrity.md

@@ -162,7 +162,7 @@ Workflow integrity maintains valid operation sequences through:
 
 **Covered in:**
 - [Foreign Keys](../30-design/030-foreign-keys.ipynb) — How foreign keys encode workflow dependencies
-- [Computation](../60-computation/010-computation.ipynb) — Automatic workflow execution and dependency resolution
+- [Populate](../40-operations/050-populate.ipynb) — Automatic workflow execution and dependency resolution
 
 ---
 

book/30-design/015-table.ipynb

@@ -352,7 +352,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Table Base Classes\n\nDataJoint provides four base classes for different data management patterns:\n\n| Base Class | Purpose | When to Use |\n|------------|---------|-------------|\n| `dj.Manual` | Manually entered data | Subject info, experimental protocols |\n| `dj.Lookup` | Reference data, rarely changes | Equipment lists, parameter sets |\n| `dj.Imported` | Data imported from external files | Raw recordings, behavioral videos |\n| `dj.Computed` | Derived from other tables | Spike sorting results, analyses |\n\nWe'll explore `Imported` and `Computed` tables in the [Computation](../60-computation/) section.\n\n```{seealso}\n- [Lookup Tables](018-lookup-tables.ipynb) — Managing reference data\n- [Computation](../60-computation/010-computation.ipynb) — Automated data processing\n```"
+   "source": "# Table Base Classes\n\nDataJoint provides four base classes for different data management patterns:\n\n| Base Class | Purpose | When to Use |\n|------------|---------|-------------|\n| `dj.Manual` | Manually entered data | Subject info, experimental protocols |\n| `dj.Lookup` | Reference data, rarely changes | Equipment lists, parameter sets |\n| `dj.Imported` | Data imported from external files | Raw recordings, behavioral videos |\n| `dj.Computed` | Derived from other tables | Spike sorting results, analyses |\n\nWe'll explore `Imported` and `Computed` tables in the [Populate](050-populate.ipynb) chapter.\n\n```{seealso}\n- [Lookup Tables](018-lookup-tables.ipynb) — Managing reference data\n- [Populate](050-populate.ipynb) — Automated data processing\n```"
   },
   {
    "cell_type": "markdown",

book/40-operations/050-populate.ipynb

@@ -0,0 +1,21 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# Populate\n\nThe `populate` operation is the engine of workflow automation in DataJoint.\nWhile `insert`, `delete`, and `update` are manual operations, `populate` automates data entry for **Imported** and **Computed** tables based on dependencies defined in the schema.\n\nThis chapter connects the theoretical foundations of the [Relational Workflow Model](../20-concepts/05-workflows.md) to the practical `populate` operation.\n\n## The Relational Workflow Model in Action\n\nRecall that the **Relational Workflow Model** is built on four fundamental concepts:\n\n1. **Workflow Entity** — Each table represents an entity type created at a specific workflow step\n2. **Workflow Dependencies** — Foreign keys prescribe the order of operations\n3. **Workflow Steps** — Distinct phases where entity types are created (manual or automated)\n4. **Directed Acyclic Graph (DAG)** — The schema forms a graph structure ensuring valid execution sequences\n\nThe Relational Workflow Model defines a new class of databases: **Computational Databases**, where computational transformations are first-class citizens of the data model. In a computational database, the schema is not merely a passive data structure—it is an executable specification of the workflow itself.\n\n## From Declarative Schema to Executable Pipeline\n\nA DataJoint schema uses **table tiers** to distinguish different workflow roles:\n\n| Tier | Color | Role in Workflow |\n|------|-------|------------------|\n| **Lookup** | Gray | Static reference data and configuration parameters |\n| **Manual** | Green | Human-entered data or data from external systems |\n| **Imported** | Blue | Data acquired automatically from instruments or files |\n| **Computed** | Red | Derived data produced by computational transformations |\n\nBecause dependencies are explicit through foreign keys, DataJoint's `populate()` method can explore the DAG top-down: for every upstream key that has not been processed, it executes the table's `make()` method inside an atomic transaction. If anything fails, the transaction is rolled back, preserving **computational validity**—the guarantee that all derived data remains consistent with its upstream dependencies.\n\nThis is the essence of **workflow automation**: each table advertises what it depends on, and `populate()` runs only the computations that are still missing.\n\n## The `populate` Method\n\nThe `populate()` method is the engine of workflow automation. When called on a computed or imported table, it:\n\n1. **Identifies missing work** — Queries the key source (the join of all upstream dependencies) and subtracts keys already present in the table\n2. **Iterates over pending keys** — For each missing key, calls the table's `make()` method\n3. **Wraps each `make()` in a transaction** — Ensures atomicity: either all inserts succeed or none do\n4. **Handles errors gracefully** — Failed jobs are logged but do not stop the remaining work\n\n```python\n# Process all pending work\nDetection.populate(display_progress=True)\n\n# Process a specific subset\nDetection.populate(Image & \"image_id < 10\")\n\n# Distribute across workers\nDetection.populate(reserve_jobs=True)\n```\n\nThe `reserve_jobs=True` option enables parallel execution across multiple processes or machines by using the database itself for job coordination.\n\n## Transactional Integrity\n\nEach `make()` call executes inside an **ACID transaction**. This provides critical guarantees for computational workflows:\n\n- **Atomicity** — The entire computation either commits or rolls back as a unit\n- **Isolation** — Partial results are never visible to other processes\n- **Consistency** — The database moves from one valid state to another\n\nWhen a computed table has [part tables](../30-design/053-master-part.ipynb), the transaction boundary encompasses both the master and all its parts. The master's `make()` method is responsible for inserting everything within a single transactional scope. See the [Master-Part](../30-design/053-master-part.ipynb) chapter for detailed coverage of ACID semantics and the master's responsibility pattern.\n\n## Case Study: Blob Detection\n\nThe [Blob Detection](../80-examples/075-blob-detection.ipynb) example demonstrates these concepts in a compact image-analysis workflow:\n\n1. **Source data** — `Image` (manual) stores NumPy arrays as `longblob` fields\n2. **Parameter space** — `BlobParamSet` (lookup) defines detection configurations\n3. **Computation** — `Detection` (computed) depends on both upstream tables\n\nThe `Detection` table uses a master-part structure: the master row stores an aggregate (blob count), while `Detection.Blob` parts store per-feature coordinates. When `populate()` runs:\n\n- Each `(image_id, blob_paramset)` combination triggers one `make()` call\n- The `make()` method fetches inputs, runs detection, and inserts both master and parts\n- The transaction ensures all blob coordinates appear atomically with their count\n\n```python\nDetection.populate(display_progress=True)\n# Detection: 100%|██████████| 6/6 [00:01<00:00, 4.04it/s]\n```\n\nThis pattern—automation exploring combinatorics, then human curation—is common in scientific workflows. After reviewing results, the `SelectDetection` manual table records the preferred parameter set for each image. Because `SelectDetection` depends on `Detection`, it implicitly has access to all `Detection.Blob` parts for the selected detection.\n\n:::{seealso}\n- [Blob Detection](../80-examples/075-blob-detection.ipynb) — Complete working example\n- [Master-Part](../30-design/053-master-part.ipynb) — Transaction semantics and dependency implications\n:::\n\n## Why Computational Databases Matter\n\nThe Relational Workflow Model provides several key benefits:\n\n| Benefit | Description |\n|---------|-------------|\n| **Reproducibility** | Rerunning `populate()` regenerates derived tables from raw inputs |\n| **Dependency-aware scheduling** | DataJoint infers job order from foreign keys (the DAG structure) |\n| **Computational validity** | Transactions ensure downstream results stay consistent with upstream inputs |\n| **Provenance tracking** | The schema documents what was computed from what |\n\n## Practical Tips\n\n- **Develop incrementally** — Test `make()` logic with restrictions (e.g., `Table.populate(restriction)`) before processing all data\n- **Monitor progress** — Use `display_progress=True` for visibility during development\n- **Distribute work** — Use `reserve_jobs=True` when running multiple workers\n- **Use master-part for multi-row results** — When a computation produces both summary and detail rows, structure them as master and parts to keep them in the same transaction"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

book/40-operations/060-orchestration.ipynb

@@ -0,0 +1,126 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-0",
+   "metadata": {},
+   "source": [
+    "# Orchestration\n",
+    "\n",
+    "While the `populate` operation provides the logic for automated computation, **orchestration** addresses the infrastructure and operational concerns of running these computations at scale:\n",
+    "\n",
+    "- **Infrastructure provisioning** — Allocating compute resources (servers, containers, cloud instances)\n",
+    "- **Dependency management** — Ensuring consistent runtime environments across workers\n",
+    "- **Automated execution** — Scheduling and triggering `populate` calls\n",
+    "- **Observability** — Monitoring job progress, failures, and system health\n",
+    "- **Performance and cost tracking** — Understanding resource utilization and expenses\n",
+    "\n",
+    "These concerns are **outside the scope of the core DataJoint library** (`datajoint-python`), which focuses on the data model and workflow logic. Orchestration is solved through complementary infrastructure.\n",
+    "\n",
+    "## The Orchestration Challenge\n",
+    "\n",
+    "A typical DataJoint workflow requires:\n",
+    "\n",
+    "1. **Database server** — MySQL/MariaDB instance with appropriate configuration\n",
+    "2. **Worker processes** — Python environments with DataJoint and domain-specific packages\n",
+    "3. **File storage** — For external blob storage (if using `dj.config['stores']`)\n",
+    "4. **Job coordination** — Managing which workers process which jobs\n",
+    "5. **Error handling** — Retrying failed jobs, alerting on persistent failures\n",
+    "6. **Scaling** — Adding workers during high-demand periods\n",
+    "\n",
+    "The `populate(reserve_jobs=True)` option handles job coordination at the database level, but provisioning and managing the workers themselves requires additional infrastructure.\n",
+    "\n",
+    "## Commercial Solution: DataJoint Works\n",
+    "\n",
+    "[DataJoint Works](https://datajoint.com) is a managed platform that provides comprehensive orchestration:\n",
+    "\n",
+    "| Feature | Description |\n",
+    "|---------|-------------|\n",
+    "| **Managed databases** | Provisioned and configured MySQL instances |\n",
+    "| **Container registry** | Store and version workflow container images |\n",
+    "| **Compute clusters** | Auto-scaling worker pools (cloud or on-premise) |\n",
+    "| **Job scheduler** | Automated triggering of `populate` operations |\n",
+    "| **Monitoring dashboard** | Real-time visibility into job status and errors |\n",
+    "| **Cost analytics** | Track compute and storage costs per workflow |\n",
+    "\n",
+    "This platform integrates directly with DataJoint schemas, providing a turnkey solution for teams that prefer managed infrastructure.\n",
+    "\n",
+    "## DIY Solutions\n",
+    "\n",
+    "Many teams build custom orchestration using standard DevOps tools. Common approaches include:\n",
+    "\n",
+    "### Containerization\n",
+    "\n",
+    "- **Docker** — Package DataJoint workflows with all dependencies\n",
+    "- **Singularity/Apptainer** — Container runtime for HPC environments\n",
+    "- **Conda environments** — Dependency management without full containerization\n",
+    "\n",
+    "### Container Orchestration\n",
+    "\n",
+    "- **Kubernetes** — Production-grade container orchestration\n",
+    "- **Docker Swarm** — Simpler container clustering\n",
+    "- **Nomad** — HashiCorp's workload orchestrator\n",
+    "\n",
+    "### Job Schedulers\n",
+    "\n",
+    "- **SLURM** — Common in academic HPC clusters\n",
+    "- **PBS/Torque** — Traditional batch scheduling\n",
+    "- **HTCondor** — High-throughput computing scheduler\n",
+    "- **Apache Airflow** — DAG-based workflow orchestration\n",
+    "- **Prefect** — Modern Python-native orchestration\n",
+    "- **Celery** — Distributed task queue\n",
+    "\n",
+    "### Cloud Infrastructure\n",
+    "\n",
+    "- **AWS Batch** — Managed batch computing on AWS\n",
+    "- **Google Cloud Run Jobs** — Serverless container execution\n",
+    "- **Azure Container Instances** — On-demand container execution\n",
+    "\n",
+    "### Monitoring and Observability\n",
+    "\n",
+    "- **Prometheus + Grafana** — Metrics collection and visualization\n",
+    "- **DataDog** — Commercial observability platform\n",
+    "- **CloudWatch / Stackdriver** — Cloud-native monitoring\n",
+    "\n",
+    "### Database Hosting\n",
+    "\n",
+    "- **Amazon RDS** — Managed MySQL on AWS\n",
+    "- **Google Cloud SQL** — Managed MySQL on GCP\n",
+    "- **Self-hosted MySQL/MariaDB** — On-premise or VM-based\n",
+    "\n",
+    "## Choosing an Approach\n",
+    "\n",
+    "The right orchestration strategy depends on your team's context:\n",
+    "\n",
+    "| Factor | Managed Platform | DIY |\n",
+    "|--------|-----------------|-----|\n",
+    "| **Setup time** | Hours | Days to weeks |\n",
+    "| **Maintenance** | Included | Team responsibility |\n",
+    "| **Customization** | Platform constraints | Full flexibility |\n",
+    "| **Cost model** | Subscription | Infrastructure costs |\n",
+    "| **Existing infrastructure** | May duplicate | Leverages investments |\n",
+    "| **Compliance requirements** | Check with vendor | Full control |\n",
+    "\n",
+    "Many teams start with DIY solutions using familiar tools, then evaluate managed platforms as workflows scale and operational overhead increases.\n",
+    "\n",
+    ":::{seealso}\n",
+    "- [DataJoint Works](https://datajoint.com) — Managed orchestration platform\n",
+    "- [Populate](050-populate.ipynb) — The underlying automation mechanism\n",
+    ":::"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}