update the foreign key chapter based on the lecture.

Author: dimitri-yatsenko

book/30-schema-design/030-foreign-keys.ipynb

@@ -11,29 +11,22 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Referential integrity\n",
+    "# Modeling Relationships with Foreign Keys\n",
     "\n",
-    "**Referential Integrity** is the guarantee made by the data management process that the entitites represented in the database remain correctly associated and mutually consistent and that relationships between them remain accurate.\n",
+    "While **entity integrity** ensures that each record uniquely represents a real-world entity, **referential integrity** ensures that the *relationships between* these entities are valid and consistent. It's a guarantee that you won't have an employee assigned to a non-existent department or a task associated with a deleted project.\n",
     "\n",
-    "Referential integrity is predicated on entity integrity. \n",
-    "Without entity integrity, referential integrity cannot be properly defined nor enforced.\n",
+    "Crucially, **referential integrity is impossible without entity integrity**. You must first have a reliable way to identify unique entities before you can define their relationships.\n",
     "\n",
-    "# Foreign keys\n",
-    "In relational databases, referential integrity is defined and enforced by the means of *foreign keys*, which establishes a reltionship between the *child table* that contains the foreign key and the *parent table* that is referenced by the foreign key. \n",
+    "In relational databases, these relationships are established and enforced using **foreign keys**. A foreign key creates a link between a **child table** (the one with the reference) and a **parent table** (the one being referenced). Think of `Employee` as the child and `Title` as the parent; an employee must have a valid, existing title.\n",
     "\n",
-    "A **foreign key** is a column or several columns in the child table referencing the primary key column(s) in the parent table.\n",
+    "A foreign key is a column (or set of columns) in the child table that refers to the primary key of the parent table. In DataJoint, a foreign key *always* references a parent's primary key, which is a highly recommended practice for clarity and consistency.\n",
     "\n",
-    "In DataJoint, the foreign *always* references the primary key of the parent table and that's the only way foreign keys are used. \n",
-    "However, more generally in SQL and relational theory, foreign keys can reference other sets of columns.\n",
-    "Such uses are esoteric and we avoid using them. \n",
-    "All foreign key references in this book will reference the primary key of the parent table.\n",
-    "\n",
-    "In the following example, we create a foreign key between an employee and their work title.  We first define a lookup table `Title` that lists all possible titles and the table `Employee` that containts references `Title`."
+    "In the following example, we define the parent table `Title` and the child table `Employee`, which references `Title`."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [
     {
@@ -42,14 +35,6 @@
      "text": [
       "Exception reporting mode: Minimal\n"
      ]
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "[2024-10-21 04:07:03,488][INFO]: Connecting root@localhost:3306\n",
-      "[2024-10-21 04:07:03,506][INFO]: Connected root@localhost:3306\n"
-     ]
     }
    ],
    "source": [
@@ -77,8 +62,6 @@
     "        (\"HR-Mgr\", \"Human Resources Manager\")\n",
     "    ]\n",
     "\n",
-    "\n",
-    "\n",
     "@schema\n",
     "class Employee(dj.Manual):\n",
     "    definition = \"\"\"\n",
@@ -101,46 +84,46 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
      "data": {
       "image/svg+xml": [
-       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"83pt\" height=\"113pt\" viewBox=\"0.00 0.00 83.25 113.12\">\n",
-       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 109.12)\">\n",
-       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-109.12 79.25,-109.12 79.25,4 -4,4\"/>\n",
+       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"84pt\" height=\"114pt\" viewBox=\"0.00 0.00 84.00 114.00\">\n",
+       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 110)\">\n",
+       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-110 80,-110 80,4 -4,4\"/>\n",
        "<!-- Title -->\n",
        "<g id=\"node1\" class=\"node\">\n",
        "<title>Title</title>\n",
-       "<g id=\"a_node1\"><a xlink:title=\"title_code           \r------------------------------\rfull_title           \r\">\n",
-       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"none\" points=\"56.5,-105.12 18.75,-105.12 18.75,-70.56 56.5,-70.56 56.5,-105.12\"/>\n",
-       "<text text-anchor=\"start\" x=\"26.75\" y=\"-85.72\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">Title</text>\n",
+       "<g id=\"a_node1\"><a xlink:title=\"title_code           &#13;------------------------------&#13;full_title           &#13;\">\n",
+       "<polygon fill=\"#000000\" fill-opacity=\"0.125490\" stroke=\"transparent\" points=\"57,-106 19,-106 19,-71 57,-71 57,-106\"/>\n",
+       "<text text-anchor=\"start\" x=\"27\" y=\"-87\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"10.00\">Title</text>\n",
        "</a>\n",
        "</g>\n",
        "</g>\n",
        "<!-- Employee -->\n",
        "<g id=\"node2\" class=\"node\">\n",
        "<title>Employee</title>\n",
-       "<g id=\"a_node2\"><a xlink:title=\"person_id            \r------------------------------\rfirst_name           \rlast_name            \r→ Title\r\">\n",
-       "<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"75.25,-34.56 0,-34.56 0,0 75.25,0 75.25,-34.56\"/>\n",
-       "<text text-anchor=\"start\" x=\"8\" y=\"-14.01\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">Employee</text>\n",
+       "<g id=\"a_node2\"><a xlink:title=\"person_id            &#13;------------------------------&#13;first_name           &#13;last_name            &#13;→ Title&#13;\">\n",
+       "<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\" points=\"76,-35 0,-35 0,0 76,0 76,-35\"/>\n",
+       "<text text-anchor=\"start\" x=\"8\" y=\"-15.4\" font-family=\"arial\" text-decoration=\"underline\" font-size=\"12.00\" fill=\"darkgreen\">Employee</text>\n",
        "</a>\n",
        "</g>\n",
        "</g>\n",
        "<!-- Title&#45;&gt;Employee -->\n",
        "<g id=\"edge1\" class=\"edge\">\n",
        "<title>Title-&gt;Employee</title>\n",
-       "<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-dasharray=\"5,2\" stroke-opacity=\"0.250980\" d=\"M37.62,-70.59C37.62,-59.82 37.62,-45.73 37.62,-34.89\"/>\n",
+       "<path fill=\"none\" stroke=\"#000000\" stroke-width=\"0.75\" stroke-dasharray=\"5,2\" stroke-opacity=\"0.250980\" d=\"M38,-70.8C38,-59.95 38,-45.87 38,-35.05\"/>\n",
        "</g>\n",
        "</g>\n",
        "</svg>"
       ],
       "text/plain": [
-       "<datajoint.diagram.Diagram at 0x7fe136962e50>"
+       "<datajoint.diagram.Diagram at 0xffff5ba52660>"
       ]
      },
-     "execution_count": 2,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -156,6 +139,23 @@
     "The parent table `Title` is above and the child table `Employee` is below."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```{admonition} A Logical Constraint, not a Physical Pointer\n",
+    ":class: tip\n",
+    "\n",
+    "A revolutionary concept in the relational model is that a foreign key is **not a physical pointer** to a location on a disk. Instead, it is a **logical constraint** enforced at runtime.\n",
+    "\n",
+    "When you try to insert a row into a child table, the database doesn't follow a pre-existing \"link.\" It performs a search on the parent table to see if a record with a matching primary key exists. If a match is found, the insert is allowed; otherwise, it is rejected.\n",
+    "\n",
+    "This is fundamentally different from other data models like HDF5, where data is often linked by direct pointers or paths [^1]. The logical nature of foreign keys gives relational databases their flexibility and integrity.\n",
+    "\n",
+    "[^1]: The HDF Group. \"HDF5 User's Guide: Groups and Links\". [https://docs.hdfgroup.org/hdf5/develop/H5.intro.html#intro-groups](https://docs.hdfgroup.org/hdf5/develop/H5.intro.html#intro-groups)\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -174,7 +174,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 5,
    "metadata": {},
    "outputs": [
     {
@@ -261,7 +261,7 @@
        " (Total: 0)"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -274,16 +274,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Effect 2: Inserts into the Child Table are Restricted\n",
     "\n",
-    "### Effect 2: Inserts into the child table are restricted unless there is a match in the parent table  \n",
-    "When inserting a new row into the child table, the database ensures that the foreign key value **must match a primary key** in the parent table. If no matching parent row exists, the insert is rejected, preventing **orphaned records** in the child table.\n",
+    "A foreign key ensures that no \"orphaned\" records are created. An insert into the child table is only permitted if the foreign key value corresponds to an existing primary key in the parent table.\n",
     "\n",
-    "For example, let's try inserting two employees. The first will use an existing title where as the other will use a new title."
+    "The rule is simple: **Inserts are restricted in the child, not the parent.** You can always add new job titles, but you cannot add an employee with a `title_code` that doesn't exist in the `Title` table."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 6,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -293,15 +293,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 7,
    "metadata": {},
    "outputs": [
     {
      "ename": "IntegrityError",
      "evalue": "Cannot add or update a child row: a foreign key constraint fails (`company`.`employee`, CONSTRAINT `employee_ibfk_1` FOREIGN KEY (`title_code`) REFERENCES `#title` (`title_code`) ON DELETE RESTRICT ON UPDATE CASCADE)",
      "output_type": "error",
      "traceback": [
-      "\u001b[0;31mIntegrityError\u001b[0m\u001b[0;31m:\u001b[0m Cannot add or update a child row: a foreign key constraint fails (`company`.`employee`, CONSTRAINT `employee_ibfk_1` FOREIGN KEY (`title_code`) REFERENCES `#title` (`title_code`) ON DELETE RESTRICT ON UPDATE CASCADE)\n"
+      "\u001b[31mIntegrityError\u001b[39m\u001b[31m:\u001b[39m Cannot add or update a child row: a foreign key constraint fails (`company`.`employee`, CONSTRAINT `employee_ibfk_1` FOREIGN KEY (`title_code`) REFERENCES `#title` (`title_code`) ON DELETE RESTRICT ON UPDATE CASCADE)\n"
      ]
     }
    ],
@@ -314,20 +314,40 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Effect 3. Deletes from the parent table are restricted for rows that have matching children  \n",
-    "A parent record cannot be deleted if it is referenced by any child records. This restriction prevents **broken relationships** between tables. The only way to delete the parent is to first delete or update the dependent child records, or to use a **cascading delete** that removes both parent and child rows.\n",
+    "### Effect 3: Deletes from the Parent Table are Restricted\n",
     "\n",
-    "- **Cascading Delete Option**: With cascading delete enabled, deleting a parent row automatically removes all its associated child rows.\n",
+    "To prevent broken relationships, a parent record cannot be deleted if any child records still refer to it.\n",
     "\n",
+    "The rule is the inverse of the insert rule: **Deletes are restricted in the parent, not the child.** You can always delete an employee, but you cannot delete a title if it is still assigned to an employee.\n",
     "\n",
-    "Deleting from `Title` will generate a warning that children in `Employee` will be deleted too. Such cascading will go down many levels through the hierarchy."
+    "In standard SQL, this operation would fail with a constraint error. DataJoint, however, implements a **cascading delete**. It will warn you that deleting the parent record will also delete all dependent child records, which can cascade through many levels of a deep hierarchy."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 8,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[2025-09-18 14:31:21,418][INFO]: Deleting 1 rows from `company`.`employee`\n",
+      "[2025-09-18 14:31:21,422][INFO]: Deleting 7 rows from `company`.`#title`\n",
+      "[2025-09-18 14:31:29,056][WARNING]: Delete cancelled\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "7"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "Title.delete()"
    ]
@@ -336,35 +356,28 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Effect 4. Restrict Updates to the Foreign Key and the Referenced Primary Key  \n",
-    "Foreign keys **restrict updates** on both the child and parent tables to maintain data consistency.\n",
-    "\n",
-    "DataJoint does not support updates of primary key values since such updates have the potential for breaking down referential integrity.\n",
-    "Normal data manipulations are performed by deletes and inserts. However SQL and relational theory more generally supports such operations. \n",
-    "\n",
-    "- **Updates in the Parent Table**: If the primary key of a parent record is updated, all dependent child records must be updated to maintain referential integrity. However, unless **cascading updates** are configured, these updates are blocked to prevent inconsistency.\n",
+    "### Effect 4: Updates to Referenced Keys are Restricted\n",
     "\n",
-    "- **Updates in the Child Table**: Similarly, updating the foreign key value in a child record is restricted to ensure that the new value matches a valid parent row.\n",
+    "To maintain referential integrity, updates to a parent's primary key or a child's foreign key are restricted.\n",
     "\n",
-    "- **Cascading Update Option**: With cascading updates enabled, changes to a parent’s primary key will automatically propagate to the related child records.\n",
+    "In general relational theory, databases can be configured to handle this with **cascading updates**, where changing a parent's primary key automatically propagates that change to all child records.\n",
     "\n",
-    "\n",
-    "### Effect 5. Create an index in the child table for fast searches on the foreign key  \n",
-    "To optimize performance, the database **automatically creates an index** on the foreign key column(s) in the child table. This index allows the database to efficiently find child records that refer to a specific parent row, improving query performance during joins and lookups."
+    "However, DataJoint does not support updating primary key values, as this can risk breaking referential integrity in complex scientific workflows. The preferred and safer pattern in DataJoint is to **delete the old record and insert a new one** with the updated information."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Summary  \n",
-    "Foreign keys ensure **referential integrity** by imposing constraints on how data is added, modified, and deleted across related tables. The five key effects are:\n",
+    "# Summary\n",
+    "\n",
+    "Foreign keys ensure referential integrity by linking a child table to a parent table. This link imposes five key effects:\n",
     "\n",
-    "1. **Schema Design**: The parent’s primary key becomes part of the child table as a foreign key.  \n",
-    "2. **Insert Restriction**: Inserts into the child table are blocked unless a matching parent row exists.  \n",
-    "3. **Delete Restriction**: Deleting a parent row is blocked unless dependent child rows are handled (or cascading delete is enabled).  \n",
-    "4. **Update Restriction**: Updates to the primary key in the parent table and foreign key in the child table are restricted to maintain consistency, unless cascading updates are explicitly allowed.\n",
-    "5. **Performance Optimization**: An index on the foreign key in the child table ensures fast searches and efficient joins.  \n"
+    "1.  **Schema Embedding**: The parent's primary key is added as columns to the child table.\n",
+    "2.  **Insert Restriction**: A row cannot be added to the **child** if its foreign key doesn't match a primary key in the **parent**.\n",
+    "3.  **Delete Restriction**: A row cannot be deleted from the **parent** if it is still referenced by any rows in the **child**.\n",
+    "4.  **Update Restriction**: Updates to the primary and foreign keys are restricted to prevent inconsistencies.\n",
+    "5.  **Performance Optimization**: An index is automatically created on the foreign key in the child table to speed up searches and joins."
    ]
   }
  ],
@@ -384,7 +397,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.10"
+   "version": "3.13.2"
   },
   "orig_nbformat": 4
  },