Skip to content

Commit 6e8b273

Browse files
improve Building Relationships
1 parent e9b5f90 commit 6e8b273

File tree

6 files changed

+1693
-1403
lines changed

6 files changed

+1693
-1403
lines changed

book/00-introduction/00-purpose.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,12 @@ title: Purpose
33
---
44

55
## This Book
6-
This book is a comprehensive guide to *DataJoint for Python*, a framework for building reliable and scalable scientific data workflows.
6+
This book is an introductory guide to *DataJoint for Python*, a framework for building reliable and scalable scientific data workflows.
77
It is designed for collaborative, data-intensive research, where the complexity of data and computations requires a principled approach.
88

99
DataJoint was developed to overcome the limitations of managing research with scripts, spreadsheets, and complex folder structures—an approach that is often slow, error-prone, and difficult to scale.
1010
The value of a more rigorous framework was demonstrated in ambitious projects like **MICrONS (Machine Intelligence from Cortical Networks)** [@10.1038/s41586-025-08790-w]. A nine-year effort to map a piece of the brain, MICrONS generated a deluge of data from electron microscopy, neurophysiology, and animal behavior. A project of this scale and complexity would have been intractable with traditional methods. DataJoint, already a mature framework by the start of the project, proved essential for managing the data pipeline and enabling a large team to collaborate effectively.
1111

12-
1312
Most research begins with ad-hoc processes, managing data with scripts, spreadsheets, and complex folder structures. This approach often proves slow, error-prone, and difficult to scale, especially in collaborative projects. This challenge became starkly apparent during my work on **MICrONS (Machine Intelligence from Cortical Networks)** [@10.1038/s41586-025-08790-w], a nine-year effort to map a piece of the brain that generated a deluge of data from electron microscopy, neurophysiology, and animal behavior.
1413

1514
The traditional methods simply collapsed under this complexity. This experience led directly to the creation of **DataJoint**, a tool designed to bring the rigor of relational databases to the dynamic and evolving world of scientific research.

book/10-setup/02-connect.ipynb

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,12 @@
2020
"This textbook comes with a Devcontainer that runs a MySQL database server with the following credentials:\n",
2121
"- database server address: `db`\n",
2222
"- username: `dev`\n",
23-
"- password: `devpass`"
23+
"- password: `devpass`\n",
24+
"\n",
25+
"\n",
26+
"These These credentials are not secret since this database is not exposed to the external world.\n",
27+
"\n",
28+
"In this container environment, the user credentials are set in as the environment variables `DJ_HOST`, `DJ_USER`, and `DJ_PASS` and that's what the DataJoint client library will use to connect to the database.\n"
2429
]
2530
},
2631
{
@@ -77,10 +82,7 @@
7782
"source": [
7883
"# Connect with DataJoint \n",
7984
"\n",
80-
"The Devcontainer that comes with this textbook contains a running MySQL server.\n",
8185
"\n",
82-
"The user credentials are set in environment variables `DJ_HOST`, `DJ_USER`, and `DJ_PASS`.\n",
83-
"These credentials are not secret since this database is not exposed to the external world.\n",
8486
"\n",
8587
"The DataJoint client library uses these environment variables to connect to the database.\n",
8688
"It sufficient to simply import the datajoint client library.\n",

book/30-schema-design/025-uuid.ipynb

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,17 @@
99
"It is often becomes necessary to assign unique identifiers to entities in a database as *surragate primary key* values. There are many schemes for generating such identifiers.\n",
1010
"One of the most common approaches in simple cases is an autoincrementing integer field. Most SQL databases support this with built-in features such as `AUTO_INCREMENT` in MySQL or `SERIAL` in PostgreSQL. It's simple and fast.\n",
1111
"\n",
12-
"However, autoincrementing ids are not suitable for large distributed systems such as social media platforms (e.g. Instagram). It may be helpful to generate unique identifiers that do not require a database connection or a round-trip to the server.\n",
13-
"\n",
12+
"However, autoincrementing IDs are not suitable for large distributed systems such as social media platforms or e-commerce websites. It may be helpful to generate unique identifiers that do not require a database connection or a round-trip to the server.\n",
1413
"\n",
1514
"Reading: \n",
16-
"* Various Unique Identifiers: https://osamadev.medium.com/understanding-unique-identifiers-uid-uuid-guid-cuid-and-nano-id-3ef2d104ecdf\n",
17-
"* UUIDs: https://en.wikipedia.org/wiki/Universally_unique_identifier\n",
18-
"* NANOID: https://github.com/ai-sdk/nanoid-python\n",
19-
"* NANOID Collision Calculator:https://zelark.github.io/nano-id-cc/\n",
15+
"* Comparing Unique Identifiers: https://osamadev.medium.com/understanding-unique-identifiers-uid-uuid-guid-cuid-and-nano-id-3ef2d104ecdf\n",
16+
"* **UUIDs**: https://en.wikipedia.org/wiki/Universally_unique_identifier\n",
17+
"* **NANOID**: https://github.com/ai-sdk/nanoid-python\n",
18+
"* **NANOID Collision Calculator**:https://zelark.github.io/nano-id-cc/\n",
2019
"* Sharding IDs at Instagram: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c\n",
21-
"* ULID: https://medium.com/@juniooor/ulid-universally-unique-lexicographically-sortable-identifier-5544e3391660\n",
22-
"* ULID: https://github.com/ulid/spec\n",
23-
"\n",
20+
"ulid-universally-unique-lexicographically-sortable-identifier-5544e3391660\n",
21+
"* **ULID**: https://github.com/ulid/spec\n",
22+
"* **ULID**: https://medium.com/@juniooor/\n",
2423
"\n",
2524
"## Universally Unique Identifiers (UUIDs)\n",
2625
"\n",
@@ -1465,7 +1464,7 @@
14651464
},
14661465
{
14671466
"cell_type": "code",
1468-
"execution_count": 49,
1467+
"execution_count": null,
14691468
"metadata": {},
14701469
"outputs": [
14711470
{

0 commit comments

Comments
 (0)