Skip to content

Commit 2c02fca

Browse files
dimitri-yatsenkodimitri-yatsenko
committed
expand Create Schemas 
1 parent 6b4ba7e commit 2c02fca

File tree

1 file changed

+126
-4
lines changed

1 file changed

+126
-4
lines changed

‎book/30-schema-design/010-schema.ipynb‎

Lines changed: 126 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,7 @@
4646
"This modular approach:\n",
4747
"* Separates tables into logical groups for better organization.\n",
4848
"* Avoids naming conflicts in large databases with multiple schemas.\n",
49-
"\n",
50-
"For more details on designing multi-schema databases, refer to the section on multi-schema designs."
49+
"\n"
5150
]
5251
},
5352
{
@@ -102,10 +101,133 @@
102101
]
103102
},
104103
{
105-
"cell_type": "markdown",
104+
"cell_type": "code",
105+
"execution_count": null,
106106
"metadata": {},
107+
"outputs": [],
107108
"source": []
108109
},
110+
{
111+
"cell_type": "markdown",
112+
"metadata": {},
113+
"source": [
114+
"# Working with Multi-Schema Databases\n",
115+
"\n",
116+
"Organizing larger databases into multiple smaller schemas (or modules) enhances clarity, modularity, and maintainability. In DataJoint, schemas serve as namespaces that group related tables together, while Python modules provide a corresponding organizational structure for the database code.\n",
117+
"\n",
118+
"## Convention: One Database Schema = One Python Module\n",
119+
"\n",
120+
"DataJoint projects are typically organized with each database schema mapped to a single Python module (`.py` file). This convention:\n",
121+
"\n",
122+
"* Promotes modularity by grouping all tables of a schema within one module.\n",
123+
"* Ensures clarity by maintaining a single schema object per module.\n",
124+
"* Avoids naming conflicts and simplifies dependency management.\n",
125+
"\n",
126+
"Each module declares its own schema object and defines all associated tables. Downstream schemas explicitly import upstream schemas to manage dependencies.\n",
127+
"\n",
128+
"## Dependency Management and Acyclic Design\n",
129+
"\n",
130+
"In multi-schema databases, dependencies between tables and schemas must form a Directed Acyclic Graph (DAG). Cyclic dependencies are not allowed. This ensures:\n",
131+
"* Foreign key constraints maintain logical order without forming loops.\n",
132+
"* Python module imports align with the dependency structure of the database.\n",
133+
"\n",
134+
"**Key Principles**:\n",
135+
"1. Tables can reference each other within a schema or across schemas using foreign keys.\n",
136+
"2. Dependencies should be topologically sorted, ensuring upstream schemas are imported into downstream schemas.\n",
137+
"\n",
138+
"# Advantages of Multi-Schema Design\n",
139+
"1. **Modularity**: Each schema focuses on a specific aspect of the pipeline (e.g., acquisition, processing, analysis).\n",
140+
"2. **Separation of Concerns**: Clear boundaries between schemas simplify navigation and troubleshooting.\n",
141+
"3. **Scalability**: Isolated schemas enable easier updates and scaling as projects grow.\n",
142+
"4. **Collaboration**: Teams can work on separate modules independently without conflicts.\n",
143+
"5. **Maintainability**: Modular design facilitates version control and debugging.\n",
144+
"\n",
145+
"# Defining Complex Databases with Multiple Schemas in DataJoint\n",
146+
"\n",
147+
"In DataJoint, defining **multiple schemas across separate Python modules** ensures that large, complex projects remain well-organized, modular, and maintainable. Each schema should be defined in a **dedicated Python module** to adhere to best practices. This structure ensures that every module maintains **only one `schema` object**, and **downstream schemas import upstream schemas** to manage dependencies correctly. This approach improves code clarity, enables better version control, and simplifies collaboration across teams.\n",
148+
"\n",
149+
"The database schema and its Python module usually have similar names, although they need not be identical. \n",
150+
"\n",
151+
"Tables can form foreign key dependencies within modules and but also across modules. \n",
152+
"In DataJoint, Such dependencies must be acyclic within each schema: dependencies cannot form closed cycles, so that the graph of dependences forms a DAG (directed acyclic graph). \n",
153+
"Then also database modules form a directed acyclic graph at a higher level: the python modules should never form cyclic import dependences and their database schemas must be topologically sorted in the same way so that tables cannot make foreign key dependencies into tables that are in downstream schemas.\n",
154+
"\n",
155+
"\n",
156+
"## Why Use Multiple Schemas in Separate Modules?\n",
157+
"\n",
158+
"Using multiple schemas across separate modules offers the following benefits:\n",
159+
"\n",
160+
"1. **Modularity and Code Organization**: Each module contains only the tables relevant to a specific schema, making the codebase easier to manage and navigate.\n",
161+
"2. **Clear Boundaries Between Schemas**: Ensures a separation of concerns, where each schema focuses on a specific aspect of the pipeline (e.g., acquisition, processing, analysis).\n",
162+
"3. **Dependency Management**: Downstream schemas explicitly **import upstream schemas** to manage table dependencies and data flow.\n",
163+
"4. **Collaboration**: Multiple developers or teams can work on separate modules without conflicts.\n",
164+
"5. **Scalability and Maintainability**: Isolating schemas into modules simplifies future updates and troubleshooting.\n",
165+
"\n",
166+
"\n",
167+
"## How to Structure Modules for Multiple Schemas\n",
168+
"\n",
169+
"Below is an example that demonstrates how to organize multiple schemas in separate Python modules.\n",
170+
"\n",
171+
"# Example Project Structure\n",
172+
"\n",
173+
"Here’s an example of how to organize multiple schemas in a DataJoint project:\n",
174+
"\n",
175+
"```\n",
176+
"my_pipeline/\n",
177+
"│\n",
178+
"├── subject.py # Defines subject_management schema\n",
179+
"├── acquisition.py # Defines acquisition schema (depends on subject_management)\n",
180+
"├── processing.py # Defines processing schema (depends on acquisition)\n",
181+
"└── analysis.py # Defines analysis schema (depends on processing)\n",
182+
"```\n",
183+
"\n",
184+
"## Step-by-Step Example\n",
185+
"\n",
186+
"1. `subject.py`:\n",
187+
" * Defines the `subject_management` schema.\n",
188+
" * Contains the Subject table and related entities.\n",
189+
"2. `acquisition.py`:\n",
190+
" * Defines the `acquisition` schema.\n",
191+
" * Depends on subject_management for subject-related data.\n",
192+
"3. `processing.py`:\n",
193+
" * Defines the `processing` schema.\n",
194+
" * Depends on `acquisition` for data to process.\n",
195+
"4. `analysis.py`:\n",
196+
" * Defines the `analysis` schema.\n",
197+
" * Depends on `processing` for processed data to analyze.\n",
198+
"\n",
199+
"By adhering to these principles, large projects remain modular, scalable, and easy to maintain.\n"
200+
]
201+
},
202+
{
203+
"cell_type": "code",
204+
"execution_count": 1,
205+
"metadata": {},
206+
"outputs": [
207+
{
208+
"name": "stdout",
209+
"output_type": "stream",
210+
"text": [
211+
"\u001b[0;32mimport\u001b[0m \u001b[0mdatajoint\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mdj\u001b[0m\u001b[0;34m\u001b[0m\n",
212+
"\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
213+
"\u001b[0;34m\u001b[0m\u001b[0;31m# Define the subject management schema\u001b[0m\u001b[0;34m\u001b[0m\n",
214+
"\u001b[0;34m\u001b[0m\u001b[0mschema\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSchema\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"subject_management\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\n",
215+
"\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\n",
216+
"\u001b[0;34m\u001b[0m\u001b[0;34m@\u001b[0m\u001b[0mschema\u001b[0m\u001b[0;34m\u001b[0m\n",
217+
"\u001b[0;34m\u001b[0m\u001b[0;32mclass\u001b[0m \u001b[0mSubject\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mManual\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
218+
"\u001b[0;34m\u001b[0m \u001b[0mdefinition\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\"\"\u001b[0m\n",
219+
"\u001b[0;34m subject_id : int\u001b[0m\n",
220+
"\u001b[0;34m ---\u001b[0m\n",
221+
"\u001b[0;34m subject_name : varchar(50)\u001b[0m\n",
222+
"\u001b[0;34m species : varchar(50)\u001b[0m\n",
223+
"\u001b[0;34m \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n"
224+
]
225+
}
226+
],
227+
"source": [
228+
"%pycat code/subject.py"
229+
]
230+
},
109231
{
110232
"cell_type": "markdown",
111233
"metadata": {},
@@ -146,7 +268,7 @@
146268
"name": "python",
147269
"nbconvert_exporter": "python",
148270
"pygments_lexer": "ipython3",
149-
"version": "3.10.14"
271+
"version": "3.11.10"
150272
}
151273
},
152274
"nbformat": 4,

0 commit comments

Comments
 (0)