Skip to content

Commit 373365b

Browse files
alexevagalexevag
committed
Implementation of NWB export utilities for Ethopy
Add NWB export module to convert Ethopy DataJoint sessions to NWB files. - Implements export_to_nwb and batch_export_to_nwb with robust error handling, logging, and flexible metadata. - Supports subject info, session/trial/condition export, activity, reward, stimulus, and state data. - Includes directory creation, overwrite protection, and config path support. - Adds detailed docstrings and logging for maintainability and user guidance. for more details check the nwb_docs.md
1 parent 7dcc558 commit 373365b

File tree

4 files changed

+1317
-0
lines changed

4 files changed

+1317
-0
lines changed

‎docs/nwb_docs.md‎

Lines changed: 367 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,367 @@
1+
# NWB Export Documentation
2+
3+
## Overview
4+
5+
The NWB export module provides functionality to export experimental data from Ethopy DataJoint tables to NWB (Neurodata Without Borders) format files. This documentation covers the main export functions and their usage.
6+
7+
## Main Functions
8+
9+
### `export_to_nwb()`
10+
11+
The primary function for exporting a single experimental session to NWB format.
12+
13+
#### Required Parameters
14+
15+
| Parameter | Type | Description |
16+
|-----------|------|-------------|
17+
| `animal_id` | `int` | Unique identifier for the animal |
18+
| `session_id` | `int` | Session identifier |
19+
20+
#### Optional Parameters
21+
22+
##### NWB File Parameters
23+
24+
| Parameter | Type | Default | Description |
25+
|-----------|------|---------|-------------|
26+
| `experimenter` | `str` | `"Unknown"` | Name of the experimenter |
27+
| `lab` | `str` | `"Your Lab Name"` | Laboratory name |
28+
| `institution` | `str` | `"Your Institution"` | Institution name |
29+
| `session_description` | `str` | Auto-generated | Description of the experimental session |
30+
31+
##### Subject Parameters
32+
33+
| Parameter | Type | Default | Description |
34+
|-----------|------|---------|-------------|
35+
| `age` | `str` | `"Unknown"` | Age in ISO 8601 duration format |
36+
| `subject_description` | `str` | `"laboratory mouse"` | Description of the subject |
37+
| `species` | `str` | `Unknown"` | Species of the subject |
38+
| `sex` | `str` | `"U"` | Sex: "M", "F", "U" (unknown), or "O" (other) |
39+
40+
##### Additional Options
41+
42+
| Parameter | Type | Default | Description |
43+
|-----------|------|---------|-------------|
44+
| `output_filename` | `str` | Auto-generated | Output filename for the NWB file |
45+
| `overwrite` | `bool` | `False` | Whether to overwrite existing files |
46+
| `return_nwb_object` | `bool` | `False` | Return both filename and NWB object |
47+
| `config_path` | `str` | `~/.ethopy/local_conf.json` | Path to the local configuration file for DataJoint and schema setup. If not provided, the default Ethopy config is used. This parameter is passed to the DataJoint connection setup and allows exporting from different databases or configurations. |
48+
49+
#### Returns
50+
51+
- `str`: Path to the saved NWB file (default)
52+
- `Tuple[str, NWBFile]`: Path and NWB object if `return_nwb_object=True`
53+
54+
#### Raises
55+
56+
- `ValueError`: If no session is found for the provided animal_id and session_id
57+
- `FileExistsError`: If output file exists and `overwrite=False`
58+
59+
### `batch_export_to_nwb()`
60+
61+
Export multiple sessions to NWB format in batch.
62+
63+
#### Parameters
64+
65+
| Parameter | Type | Description |
66+
|-----------|------|-------------|
67+
| `session_list` | `List[Tuple[int, int]]` | List of (animal_id, session_id) tuples |
68+
| `output_directory` | `str` | Directory to save NWB files (default: "nwb_exports") |
69+
| `**kwargs` | | Additional parameters passed to `export_to_nwb()` |
70+
71+
#### Returns
72+
73+
- `List[str]`: List of successfully exported filenames
74+
75+
## Usage Examples
76+
77+
### Basic Usage
78+
79+
Export with minimal parameters using all defaults:
80+
81+
```python
82+
filename = export_to_nwb(animal_id=123, session_id=1)
83+
```
84+
85+
This will create a file named `nwb_animal_123_session_1.nwb` with default metadata.
86+
87+
### Custom Parameters
88+
89+
Export with custom metadata and subject information:
90+
91+
```python
92+
filename = export_to_nwb(
93+
animal_id=123,
94+
session_id=1,
95+
experimenter="Alex Smith",
96+
lab="Systems Neuroscience Lab",
97+
institution="FORTH IMBB",
98+
session_description="2AFC task",
99+
age="P120D", # 120 days old
100+
subject_description="Wild-type C57BL/6J mouse, head-fixed",
101+
sex="F",
102+
overwrite=True
103+
)
104+
```
105+
106+
### Custom Filename
107+
108+
Specify a custom output filename:
109+
110+
```python
111+
filename = export_to_nwb(
112+
animal_id=123,
113+
session_id=1,
114+
output_filename="TwoAFCexperiment_session_1.nwb",
115+
experimenter="Alex"
116+
)
117+
```
118+
119+
### Clarify the configuration path
120+
```python
121+
filename = export_to_nwb(
122+
animal_id=123,
123+
session_id=1,
124+
config_path="/path/to/custom_config.json"
125+
)
126+
```
127+
Example of configuration file
128+
129+
The configuration file is a JSON file that specifies how to connect to your DataJoint database and which schema names to use for your experiment. You can provide a custom configuration file using the `config_path` parameter in `export_to_nwb`. This allows you to export data from different databases or with different schema setups, without changing your code.
130+
131+
**Parameter explanations:**
132+
133+
- **dj_local_conf**:
134+
Contains the database connection settings for DataJoint.
135+
- `database.host`: The hostname or IP address of your MySQL server.
136+
- `database.user`: The username for connecting to the database.
137+
- `database.password`: The password for the database user.
138+
- `database.port`: The port number for the database (default is usually 3306).
139+
- `database.reconnect`: Whether to automatically reconnect if the connection drops.
140+
- `database.use_tls`: Whether to use TLS/SSL for the connection.
141+
- `datajoint.loglevel`: The logging level for DataJoint messages (e.g., "WARNING", "INFO").
142+
143+
- **SCHEMATA**:
144+
Maps logical names to the actual schema names in your database. This tells Ethopy which schemas to use for each data type.
145+
- `experiment`: The schema name for experiment/session tables.
146+
- `stimulus`: The schema name for stimulus tables.
147+
- `behavior`: The schema name for behavior tables.
148+
149+
**Example:**
150+
```json
151+
{
152+
"dj_local_conf": {
153+
"database.host": "localhost",
154+
"database.user": "myuser",
155+
"database.password": "mypassword",
156+
"database.port": 3306,
157+
"database.reconnect": true,
158+
"database.use_tls": false,
159+
"datajoint.loglevel": "WARNING"
160+
},
161+
"SCHEMATA": {
162+
"experiment": "lab_experiments",
163+
"stimulus": "lab_stimuli",
164+
"behavior": "lab_behavior",
165+
}
166+
}
167+
```
168+
169+
### Return NWB Object for Further Processing
170+
171+
Get both the filename and NWB object for additional processing:
172+
173+
```python
174+
filename, nwb_obj = export_to_nwb(
175+
animal_id=123,
176+
session_id=1,
177+
return_nwb_object=True
178+
)
179+
180+
# Now you can access the NWB object directly
181+
print(f"NWB file contains {len(nwb_obj.trials)} trials")
182+
```
183+
### Overwriting
184+
The code raises FileExistsError if the file exists and overwrite=False.
185+
186+
```python
187+
filename, nwb_obj = export_to_nwb(
188+
animal_id=123,
189+
session_id=1,
190+
overwrite=True
191+
)
192+
```
193+
194+
### Batch Export
195+
196+
Export multiple sessions at once:
197+
198+
```python
199+
# Define sessions to export
200+
animal_session_list = [
201+
(123, 1),
202+
(123, 2),
203+
(124, 1),
204+
(124, 2)
205+
]
206+
207+
# Batch export with common parameters
208+
exported_files = batch_export_to_nwb(
209+
sessions,
210+
experimenter="Dr. Smith",
211+
lab="Vision Lab",
212+
institution="FORTH IMBB",
213+
output_directory="my_nwb_exports"
214+
)
215+
216+
print(f"Successfully exported {len(exported_files)} files")
217+
```
218+
219+
## Parameter Guidelines
220+
221+
### Age Format (ISO 8601 Duration)
222+
223+
The age parameter should follow ISO 8601 duration format:
224+
225+
- `P90D` = 90 days
226+
- `P3M` = 3 months
227+
- `P1Y6M` = 1 year and 6 months
228+
- `P2Y` = 2 years
229+
- `P1Y2M15D` = 1 year, 2 months, and 15 days
230+
231+
### Sex Values
232+
233+
| Value | Description |
234+
|-------|-------------|
235+
| `"M"` | Male |
236+
| `"F"` | Female |
237+
| `"U"` | Unknown |
238+
| `"O"` | Other |
239+
240+
## Data Included in NWB Files
241+
242+
The export function includes the following data types:
243+
244+
### Core Data
245+
- **Session metadata**: Timestamps, experimenter, lab, institution
246+
- **Subject information**: Species, age, sex, description
247+
- **Trials**: Trial timing, conditions, and metadata
248+
249+
### Experimental Data
250+
- **Conditions**: Experiment, stimulus, and behavior condition parameters
251+
- **Activity data**: Behavioral activity measurements
252+
- **Reward data**: Reward delivery timestamps and amounts
253+
- **Stimulus data**: Stimulus presentation timing and parameters
254+
- **States data**: State transition timestamps for each trial
255+
256+
### Data Organization
257+
- **Processing modules**: Data organized into logical modules (Conditions, Activity, Reward, States)
258+
- **Dynamic tables**: Flexible storage for experimental parameters
259+
- **Time series**: Timestamped behavioral events
260+
- **Behavioral events**: Reward delivery and other behavioral markers
261+
262+
## Error Handling
263+
264+
The export functions provide comprehensive error handling:
265+
266+
### Common Errors
267+
268+
1. **Session not found**: If the specified animal_id and session_id combination doesn't exist
269+
2. **File exists**: If the output file already exists and `overwrite=False`
270+
3. **No valid trials**: If no trials with both PreTrial and InterTrial states are found
271+
4. **Database connection**: If DataJoint connection fails
272+
273+
**Missing parameters**
274+
If certain data (such as specific trial fields, behavioral events, or condition tables) are missing or incomplete for a session, the export function will skip those data elements and continue the export process.
275+
A warning message will be logged to inform you about any missing or incomplete data, so you can review and address these issues if needed. This ensures that a valid NWB file is still created even when some parts of the dataset are unavailable.
276+
277+
### Error Recovery
278+
279+
- The function will attempt to create the NWB file even if some data is missing
280+
- Warning messages are displayed for missing or incomplete data
281+
- Batch export continues processing remaining sessions if individual exports fail
282+
283+
## Best Practices
284+
285+
### File Naming
286+
- Use descriptive filenames that include animal ID, session ID, and experiment type
287+
- Use consistent naming conventions across your lab
288+
289+
### Metadata Completeness
290+
- Always provide experimenter name and institution
291+
- Include detailed session descriptions
292+
- Specify accurate subject information (age, sex, strain)
293+
294+
### Batch Processing
295+
- Use batch export for multiple sessions with similar parameters
296+
- Create separate output directories for different experiments
297+
- Monitor the console output for failed exports
298+
299+
### File Management
300+
- Regularly backup NWB files
301+
- Use version control for analysis scripts
302+
- Document any post-processing steps applied to NWB files
303+
304+
## Troubleshooting
305+
306+
### Common Issues
307+
308+
1. **Import errors**: Ensure all required packages are installed (pynwb, datajoint, numpy, pandas)
309+
2. **Database connection**: Verify DataJoint configuration and database access
310+
3. **Memory issues**: For large sessions, consider processing in smaller batches
311+
4. **File permissions**: Ensure write permissions for the output directory
312+
313+
## Advanced Usage
314+
315+
### Custom Processing
316+
317+
If you need to modify the NWB file after creation:
318+
319+
```python
320+
# Get the NWB object
321+
filename, nwb_obj = export_to_nwb(
322+
animal_id=123,
323+
session_id=1,
324+
return_nwb_object=True
325+
)
326+
327+
# Add custom data
328+
from pynwb import TimeSeries
329+
custom_data = TimeSeries(
330+
name="custom_signal",
331+
data=[1, 2, 3, 4, 5],
332+
timestamps=[0.1, 0.2, 0.3, 0.4, 0.5],
333+
unit="arbitrary_units"
334+
)
335+
nwb_obj.add_acquisition(custom_data)
336+
337+
# Save the modified file
338+
from pynwb import NWBHDF5IO
339+
with NWBHDF5IO("modified_" + filename, "w") as io:
340+
io.write(nwb_obj)
341+
```
342+
343+
### Integration with Analysis Pipelines
344+
345+
The exported NWB files can be easily integrated with analysis pipelines:
346+
347+
```python
348+
from pynwb import NWBHDF5IO
349+
350+
# Read the NWB file
351+
with NWBHDF5IO("nwb_animal_123_session_1.nwb", "r") as io:
352+
nwb_file = io.read()
353+
354+
# Access trial data
355+
trials_df = nwb_file.trials.to_dataframe()
356+
357+
# Access behavioral events
358+
rewards = nwb_file.processing['Reward']['BehavioralEvents']['response_reward']
359+
360+
# Access conditions
361+
conditions = nwb_file.processing['Conditions']
362+
363+
# get Experiment conditions for each trial (similar for Stimulus/stim_hash, Behavior/beh_hash)
364+
exp_conds = nwb_file.processing['Conditions'].get('Experiment').to_dataframe()
365+
trials = nwb_file.trials.to_dataframe()
366+
merged_df = trials.merge(exp_conds, on="cond_hash", how="inner")
367+
```

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ nav:
8888
- Logging: logging.md
8989
- Setup Configuration Index: setup_configuration_idx.md
9090
- Database Setup: database.md
91+
- Export to NWB: nwb_docs.md
9192
- Troubleshooting: troubleshooting.md
9293
- API Reference:
9394
- Logger Module: API/logger.md

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ dependencies = [
3535
"pyfiglet",
3636
"scikit-learn>=1.3.2",
3737
"setuptools>=75.3.0",
38+
"pynwb",
3839
]
3940

4041
[project.optional-dependencies]

0 commit comments

Comments
 (0)