Skip to content

Commit c9169bf

Browse files
Copilotsamuelbray32
Copilot
and
samuelbray32
committed
Create comprehensive GitHub Copilot instructions for trodes_to_nwb
Co-authored-by: samuelbray32 <[email protected]>
1 parent bf8d56e commit c9169bf

File tree

1 file changed

+125
-0
lines changed

1 file changed

+125
-0
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# trodes_to_nwb
2+
3+
trodes_to_nwb converts data from SpikeGadgets .rec files to the NWB 2.0+ Data Format. It validates NWB files using the NWB Inspector for compatibility with the DANDI archive.
4+
5+
Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here.
6+
7+
## Working Effectively
8+
9+
### Environment Setup
10+
- Bootstrap the development environment:
11+
- `conda env create -f environment.yml` -- takes 5 minutes to complete. NEVER CANCEL. Set timeout to 10+ minutes.
12+
- `conda activate trodes_to_nwb`
13+
- **CRITICAL**: pip install will fail due to network timeouts to PyPI. Use the PYTHONPATH workaround instead:
14+
- `export PYTHONPATH=/home/runner/work/trodes_to_nwb/trodes_to_nwb/src:$PYTHONPATH`
15+
16+
### Development Workflow
17+
- **Working code setup**: The package works without pip install using PYTHONPATH:
18+
- `cd /home/runner/work/trodes_to_nwb/trodes_to_nwb`
19+
- `source /usr/share/miniconda/etc/profile.d/conda.sh && conda activate trodes_to_nwb`
20+
- `export PYTHONPATH=/home/runner/work/trodes_to_nwb/trodes_to_nwb/src:$PYTHONPATH`
21+
- `python -c "import trodes_to_nwb; print('Import successful')"`
22+
23+
### Building and Testing
24+
- **Run tests**: `python -m pytest src/trodes_to_nwb/tests/ -v` -- takes 1-2 minutes. NEVER CANCEL. Set timeout to 5+ minutes.
25+
- **NOTE**: Many tests fail due to missing large .rec data files (normal for development)
26+
- Integration tests that pass validate core functionality without data files
27+
- Tests that require .rec files will show `FileNotFoundError` (expected)
28+
- **Lint code**: `black --check --diff src/` -- takes < 1 second
29+
- **Format code**: `black src/` to auto-format files
30+
31+
### Validation Steps
32+
- **Always run the basic functionality test** after making changes:
33+
- Copy the validation script from `/tmp/test_basic_functionality.py` if available
34+
- Or run: `python -c "from trodes_to_nwb.convert import create_nwbs; import trodes_to_nwb; print(f'Package working, version: {trodes_to_nwb.__version__}')"` (takes ~2 seconds)
35+
- **Always run black formatting** before committing: `black src/`
36+
- **Test core imports** work: `from trodes_to_nwb.convert import create_nwbs`
37+
38+
## Key Information
39+
40+
### Package Structure
41+
- **Main package**: `src/trodes_to_nwb/`
42+
- **Core conversion**: `src/trodes_to_nwb/convert.py` - contains `create_nwbs()` function
43+
- **Tests**: `src/trodes_to_nwb/tests/` - unit and integration tests
44+
- **Documentation**: `docs/`, `notebooks/conversion_tutorial.ipynb`
45+
- **Metadata**: Device configs in `src/trodes_to_nwb/device_metadata/`
46+
47+
### Critical Installation Notes
48+
- **NEVER use pip install -e .** - it will timeout connecting to PyPI for ffmpeg package
49+
- **FFmpeg is available via conda** - don't try to install via pip
50+
- **Use PYTHONPATH approach** for development - fully functional
51+
- **Environment creation partially fails** - conda packages install but pip dependencies timeout (this is expected)
52+
53+
### Main Function Usage
54+
```python
55+
from trodes_to_nwb.convert import create_nwbs
56+
57+
# Core parameters:
58+
# path: pathlib.Path - directory containing .rec files and metadata YAML
59+
# output_dir: str - where to save .nwb files (default: '/stelmo/nwb/raw')
60+
# header_reconfig_path: optional path to header reconfiguration file
61+
# device_metadata_paths: optional list of device metadata YAML files
62+
# convert_video: bool - whether to convert .h264 to .mp4 (requires ffmpeg)
63+
# n_workers: int - parallel processing workers (default: 1)
64+
# query_expression: str - filter which files to convert (e.g., "animal == 'sample'")
65+
```
66+
67+
### File Naming Convention
68+
Data files must follow: `{YYYYMMDD}_{animal}_{epoch}_{tag}.{extension}`
69+
- `.rec`: binary ephys recording
70+
- `.h264`: video file
71+
- `.videoPositionTracking`: position tracking data
72+
- `.cameraHWSync`: position timestamps
73+
- `.stateScriptLog`: experimental parameters
74+
75+
Metadata file: `{YYYYMMDD}_{animal}_metadata.yml`
76+
77+
### GitHub Workflows
78+
- **Tests**: `.github/workflows/test_package_build.yml` - builds package, runs tests
79+
- **Linting**: `.github/workflows/lint.yml` - runs black formatting check
80+
- **CI downloads test data** from UCSF Box during workflow runs
81+
- **Coverage reports** upload to Codecov
82+
83+
## Common Tasks
84+
85+
### Repository Overview
86+
```
87+
├── src/trodes_to_nwb/ # Main package
88+
│ ├── convert.py # Core conversion functions
89+
│ ├── tests/ # Test suite
90+
│ └── device_metadata/ # Device configurations
91+
├── docs/ # Documentation
92+
├── notebooks/ # Jupyter tutorials
93+
├── environment.yml # Conda environment specification
94+
└── pyproject.toml # Package configuration
95+
```
96+
97+
### Timing Expectations
98+
- **Environment creation**: 5 minutes (conda packages only, pip fails)
99+
- **Package import**: 2 seconds
100+
- **Test suite**: 1-2 minutes (many tests skip due to missing data)
101+
- **Black linting**: < 1 second
102+
- **Basic functionality validation**: 2 seconds
103+
104+
### Known Issues and Workarounds
105+
- **pip install timeouts**: Use PYTHONPATH instead of pip install
106+
- **Missing test data**: Large .rec files not in repo, downloaded in CI
107+
- **ffmpeg dependency**: Use conda version, not pip version
108+
- **Network connectivity**: PyPI access may be limited in some environments
109+
110+
### Example Development Session
111+
```bash
112+
cd /home/runner/work/trodes_to_nwb/trodes_to_nwb
113+
conda activate trodes_to_nwb
114+
export PYTHONPATH=/home/runner/work/trodes_to_nwb/trodes_to_nwb/src:$PYTHONPATH
115+
116+
# Test your changes
117+
python -c "from trodes_to_nwb.convert import create_nwbs; print('Working')"
118+
black src/
119+
python -m pytest src/trodes_to_nwb/tests/ -x # Stop on first failure
120+
```
121+
122+
### Before Committing
123+
- Run `black src/` to format code
124+
- Verify imports work: `python -c "import trodes_to_nwb"`
125+
- Consider running a few tests: `python -m pytest src/trodes_to_nwb/tests/integration-tests/ -v`

0 commit comments

Comments
 (0)