Optical physiology in spyglass

[rozmar]

Hello,
I am starting a lab that's going to be heavy on mouse behavior and planar optical physiology. I wonder if anyone has started working on optical schemas already in spyglass. I have some older datajoint schemas for 2p imaging and head fixed trial-based behavior, but would be happy to adopt well established schemas from others, if that makes sense for me..
Any information would be much appreciated about previous or ongoing work that I could join/collaborate with/learn from.

[samuelbray32]

Hi @rozmar - Thanks for getting in touch! Brief summary on our status on these:

Optogenetics
We plan to support ingestion of optogenetics metadata stored objects from the ndx-optogenetics extension withing the core spyglass package. If there is information not in these objects that you think would be common across labs feel free to let us know

Optophysiology
There have been a few discussions of adding new ingestion pipelines for photometry (#1284) data, miniscope data (@josueregalado96) in recent months, but I'm not aware of anything ready for PR as a shared resource. I've put together some docs on Spyglass development, and specifically custom pipelines that might be useful. We'd be grateful for contributions - and happy to meet to chat over specifics as you're drafting. To aid in that:

• Do you have any sample data you can post in nwb format?
• Could you point us to the datajoint pipeline(s) you mentioned?

[rozmar]

Hi @samuelbray32 ! Thank you for the quick response and the suggestions!

This is the pipeline I used earlier for 2p imaging: https://github.com/rozmar/jGCaMP8_mouse_cell_attached_pipeline
Note that I'll do a different experiment with more behavior and multi-plane imaging, so I'll definitely need to change the schemas. But the raw data will be still generated by ScanImage, primary analysis likely with Suite2p. It would be great if you could connect me with others going down this path, even if their schema is not ready either. - I'll also try to recruit more labs. :)

We will first generate behavior only and behavior+optogenetics experiments, which will be the pilot dataset for spyglass (that should work mostly out of the box with minimal custom schemas, I suppose), so we have some time to develop the imaging analysis part, but I'd like to get to it as soon as possible and not leave it to the last minute.

I can surely package some older data to .nwb (in a few weeks, possibly). Should I package the raw data + metadata in nwb, or should I first do some primary analysis (motion correction, ROI segmentation trace extraction), and package only this lighter data in nwb. Any other specifications?

This is off-topic, but I wanted to ask about best practices for setting up the server. I am interested in every technical detail that I'll need to keep an eye on. There is an already existing hardware/software environment at the institute that I was planning to use, or upgrade as soon as possible, if needed - so this is higher priority.

[CBroz1]

Should I package the raw data + metadata in nwb, or should I first do some primary analysis (motion correction, ROI segmentation trace extraction), and package only this lighter data in nwb. Any other specifications?

For pipeline development, I recommend putting together a 'minimum viable product' raw dataset - this is something that contains all the relevant modalities, and standard cases, but is maybe only a few trials or a few seconds of recording. We use a small example file to run all of our pytests. Something small is easily sharable across the dev team and makes it quick to iterate on design in 'test-driven development'

best practices for setting up the server

We have the benefit of having a local sysadmin to take care of server maintenance and backups. From his perspective, he set up and maintains a MySQL 8 server, with some custom configurations for meeting UCSF IT standards. The Spyglass team manages user permissions and file cleanup processes. I put together some docs here and here to cover some of the backup procedures and user/file management. If you already have a MySQL 8 server, and enough hard drive space to handle your data, it's worth reading up on user permission management best practices and the tools MySQL offers

[rozmar]

Thank you for the detailed response.
Can you comment more on the hardware requirements side?
storage: Would SAS with SSD cache be good?
I assume network speed has to be as fast as possible when communicating with the sql server. The institute currently has 1Gb/s network, should we install a separate faster network (e.g.10Gb/s) for analyzing data?
Any other hardware-side suggestion would be much appreciated, I don't have a good intuition.

[CBroz1]

Unfortunately, I know relatively little about the hardware requirements. We use a series of headless ubuntu nodes with access to both the sql and storage instances, all in the same physical space. The connection speed to the sql instance is less important than the connection between python and storage instances. Most sql queries are only fetching metadata, while raw/analysis files are stored on disk. It's highly dependent on your experimental setup, but, for reference, our average raw file is 9.8 GB and our average analysis file es 66 MB