Store ElectricalSeries in uV for spyglass compatability

[calderast]

Spyglass ignores ElectricalSeries conversion factors and just assumes everything is in uV. I will submit a spyglass PR to edit this, but just storing ours in uV is a reasonable workaround for now.

P.S. I am writing this in the airport on the way to Berlin, so I didn't test this even a little bit. But I wanted to submit it rn so people aren't blocked. Sorry to my fans (and haters) if this breaks anything

[rly]

Also see discussion in LorenFrankLab/trodes_to_nwb#140

FYI this goes against the NWB standard where raw array data * global conversion factor * channel conversion factor should be stored in volts. https://github.com/NeurodataWithoutBorders/nwb-schema/blob/dev/core/nwb.ecephys.yaml#L36 . So PyNWB does not let you pass unit="microvolts" to ElectricalSeries.

Also if the array data are stored in microvolts, then the conversion factor (or channel conversion factor) should then be 1e6.

[calderast]

Also see discussion in LorenFrankLab/trodes_to_nwb#140

FYI this goes against the NWB standard where raw array data * global conversion factor * channel conversion factor should be stored in volts. https://github.com/NeurodataWithoutBorders/nwb-schema/blob/dev/core/nwb.ecephys.yaml#L36 . So PyNWB does not let you pass unit="microvolts" to ElectricalSeries.

Also if the array data are stored in microvolts, then the conversion factor (or channel conversion factor) should then be 1e6.

Got it, I’ll change that to be a conversion factor to V - FWIW, I think the current way we store things (raw data and conversion factor, consistent with the NWB standard) is preferable, and I agree with all of Kyu’s points in the spyglass discussion. But I am trying to get better at choosing the path of least resistance re spyglass… so I will begrudgingly carry on 🥲

[rly]

Got it, I’ll change that to be a conversion factor to V - FWIW, I think the current way we store things (raw data and conversion factor, consistent with the NWB standard) is preferable, and I agree with all of Kyu’s points in the spyglass discussion. But I am trying to get better at choosing the path of least resistance re spyglass… so I will begrudgingly carry on 🥲

Sounds good! The ultimate goal of all this is to do science, so yeah sometimes the quick fixes make the most sense.

Have a good trip!

[calderast]

@rly - by converting to uV I also end up changing the datatype from int16 to float32, which is waaaay bigger - what do you think about this? Now that I actually think about it I feel that doubling the size of already big data is not wise ... :(

[rly]

Sorry for my delay in response. If I remember correctly, the Frank Lab data are stored in uV as int16. The few datasets that I have looked at have their raw electricalseries data as int16. I think if you are storing as uV, rounding/truncating will not have a significant impact. Non-zero, but probably insignificant.

[rly]

See also LorenFrankLab/rec_to_nwb#54

[rly]

and this line in the new trodes_to_nwb. https://github.com/LorenFrankLab/trodes_to_nwb/blob/main/src/trodes_to_nwb/convert_ephys.py#L299

[calderast]

Left this PR hanging for a while because I thought we had maybe fixed things on the spyglass end. I am not 100% sure I'm correct anymore on that front, so going to move forward with this change. I think we can safely accept the loss of precision that comes from storing our raw ephys data in uV as int16, as the Frank Lab makes the same choice and the noise level of our recordings makes a sub-uV level of precision unnecessary.

Note that I am making this choice for spyglass compatibility (as ElectricalSeries conversion factors are often ignored when they shouldn't be, as Frank Lab stores all of their data in uV so spyglass assumes this). The NWB standard is to store data in ADC units with conversion factor to V, which I would prefer.