File created with MatNWB cannot be opened in PyNWB

[esennesh]

Hi folks,

I'm trying to open a file that my lab generated with a pipeline that outputs NWB files. I have confirmed that I can open it with MatNWB. PyNWB, on the other hand, gives seemingly dissonant results. The validation utility says the file is fine:

<frozen runpy>:128: RuntimeWarning: 'pynwb.validate' found in sys.modules after import of package 'pynwb', but prior to execution of 'pynwb.validate'; this may result in unpredictable behaviour
Validating sub-1169714193_ses-1180116198.nwb against cached namespace information using namespace 'ndx-aibs-ecephys'.
Validating sub-1169714193_ses-1180116198.nwb against cached namespace information using namespace 'ndx-aibs-stimulus-template'.
Validating sub-1169714193_ses-1180116198.nwb against cached namespace information using namespace 'ndx-ellipse-eye-tracking'.
 - no errors found.

but the inspector complains about the file:

/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.5.0 because version 1.8.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.4.0 because version 2.6.0-alpha is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.1.0 because version 0.5.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
Traceback (most recent call last):
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1258, in construct
    obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1271, in __new_container__
    obj.__init__(**kwargs)
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/pynwb/ecephys.py", line 94, in __init__
    super().__init__(**kwargs)
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eli/miniforge3/envs/ephys/lib/python3.11/site-packages/pynwb/base.py", line 177, in __init__
    raise ValueError('Specifying rate and timestamps is not supported.')
ValueError: Specifying rate and timestamps is not supported.

If I try to edit the PyNWB source to "dodge" that error, it just keeps finding more. If I couldn't open the file without trouble in Matlab, I would think there's something really wrong with it! Has anyone run into this kind of issue before?

[oruebel]

Based on the error, it looks like the file is indeed not valid, because both timestamps and rate are specified at the same time. I created the following issues NeurodataWithoutBorders/pynwb#1786 on PyNWB to try and allow reading of the file (while still preventing a new file to be created with such an error). Is the file available online or could you share it to make it easier to debug and see if it is feasible to make the file readable despite the errors.

[esennesh]

I can comment out the lines raising the errors in PyNWB or switch them to warnings instead of errors, and eventually it just goes forward to having expected a boolean in place of an int32. If the file is invalid, why does the validator pass it and why can MatNWB read and write it just fine?

(The file was given to our lab by the Allen Institute and should, to my understand, become publicly available by DANDI in March.)

[oruebel]

Would it be possible to share the file with us privately to help with debugging the issue? This would be very helpful since it sounds like there may be multiple issues at play here and it would help to figure out which issues we need to address on our end (in either PyNWB and/or MatNWB) and/or in how the data is being generated.

[esennesh]

Yes, I've asked my PI and I can share the file. I would greatly appreciate the help!

[esennesh]

@oruebel can I send you the file in a Google Drive link? This is concerning me more and more, because it also turns out that I can't even re-export the same file in Matlab after reading it in:

nwb = nwbRead([datadir '/' nwb_name '.nwb'], 'ignorecache');
nwbExport(nwb, [datadir '/' nwb_name '.nwb']);

Error using io.getBaseType
Type int64 is not a supported raw type

Error in io.mapData2H5 (line 14)
tid = io.getBaseType(class(data));

Error in io.writeAttribute (line 3)
[tid, sid, data] = io.mapData2H5(fid, data, varargin{:});

Error in types.core.TimeSeries/export (line 380)
io.writeAttribute(fid, [fullpath '/timestamps/interval'], obj.timestamps_interval);

Error in types.core.SpatialSeries/export (line 154)
refs = export@types.core.TimeSeries(obj, fid, fullpath, refs);

Error in types.ndx_ellipse_eye_tracking.EllipseSeries/export (line 250)
refs = export@types.core.SpatialSeries(obj, fid, fullpath, refs);

Error in types.ndx_ellipse_eye_tracking.EllipseEyeTracking/export (line 69)
refs = obj.corneal_reflection_tracking.export(fid, [fullpath '/corneal_reflection_tracking'], refs);

Error in types.untyped.Set/export (line 181)
refs = v.export(fid, propfp, refs);

Error in types.core.NWBFile/export (line 844)
refs = obj.acquisition.export(fid, [fullpath '/acquisition'], refs);

Error in NwbFile/export (line 62)
refs = export@types.core.NWBFile(obj, output_file_id, '/', {});

Error in nwbExport (line 36)
nwb(i).export(filename);

[oruebel]

Yes, sharing via google drive works. It would be good to also share with @lawrence-mbf and @rly so we can take a look.