"bins must be monotonically increasing or decreasing" error

[gshvarts]

Running into ValueError: bins must be monotonically increasing or decreasing when trying to create nwbs from trodes files.

Working on getting the most recent error stack but here's an error stack from running into the same error earlier this year to start with:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], [line 6](vscode-notebook-cell:?execution_count=2&line=6)
      [3](vscode-notebook-cell:?execution_count=2&line=3) # restrict conversion to a single session, ignores the reconfig metadata in the download
      [4](vscode-notebook-cell:?execution_count=2&line=4) query_expression = f"animal == 'herman' and date == 20211115"
----> [6](vscode-notebook-cell:?execution_count=2&line=6) create_nwbs(
      [7](vscode-notebook-cell:?execution_count=2&line=7)     path=data_path,
      [8](vscode-notebook-cell:?execution_count=2&line=8)     probe_metadata_paths=None,
      [9](vscode-notebook-cell:?execution_count=2&line=9)     output_dir="[/home/gl-willow/banyan/nwb/raw/](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/nwb/raw/)",  # saving to a temporary directory to not clutter up the database.  Please delete when done.
     [10](vscode-notebook-cell:?execution_count=2&line=10)     query_expression=query_expression
     [11](vscode-notebook-cell:?execution_count=2&line=11) )

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185), in create_nwbs(path, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, n_workers, query_expression, disable_ptp)
    [183](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:183) else:
    [184](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:184)     for session, session_df in file_info.groupby(["date", "animal"]):
--> [185](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185)         _create_nwb(
    [186](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:186)             session,
    [187](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:187)             session_df,
    [188](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:188)             header_reconfig_path,
    [189](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:189)             probe_metadata_paths,
    [190](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:190)             output_dir,
    [191](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:191)             video_directory,
    [192](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:192)             convert_video,
    [193](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:193)             disable_ptp,
    [194](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:194)         )

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:305](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:305), in _create_nwb(session, session_df, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, disable_ptp)
    [299](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:299)     add_position(
    [300](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:300)         nwb_file,
    [301](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:301)         metadata,
    [302](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:302)         session_df,
    [303](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:303)     )
    [304](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:304) else:
--> [305](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:305)     add_position(
    [306](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:306)         nwb_file,
    [307](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:307)         metadata,
    [308](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:308)         session_df,
    [309](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:309)         ptp_enabled=ptp_enabled,
    [310](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:310)         rec_dci_timestamps=rec_dci_timestamps,
    [311](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:311)         sample_count=nwb_file.processing["sample_count"]
    [312](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:312)         .data_interfaces["sample_count"]
    [313](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:313)         .data,
    [314](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:314)     )
    [316](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:316) # write file
    [317](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:317) output_path = Path(f"{output_dir}[/](https://file+.vscode-resource.vscode-cdn.net/){session[1]}{session[0]}.nwb")

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:836](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:836), in add_position(nwb_file, metadata, session_df, ptp_enabled, rec_dci_timestamps, sample_count)
    [833](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:833) else:
    [834](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:834)     dio_camera_timestamps_epoch = None
--> [836](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:836) position_df = get_position_timestamps(
    [837](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:837)     position_timestamps_filepath,
    [838](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:838)     position_tracking_filepath,
    [839](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:839)     ptp_enabled=ptp_enabled,
    [840](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:840)     rec_dci_timestamps=rec_dci_timestamps,
    [841](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:841)     dio_camera_timestamps=dio_camera_timestamps_epoch,
    [842](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:842)     sample_count=sample_count,
    [843](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:843)     epoch_interval=[epoch_start, epoch_end] if not ptp_enabled else None,
    [844](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:844) )
    [846](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:846) # TODO: Doesn't handle multiple cameras currently
    [847](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:847) camera_id = epoch_to_camera_ids.loc[epoch].camera_id[0]

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:624](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:624), in get_position_timestamps(position_timestamps_filepath, position_tracking_filepath, rec_dci_timestamps, dio_camera_timestamps, sample_count, ptp_enabled, epoch_interval)
    [620](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:620) frame_count = np.asarray(video_timestamps.HWframeCount)
    [622](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:622) is_valid_camera_time = np.isin(video_timestamps.index, sample_count)
--> [624](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:624) epoch_start_ind = np.digitize(epoch_interval[0], rec_dci_timestamps)
    [625](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:625) epoch_end_ind = np.digitize(epoch_interval[1], rec_dci_timestamps)
    [627](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:627) camera_systime = rec_dci_timestamps[
    [628](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:628)     wrapped_digitize(
    [629](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:629)         video_timestamps.index[is_valid_camera_time],
ref='~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:0'>0</a>;32m   (...)
    [632](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:632)     + epoch_start_ind
    [633](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:633) ]

File [~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5724](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5724), in digitize(x, bins, right)
   [5722](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5722) mono = _monotonicity(bins)
   [5723](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5723) if mono == 0:
-> [5724](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5724)     raise ValueError("bins must be monotonically increasing or decreasing")
   [5726](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5726) # this is backwards because the arguments below are swapped
   [5727](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5727) side = 'left' if right else 'right'

ValueError: bins must be monotonically increasing or decreasing

Here's the output from while the create_nwbs function was running as well, in case that's helpful at all:

22-Jan-24 16:58:18 Creating NWB file for session: (20211115, 'herman')
22-Jan-24 16:58:18 	rec_filepaths: ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec', '/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec', '/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec', '/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec', '/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec']
22-Jan-24 16:58:18 CREATING REC DATA ITERATORS
22-Jan-24 16:58:18 Parsing headers
22-Jan-24 16:58:19 Parsing header COMPLETE
22-Jan-24 17:27:53 # iterators: 14
22-Jan-24 17:50:04 Reading timestamps COMPLETE
22-Jan-24 17:50:07 	metadata_filepath: [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_metadata.yml](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_metadata.yml)
22-Jan-24 17:50:08 CREATING HARDWARE MAPS
22-Jan-24 17:50:08 CREATING METADATA ENTRIES
[/home/gl-willow/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/pynwb/file.py:471](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/pynwb/file.py:471): UserWarning: Date is missing timezone information. Updating to local timezone.
  args_to_set['session_start_time'] = _add_missing_timezone(session_start_time)
22-Jan-24 17:50:08 ERROR: associated file [/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_01_s1.stateScriptLog](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_01_s1.stateScriptLog) does not exist
22-Jan-24 17:50:08 [Errno 2] No such file or directory: '/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_01_s1.stateScriptLog'
22-Jan-24 17:50:08 ERROR: associated file [/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_02_r1.stateScriptLog](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_02_r1.stateScriptLog) does not exist
22-Jan-24 17:50:08 [Errno 2] No such file or directory: '/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_02_r1.stateScriptLog'
22-Jan-24 17:50:08 ERROR: associated file [/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_03_s2.stateScriptLog](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_03_s2.stateScriptLog) does not exist
22-Jan-24 17:50:08 [Errno 2] No such file or directory: '/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_03_s2.stateScriptLog'
22-Jan-24 17:50:08 ERROR: associated file [/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_04_r2.stateScriptLog](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_04_r2.stateScriptLog) does not exist
22-Jan-24 17:50:08 [Errno 2] No such file or directory: '/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_04_r2.stateScriptLog'
22-Jan-24 17:50:08 ERROR: associated file [/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_05_s3.stateScriptLog](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_05_s3.stateScriptLog) does not exist
22-Jan-24 17:50:08 [Errno 2] No such file or directory: '/home/gl-willow/banyan/raw/anna/herman/raw/20211115/20211115_herman_05_s3.stateScriptLog'
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
cp: cannot create regular file '/20211115_herman_01_s1.1.h264': Permission denied
22-Jan-24 17:50:09 Video copy completed. [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.h264](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.h264) has been copied to [/20211115_herman_01_s1.1.h264](https://file+.vscode-resource.vscode-cdn.net/20211115_herman_01_s1.1.h264)
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
cp: cannot create regular file '/20211115_herman_02_r1.1.h264': Permission denied
22-Jan-24 17:50:09 Video copy completed. [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.1.h264](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.1.h264) has been copied to [/20211115_herman_02_r1.1.h264](https://file+.vscode-resource.vscode-cdn.net/20211115_herman_02_r1.1.h264)
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
cp: cannot create regular file '/20211115_herman_03_s2.1.h264': Permission denied
22-Jan-24 17:50:09 Video copy completed. [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.1.h264](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.1.h264) has been copied to [/20211115_herman_03_s2.1.h264](https://file+.vscode-resource.vscode-cdn.net/20211115_herman_03_s2.1.h264)
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
cp: cannot create regular file '/20211115_herman_04_r2.1.h264': Permission denied
22-Jan-24 17:50:09 Video copy completed. [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.1.h264](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.1.h264) has been copied to [/20211115_herman_04_r2.1.h264](https://file+.vscode-resource.vscode-cdn.net/20211115_herman_04_r2.1.h264)
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
cp: cannot create regular file '/20211115_herman_05_s3.1.h264': Permission denied
22-Jan-24 17:50:09 Video copy completed. [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.1.h264](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.1.h264) has been copied to [/20211115_herman_05_s3.1.h264](https://file+.vscode-resource.vscode-cdn.net/20211115_herman_05_s3.1.h264)
22-Jan-24 17:50:09 ADDING EPHYS DATA
22-Jan-24 17:50:09 Parsing headers
22-Jan-24 17:50:09 Parsing header COMPLETE
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec)
22-Jan-24 18:12:43 # iterators: 14
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec)
Interpolate memmap:  [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec)
22-Jan-24 18:32:35 Reading timestamps COMPLETE
22-Jan-24 18:32:36 ADDING DIO DATA
22-Jan-24 23:24:57 ADDING ANALOG DATA
22-Jan-24 23:24:57 Parsing headers
22-Jan-24 23:24:57 Parsing header COMPLETE
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec)
compute multiplex cache [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec)
22-Jan-24 23:33:22 # iterators: 14
22-Jan-24 23:33:22 Reading timestamps COMPLETE
22-Jan-24 23:33:24 ADDING SAMPLE COUNTS
22-Jan-24 23:43:33 ADDING EPOCHS
22-Jan-24 23:43:33 ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.rec']
22-Jan-24 23:43:33 ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_02_r1.rec']
22-Jan-24 23:43:33 ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_03_s2.rec']
22-Jan-24 23:43:33 ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_04_r2.rec']
22-Jan-24 23:43:33 ['/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_05_s3.rec']
22-Jan-24 23:43:33 ADDING POSITION
22-Jan-24 23:43:33 PTP enabled: False
22-Jan-24 23:43:34 1
22-Jan-24 23:43:34 	position_timestamps_filepath: [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.videoTimeStamps.cameraHWSync](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.videoTimeStamps.cameraHWSync)
22-Jan-24 23:43:34 	position_tracking_filepath: [/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.videoPositionTracking](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/raw/anna/herman/20211115/20211115_herman_01_s1.1.videoPositionTracking)
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
22-Jan-24 23:43:34 repeat timestamps ind: []
22-Jan-24 23:43:34 big frame jumps: []
22-Jan-24 23:43:34 	non_repeat_timestamp_labels = [1]
[/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127): FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) [/](https://file+.vscode-resource.vscode-cdn.net/) '(1,)type'.
  return np.dtype(typearr)
22-Jan-24 23:43:34 Camera frame rate estimated from DIO camera ticks: 59523646659.0 frames/s

From our initial poking around, it looks like it might be an issue with rec_dci_timestamps not being monotonically increasing, but not sure why that might happen or how we could address that issue without having to scrap the epoch.

Thanks for your help!

[samuelbray32]

Ok, so assuming it's similar to what's been seen before I'm guessing it's caused by overflow of the int32 value of trodes timestamps described in #59

I would think that would be fixed by #61 and #80

Is this coming from a very long epoch? I could see errors being caused if the trodes timestamps are not unique within a epoch, but that would require the number of samples to be larger than 4294967295 (e.g. ~40 hours at 30kHz sampling rate)

[samuelbray32]

If you can enter debug mode and find out more about sample_count[epoch_start_ind:epoch_end_ind] that would be helpful. particularly, I'm looking for a case of multiple wraps and that array's size:

val = sample_count[epoch_start_ind:epoch_end_ind]
print(val.shape)
delta = np.diff(val)
print(delta[delta<=0])

[gshvarts]

Update is that we realized we were still running our trodes_to_nwb on a previous branch where this non-ptp epoch restriction was still being tested out (origin/non_ptp_epoch_restriction)! We switched back to main and pulled in all the recent updates and now we're getting an earlier error that might have to do with some updates that have been made in the past couple months. Do you have any idea what this might be about?

This is the output:

12-Dec-24 13:03:46 Creating NWB file for session: (20211218, 'chip')
12-Dec-24 13:03:46 	rec_filepaths: ['/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_01_s1.rec', '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_01_s1.rec', '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_02_r1.rec', '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_02_r1.rec', '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_03_s2.rec', '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_03_s2.rec', '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_04_r2.rec', '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_04_r2.rec', '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_05_s3.rec', '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_05_s3.rec']
12-Dec-24 13:03:46 CREATING REC DATA ITERATORS
12-Dec-24 13:03:46 Parsing headers

And here's the error stack:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb) Cell 6 line 8
      [5](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=4) # restrict conversion to a single session, ignores the reconfig metadata in the download
      [6](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=5) query_expression = f"animal == 'chip' and date == 20211218"
----> [8](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=7) create_nwbs(
      [9](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=8)     path=data_path,
     [10](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=9)     probe_metadata_paths=None,
     [11](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=10)     output_dir="[/home/gl-willow/banyan/nwb/raw/](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/banyan/nwb/raw/)",  # saving to a temporary directory to not clutter up the database.  Please delete when done.
     [12](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=11)     query_expression=query_expression
     [13](vscode-notebook-cell:/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/conversion_tutorial.ipynb#X16sZmlsZQ%3D%3D?line=12) )

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185), in create_nwbs(path, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, n_workers, query_expression, disable_ptp)
    [183](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:183) else:
    [184](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:184)     for session, session_df in file_info.groupby(["date", "animal"]):
--> [185](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185)         _create_nwb(
    [186](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:186)             session,
    [187](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:187)             session_df,
    [188](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:188)             header_reconfig_path,
    [189](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:189)             probe_metadata_paths,
    [190](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:190)             output_dir,
    [191](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:191)             video_directory,
    [192](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:192)             convert_video,
    [193](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:193)             disable_ptp,
    [194](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:194)         )

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:216](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:216), in _create_nwb(session, session_df, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, disable_ptp)
    [214](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:214) logger.info("CREATING REC DATA ITERATORS")
    [215](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:215) # make generic rec file data chunk iterator to pass to functions
--> [216](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:216) rec_dci = RecFileDataChunkIterator(
    [217](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:217)     rec_filepaths, interpolate_dropped_packets=False, stream_id="trodes"
    [218](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:218) )
    [219](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:219) rec_dci_timestamps = (
    [220](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:220)     rec_dci.timestamps
    [221](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:221) )  # pass these when creating other non-interpolated rec iterators to save time
    [223](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:223) rec_header = read_header(rec_filepaths[0])

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72), in RecFileDataChunkIterator.__init__(self, rec_file_path, nwb_hw_channel_order, conversion, stream_index, stream_id, is_analog, interpolate_dropped_packets, timestamps, **kwargs)
     [65](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:65) self.neo_io = [
     [66](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:66)     SpikeGadgetsRawIO(
     [67](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:67)         filename=file, interpolate_dropped_packets=interpolate_dropped_packets
     [68](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:68)     )
     [69](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:69)     for file in rec_file_path
     [70](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:70) ]  # get all streams for all files
     [71](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:71) logger.info("Parsing headers")
---> [72](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72) [neo_io.parse_header() for neo_io in self.neo_io]
     [73](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:73) # TODO see what else spikeinterface does and whether it is necessary
     [74](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:74) logger.info("Parsing header COMPLETE")

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72), in <listcomp>(.0)
     [65](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:65) self.neo_io = [
     [66](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:66)     SpikeGadgetsRawIO(
     [67](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:67)         filename=file, interpolate_dropped_packets=interpolate_dropped_packets
     [68](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:68)     )
     [69](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:69)     for file in rec_file_path
     [70](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:70) ]  # get all streams for all files
     [71](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:71) logger.info("Parsing headers")
---> [72](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:72) [neo_io.parse_header() for neo_io in self.neo_io]
     [73](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:73) # TODO see what else spikeinterface does and whether it is necessary
     [74](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_ephys.py:74) logger.info("Parsing header COMPLETE")

File [~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:179](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:179), in BaseRawIO.parse_header(self)
    [166](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:166) def parse_header(self):
    [167](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:167)     """
    [168](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:168)     This must parse the file header to get all stuff for fast use later on.
    [169](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:169) 
ref='~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:0'>0</a>;32m   (...)
    [177](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:177) 
    [178](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:178)     """
--> [179](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:179)     self._parse_header()
    [180](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/neo/rawio/baserawio.py:180)     self._check_stream_signal_channel_characteristics()

File [~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:359](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:359), in SpikeGadgetsRawIO._parse_header(self)
    [356](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:356)     self._mask_channels_bytes[stream_id] = mask
    [357](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:357)     self._mask_streams[stream_id] = np.any(mask, axis=0)
--> [359](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:359) signal_streams = np.array(signal_streams, dtype=_signal_stream_dtype)
    [360](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:360) signal_channels = np.array(signal_channels, dtype=_signal_channel_dtype)
    [362](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/Documents/anna/trodes_to_nwb/notebooks/~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/spike_gadgets_raw_io.py:362) # remove some stream if no wanted

ValueError: could not assign tuple of length 3 to structure with 2 fields.

When we ran %debug, this was the value of signal_streams:
[('Controller_DIO_digital', 'Controller_DIO_digital', ''), ('ECU_digital', 'ECU_digital', ''), ('ECU_analog', 'ECU_analog', ''), ('trodes', 'trodes', '')]

Do you have any insight into this error? Would be helpful before we can check if the original monotonically increasing bins error is still an issue. Thanks!!

[samuelbray32]

#99

The code was updated for new neo_io release. Update to version

neo>=0.13.4

[gshvarts]

Anna, here, on gabby's login: we've gotten past the neo error but now have a new digitize error; stack pasted below. One thing that clearly sticks out is that the framerate has been estimated to be wildly high; something must be up with our camsync dio and i will investigate that, but i think that might be separate from the timestamps not being monotonically increasing/decreasing. thoughts?

18-Dec-24 19:32:21 PTP enabled: False
18-Dec-24 19:32:21 1
18-Dec-24 19:32:21 position_timestamps_filepath: /home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_01_s1.1.videoTimeStamps.cameraHWSync
18-Dec-24 19:32:21 position_tracking_filepath: /home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_01_s1.1.videoPositionTracking
/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
return np.dtype(typearr)
18-Dec-24 19:32:21 repeat timestamps ind: []
18-Dec-24 19:32:21 big frame jumps: []
18-Dec-24 19:32:21 non_repeat_timestamp_labels = [1]
/home/gl-willow/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:127: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
return np.dtype(typearr)
18-Dec-24 19:32:21 Camera frame rate estimated from DIO camera ticks: 937274636871.5 frames/s

---

ValueError Traceback (most recent call last)
Cell In[2], line 8
5 # restrict conversion to a single session, ignores the reconfig metadata in the download
6 query_expression = f"animal == 'chip' and date == 20211218"
----> 8 create_nwbs(
9 path=data_path,
10 probe_metadata_paths=None,
11 output_dir="/home/gl-willow/banyan/nwb/raw/", # saving to a temporary directory to not clutter up the database. Please delete when done.
12 query_expression=query_expression
13 )

File ~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:185, in create_nwbs(path, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, n_workers, query_expression, disable_ptp)
183 else:
184 for session, session_df in file_info.groupby(["date", "animal"]):
--> 185 _create_nwb(
186 session,
187 session_df,
188 header_reconfig_path,
189 probe_metadata_paths,
190 output_dir,
191 video_directory,
192 convert_video,
193 disable_ptp,
194 )

File ~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert.py:307, in _create_nwb(session, session_df, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, disable_ptp)
301 add_position(
302 nwb_file,
303 metadata,
304 session_df,
305 )
306 else:
--> 307 add_position(
308 nwb_file,
309 metadata,
310 session_df,
311 ptp_enabled=ptp_enabled,
312 rec_dci_timestamps=rec_dci_timestamps,
313 sample_count=nwb_file.processing["sample_count"]
314 .data_interfaces["sample_count"]
315 .data,
316 )
318 # write file
319 output_path = Path(f"{output_dir}/{session[1]}{session[0]}.nwb")

File ~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:837, in add_position(nwb_file, metadata, session_df, ptp_enabled, rec_dci_timestamps, sample_count)
834 else:
835 dio_camera_timestamps_epoch = None
--> 837 position_df = get_position_timestamps(
838 position_timestamps_filepath,
839 position_tracking_filepath,
840 ptp_enabled=ptp_enabled,
841 rec_dci_timestamps=rec_dci_timestamps,
842 dio_camera_timestamps=dio_camera_timestamps_epoch,
843 sample_count=sample_count,
844 epoch_interval=[epoch_start, epoch_end] if not ptp_enabled else None,
845 )
847 # TODO: Doesn't handle multiple cameras currently
848 camera_id = epoch_to_camera_ids.loc[epoch].camera_id[0]

File ~/Documents/anna/trodes_to_nwb/src/trodes_to_nwb/convert_position.py:622, in get_position_timestamps(position_timestamps_filepath, position_tracking_filepath, rec_dci_timestamps, dio_camera_timestamps, sample_count, ptp_enabled, epoch_interval)
618 pause_mid_time = -1
620 frame_count = np.asarray(video_timestamps.HWframeCount)
--> 622 epoch_start_ind = np.digitize(epoch_interval[0], rec_dci_timestamps)
623 epoch_end_ind = np.digitize(epoch_interval[1], rec_dci_timestamps)
624 is_valid_camera_time = np.isin(
625 video_timestamps.index, sample_count[epoch_start_ind:epoch_end_ind]
626 )

File ~/mambaforge/envs/trodes_to_nwb/lib/python3.10/site-packages/numpy/lib/function_base.py:5724, in digitize(x, bins, right)
5722 mono = _monotonicity(bins)
5723 if mono == 0:
-> 5724 raise ValueError("bins must be monotonically increasing or decreasing")
5726 # this is backwards because the arguments below are swapped
5727 side = 'left' if right else 'right'

ValueError: bins must be monotonically increasing or decreasing

[gshvarts]

a couple potentially important things to add:

1. turns out our framerate estimates are wildly high for ALL non-ptp data, including the many days of data that have been converted to NWB successfully. so, that was a red herring for this issue but also potentially reveals an issue will many additional days of data! does this framerate value get stored in the NWB file, or is it just printed out as a sanity check?
2. i poked around in debug mode a bit and found out that in this epoch, there are 5 instances of negative differences between rec_dci_timestamps that are causing this error. they seem to correspond to points at the end of each epoch (between each epoch?) where the time jumps back 2-4sec and then continues incrementing normally. perhaps that is useful info to have?

[samuelbray32]

I'm not seeing the warning "Timestamps are not sequential. This may cause problems with some software or data analysis." in your output, which suggests that the timestamps within each epoch are ordered and the error is between adjacent epochs. Just curious, was the end/start of epoch files very close together during data collection?

[edeno]

Frame rate is just printed out as a sanity check FYI

[samuelbray32]

Following up eric's comment

• the big number printed out comes from not converting correctly between seconds and nanoseconds before printout. Noting to correct in future

[edeno]

Not sure what's causing the time points to go back. I suppose there could be a gap in timestamps and then the linear fit is not starting in the right place for some reason.

[gshvarts]

ok thanks for the info about the framerate, that's comforting!

agreed that since we don't see the "Timestamps are not sequential" printout, this must be coming from between the epochs, but I'm still mystified how one epoch could appear to start before the previous one ended. i don't think this day had particularly short changeover times and i don't think that would be it since it seems to be something that happens between every epoch and because we're getting this errror for quite a few days of data for this animal (but not all!)

[samuelbray32]

I think the simplest next step is to look into the timestamps data coming from your rec files and see if anything looks strange. Here's a short script to access that data

from trodes_to_nwb.convert_ephys import RecFileDataChunkIterator
import numpy as np
import matplotlib.pyplot as plt

path_list = ["/cumulus/amankili/Banner/raw/20220224/20220224_Banner_01_sleepBan77mWnostim.rec",
             "/cumulus/amankili/Banner/raw/20220224/20220224_Banner_02_wtrackBan77mWlockout80mstheta90.rec"
             ] # list of rec files for the epochs you want to check

rec_dci = RecFileDataChunkIterator(path_list,stream_id="trodes")
# get the regressed systime and sysclock for the combined epochs
regressed_systime = [] # the timestamps after linear regression
sysclock = [] # the raw timestamps from the system clock
for neo_io in rec_dci.neo_io:
    regressed_systime.extend(neo_io.get_regressed_systime(0,None))
    sysclock.extend(neo_io.get_sys_clock(0,None) / 1e9 ) # convert to seconds
regressed_systime = np.array(regressed_systime)
sysclock = np.array(sysclock)
rec_dci_timestamps = rec_dci.timestamps

# looking at the data
ind = slice(0,10000) # plot the first 10000 timestamps (change this to plot more or less)
fig1 = plt.figure()
plt.plot(regressed_systime[ind], label = "regressed systime")
plt.plot(rec_dci_timestamps[ind], label = "rec_dci timestamps")
plt.plot(sysclock[ind], label = "sysclock")
plt.xlabel("index")
plt.ylabel("time (s)")
plt.legend()

# looking at the residuals between the original and regressed systime
fig2 = plt.figure()
plt.plot(sysclock[ind] - regressed_systime[ind], label="sysclock - regressed systime")
plt.legend()

[samuelbray32]

I would have thought that this check would have caught non-monotonic timestamps errors before the point you reached, so curious to see the results

[gshvarts]

Ok update on this! It was an issue with our file structure for a couple of our sessions and it's no longer coming up after fixing that!

Still sharing the issue we were running into, just in case this is something that's worth adding a check for:

I noticed while debugging that the rec_filepaths variable was pulling in duplicate copies of each epoch from different folders:

['/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_01_s1.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_01_s1.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_02_r1.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_02_r1.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_03_s2.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_03_s2.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_04_r2.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_04_r2.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211218/20211218_chip_05_s3.rec',
 '/home/gl-willow/banyan/raw/anna/chip/20211229/20211218/20211218_chip_05_s3.rec']

Turns out we mistakenly had a copy of several dates of data within the 20211229 folder (probably leftover from some big copying over of data) that were getting flagged by the _get_file_paths function. I think that resulted in the duplicated epoch appearing to 'hop back in time' in terms of the timestamps and resulting in this monotonically increasing or decreasing error.

We also ran into a similar issue with another date where we had a folder named 20210405_bad_eps in addition to 20210405 that was still getting pulled from, and the renamed epoch numbering for the good epochs were getting all jumbled with the bad epochs to result in a series that wasn't chronologically sound.

I think this is mostly an issue with us not being aware of how strictly we should follow the file structure conventions. However, if you anticipate that this could be an issue for others in the future too and there's an easy way to make the _get_file_paths function pull a more specific list or throw a warning if it's generating .rec files from different folders or something, maybe that could be an extra precautionary step against issues like this in the future.

Thanks for your help in narrowing down the issue!

[samuelbray32]

Yup that would definitely do it. When we compile the files to include, we trace down all subfolders in the path given (to try to be less dependent on the user's organization) and include all matching file types. Can definitely think about a cheap place to check for repeated epochs and raise an error. Glad you found it!