Reduce the space of options to test at least initially

[rly]

The current benchmark suite takes over 24 hours to run, despite each benchmark x test case taking around 30 seconds on average. Some benchmark x test cases take at least several minutes to run, and there are a lot of benchmark x test cases (around 800).

We're testing a lot of benchmarks, e.g., fsspec x {http, s3} x {no cache, with cache, with preloaded cache}. Unless there are interactions, I think we can remove some of these combinations from the full matrix and then run selected tests later once we have a set of candidate best options.

I will let my current full run finish so that we can get a sense of the worst offenders to remove.

I also think we should remove the 5 ranges of file slicing until we have the candidate best options.

[CodyCBakerPhD]

FWIW I never run nwb_benchmarks without --bench on some specific subset

[rly]

All benchmarks have now finished on my end. Some of the LINDI tests failed but I think that's because I accidentally closed my laptop.

The worst offenders:

• HDF5PyNWBFileReadBenchmark.time_read_hdf5_pynwb_fsspec_https_no_cache - IcephysTestCase (7 min)
• HDF5PyNWBFileReadBenchmark.time_read_hdf5_pynwb_fsspec_s3_no_cache - EcephysTestCase (8 min)
• HDF5PyNWBFileReadBenchmark.time_read_hdf5_pynwb_fsspec_s3_with_cache - EcephysTestCase (4 min)
• HDF5PyNWBFsspecS3PreloadedNoCacheFileReadBenchmark.time_read_hdf5_pynwb_fsspec_s3_preloaded_no_cache - EcephysTestCase (4 min), IcephysTestCase (3 min)
• ZarrPyNWBFileReadBenchmark.time_read_zarr_pynwb_s3_force_no_consolidated - IcephysTestCase (15 min)
• LindiLocalJSONContinuousSliceBenchmark.time_slice - OphysTestCaseX (3 to 15 minutes)

Based on this n=1, as expected, in almost all cases, reading a file with pynwb and fsspec using a cache (empty or preloaded) was much faster than without a cache. In general, reading a file with pynwb and fsspec with an https filesystem was faster than or about the same as the S3 filesystem. Caching with remfile doesn't seem to make a difference.

There were some outliers, and I was definitely using my laptop during the ~36 hours to run this, so I'll need to run it again. I'll just run the timing benchmarks now.

Slice tests may be affected by caching (I think remfile tries to read and cache extra data smartly in advance) so we should probably keep testing cache vs no cache there. But we could also go without and just say that from our initial tests, caching is always recommended, so we do not further test the no cache case. We could probably drop reading using fsspec with the S3 filesystem for slice tests.

[oruebel]

Based on this n=1, as expected, in almost all cases, reading a file with pynwb and fsspec using a cache (empty or preloaded) was much faster than without a cache

Will be interesting to see the network traffic results for this. Based no what we saw last year, I'd expect the network traffic to be much higher, indicating that there are a lot of redundant calls. Since caching is not as big a factor for remfile, the offender is likely ffspec itself, rather than PyNWB.

There were some outliers, and I was definitely using my laptop during the ~36 hours to run this, so I'll need to run it again. I'll just run the timing benchmarks now.

We should do a full run on a machine that we don't use. I also have my old M1 MacBook Pro that we could use to make runs.

But we could also go without and just say that from our initial tests, caching is always recommended, so we do not further test the no cache case.

I think being able to quantify how large the impact of caching is on time and network traffic will be useful for the paper. So I would vote to keep both with/without cache options.

• ZarrPyNWBFileReadBenchmark.time_read_zarr_pynwb_s3_force_no_consolidated - IcephysTestCase (15 min)

Will be interesting to the the contrast between with/without consolidated metadata for Zarr.

• LindiLocalJSONContinuousSliceBenchmark.time_slice - OphysTestCaseX (3 to 15 minutes)

I'm curious to see how results differ when running close to the data in S3. I think when we ran in the DANDIHub, there was also an issue with performance for one of the files (not sure if it was OPhys or a different file). Maybe the chunking on OPhys is not good for LINDI?

[CodyCBakerPhD]

We should do a full run on a machine that we don't use. I also have my old M1 MacBook Pro that we could use to make runs.

You could also set it up with a crontab (36 hour repeat?) and just forget about it for a week then check back after

[rly]

OK I'll keep tests as is for now then.

I am running the benchmark suite 5 times in succession on the Mac Pro (trash can mac) at LBNL. Should be done in a week. All tests run now so @stephprince you could start aggregating the results from @CodyCBakerPhD 's result parquet files. @CodyCBakerPhD are those results stored somewhere besides your pythonanywhere server? I don't see https://github.com/neurodatawithoutborders/nwb-benchmarks-database being updated.

[CodyCBakerPhD]

Server isn't really for 'storing'. Sent PR updating docs with link in case you forgot it

I don't see https://github.com/neurodatawithoutborders/nwb-benchmarks-database being updated.

Can be triggered explicitly (not currently set to a CRON) on the server by either curling to the second endpoint at https://codycbakerphd.pythonanywhere.com/ or easiest way is to just use the Swagger button

@stephprince may find that they need to adjust the 'to Parquet' function to get what they want for latest benchmarks

[rly]

Ah, sorry, I see now. I was confused between https://github.com/neurodatawithoutborders/nwb-benchmarks-results and https://github.com/neurodatawithoutborders/nwb-benchmarks-database.

[stephprince]

All tests run now so @stephprince you could start aggregating the results from @CodyCBakerPhD 's result parquet files.

sounds good @rly!

Can be triggered explicitly (not currently set to a CRON) on the server by either curling to the second endpoint at https://codycbakerphd.pythonanywhere.com/ or easiest way is to just use the Swagger button

@stephprince may find that they need to adjust the 'to Parquet' function to get what they want for latest benchmarks

@CodyCBakerPhD thanks for pointing me to the endpoint to update the database. I tried using the Swagger button for data/update-database and it looks like the 'to Parquet' function will need to be updated. I will work on that.