Add blog post on assessing NWB compression options for DANDI Archive #112
Conversation
✅ Deploy Preview for dandi-about ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
- Introduced a comprehensive analysis of HDF5 and Zarr compression options for NWB files. - Provided recommendations based on file size, read speed, and write speed. - Included a detailed testing approach with results and visualizations. - Added example code for testing various compression algorithms and settings.
|
@kabilar can you take a look at this please? |
|
@kabilar ping |
|
|
||
| ### Key Findings: | ||
|
|
||
| 1. **Best Overall Performance: zstd level 4** |
There was a problem hiding this comment.
I might be misunderstanding something, but I don't see zstd level 4 in the plot. Is this referring to zstd (9,) or blosc zstd 4?
There was a problem hiding this comment.
A few suggestions:
- Please add units to the axes.
- The top left plot is missing a y-label.
- Replace
namein the legend withCompression algorithm + level.
There was a problem hiding this comment.
Perhaps file size should be replaced with compression ratio so that it is easier to interpret?
| 2. **Fastest Read Performance: blosc lz4** | ||
| - If optimizing primarily for read speed, blosc lz4 is 1.25x faster than zstd | ||
| - For blosc lz4, level 4 offers a good balance between size and speed |
There was a problem hiding this comment.
I might be misunderstanding, but it looks like blosc lz4hc and zstd (0,) has a faster read speed than blosc lz4.
kabilar
left a comment
kabilar
left a comment
There was a problem hiding this comment.
Thanks @bendichter. This is great. I have provided a few suggestions above.
|
Hi @bendichter, following up here. Thank you. |
…thm analysis and results
|
Hi @kabilar how does this look? Ready to publish? |
|
Hi Ben, thanks for the reminder. This is in my queue, but haven't had a chance to review it yet. |
No description provided.