Fix BDV N5 setup over-allocation#1207
Conversation
There was a problem hiding this comment.
Pull request overview
Adjusts BDV N5/HDF5 writer behavior to prevent creating an extra (unwritten) “position worth” of setup groups at the end of a completed acquisition by changing when dynamic setup expansion occurs in BigDataViewerDataSource.write().
Changes:
- Move dynamic setup expansion to occur immediately before a write that targets a new position.
- Remove the post-write expansion logic that ran after incrementing
_current_frame.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| self.shape_c * (pos + 1), | ||
| create_flag=False, | ||
| ) | ||
| self.positions = pos + 1 |
There was a problem hiding this comment.
When pos >= self.positions, this block updates self.positions before computing ds_name. Since _h5_ds_name()/_n5_ds_name() derive the setup id from c * self.positions + p, increasing self.positions mid-acquisition changes the mapping for all channels/positions and can cause writes for the new position to target an existing setup (overwriting previous data) rather than the newly created setups. The setup allocation here (using shape_c * old_positions .. shape_c * (pos+1)) also implies a different linearization than c * positions + p, so dynamic position growth is inconsistent. Consider using a setup-id mapping that does not change when positions grows (or update both dataset naming and metadata generation to a consistent, append-friendly ordering) so new positions never collide with existing setups.
| self.positions = pos + 1 |
Motivation
setupgroups at the end of a complete acquisition because setup expansion was performed after advancing the frame index.Description
BigDataViewerDataSource.write()to happen prior to writing when a write targets a new position (if pos >= self.positions: ...)._current_frame, preventing creation of an extra empty position.src/navigate/model/data_sources/bdv_data_source.pyand preserves on-demand growth of positions only when a real write targets them.Testing
zarrimage contained onlysetup0andsetup1andds.positions == 1(assertions passed).pytestBDV tests but full test runs were blocked in this environment by missing optional test/runtime dependencies (pytest-covaddopts,tifffileAPI mismatch forimsave, andlibGLrequired bycv2).Codex Task