Skip to content

Commit 7d595ee

Browse files
committed
BUG: Honor compression in the OOC streaming write path
DatasetIO::createEmptyDataset created the dataset with no creation property list, so out-of-core arrays large enough to take the two-step streaming write (createEmptyDataset + hyperslab writes) were always written contiguous and uncompressed, even when WriteOptions requested compression. The single-shot writeSpan path already applied it. * Build the dataset creation property list via BuildChunkedDeflateDcpl in createEmptyDataset, matching writeSpan * Preserves the existing fall-throughs to contiguous storage for compression level 0 and for arrays below the small-array threshold Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
1 parent c04347c commit 7d595ee

1 file changed

Lines changed: 22 additions & 1 deletion

File tree

src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1139,10 +1139,31 @@ Result<> DatasetIO::createEmptyDataset(const DimsType& dims)
11391139
return MakeErrorResult(-1021, "createEmptyDataset error: Unable to create dataspace.");
11401140
}
11411141

1142+
// Build the dataset creation property list so that a compression-requesting
1143+
// write (m_CompressionLevel > 0) produces a chunked + deflated dataset, matching
1144+
// the single-shot writeSpan() path. BuildChunkedDeflateDcpl falls through to
1145+
// H5P_DEFAULT (contiguous) for compression level 0 and for arrays below the
1146+
// small-array threshold. This is what lets the two-step OOC streaming write
1147+
// honor compression: without it the dataset was always created contiguous, so
1148+
// large OOC arrays were written uncompressed even when WriteOptions requested
1149+
// compression.
1150+
auto dcplResult = BuildChunkedDeflateDcpl(dims, sizeof(T), m_CompressionLevel);
1151+
if(dcplResult.invalid())
1152+
{
1153+
H5Sclose(dataspaceId);
1154+
return MakeErrorResult(dcplResult.errors().front().code, fmt::format("createEmptyDataset error: unable to build dataset creation property list for dataset '{}' in file '{}': {}", getNamePath(),
1155+
getFilePath().string(), dcplResult.errors().front().message));
1156+
}
1157+
const hid_t dcpl = dcplResult.value();
1158+
11421159
// Create (or reopen) the dataset. The dataset is left empty; data will
11431160
// be written later via writeSpanHyperslab().
1144-
auto datasetId = createOrOpenDataset<T>(dataspaceId);
1161+
auto datasetId = createOrOpenDataset<T>(dataspaceId, dcpl);
11451162
H5Sclose(dataspaceId);
1163+
if(dcpl != H5P_DEFAULT)
1164+
{
1165+
H5Pclose(dcpl);
1166+
}
11461167
if(datasetId < 0)
11471168
{
11481169
return MakeErrorResult(-1022, "createEmptyDataset error: Unable to create dataset.");

0 commit comments

Comments
 (0)