Skip to content

Commit f84df91

Browse files
Victor-C-Zhangmeta-codesync[bot]
authored andcommitted
Make numeric segmenter chunk size and head graph configurable (#603)
Summary: Pull Request resolved: #603 Make the numeric segmenter accept local params for the chunk byte size and head graph, replacing the previously hardcoded values. - Chunk size: read via ZL_Segmenter_getLocalIntParam() using ZL_SEGM_NUMERIC_CHUNK_BYTE_SIZE_MAX_PID. Falls back to 16MB default if not provided. - Head graph: read via ZL_Segmenter_getCustomGraphs(). Falls back to ZL_GRAPH_NUMERIC_COMPRESS if no custom graph is provided. Both changes are backward-compatible — existing callers that don't provide these params get identical behavior to before. Differential Revision: D99729917
1 parent 51100d9 commit f84df91

2 files changed

Lines changed: 19 additions & 7 deletions

File tree

src/openzl/compress/segmenters/segmenter_numeric.c

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#include "openzl/compress/segmenters/segmenter_numeric.h"
44
#include "openzl/common/assertion.h"
55
#include "openzl/compress/private_nodes.h"
6+
#include "openzl/zl_selector.h" // ZL_LP_INVALID_PARAMID ...which is probably not where this constant should be
67

78
ZL_Report SEGM_numeric(ZL_Segmenter* sctx)
89
{
@@ -14,14 +15,23 @@ ZL_Report SEGM_numeric(ZL_Segmenter* sctx)
1415
size_t const width = ZL_Input_eltWidth(input);
1516
ZL_ASSERT(width == 1 || width == 2 || width == 4 || width == 8);
1617

17-
// Note: Currently, static chunk size.
18-
// Tomorrow: global parameter, then local parameter.
19-
size_t const chunkByteSizeMax = 16 << 20;
20-
size_t const chunkEltSizeMax = chunkByteSizeMax / width;
18+
ZL_IntParam const chunkParam = ZL_Segmenter_getLocalIntParam(
19+
sctx, ZL_SEGM_NUMERIC_CHUNK_BYTE_SIZE_MAX_PID);
20+
size_t const chunkByteSizeMax =
21+
(chunkParam.paramId != ZL_LP_INVALID_PARAMID)
22+
? (size_t)chunkParam.paramValue
23+
: (16 << 20) /* default to 16MB */;
24+
ZL_ERR_IF_LT(
25+
chunkByteSizeMax,
26+
width,
27+
nodeParameter_invalid,
28+
"chunk size must produce at least one element");
29+
size_t const chunkEltSizeMax = chunkByteSizeMax / width;
2130

22-
// Note: Currently, static head graph.
23-
// Tomorrow: selectable
24-
ZL_GraphID const headGraph = ZL_GRAPH_NUMERIC_COMPRESS;
31+
ZL_GraphIDList const customGraphs = ZL_Segmenter_getCustomGraphs(sctx);
32+
ZL_GraphID const headGraph = (customGraphs.nbGraphIDs >= 1)
33+
? customGraphs.graphids[0]
34+
: ZL_GRAPH_NUMERIC_COMPRESS;
2535

2636
size_t numElts = ZL_Input_numElts(input);
2737
while (numElts > chunkEltSizeMax) {

src/openzl/compress/segmenters/segmenter_numeric.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ ZL_BEGIN_C_DECLS
1010

1111
ZL_Report SEGM_numeric(ZL_Segmenter* sctx);
1212

13+
#define ZL_SEGM_NUMERIC_CHUNK_BYTE_SIZE_MAX_PID 1
14+
1315
#define SEGM_NUMERIC_DESC \
1416
{ \
1517
.name = "!zl.segmenter_numeric", \

0 commit comments

Comments
 (0)