Skip to content

Commit 3e0653f

Browse files
authored
Optimize bitmask finding some more. (#326)
``` | Before [e26fb39] <main> | After [9753dfea] <optimize-again> | Ratio | Benchmark (Parameter) | |----------------------------|-------------------------------------|---------|------------------------------------------------------------| | 4.10±0.05ms | 3.64±0.04ms | 0.89 | cohorts.ERA5MonthHour.time_find_group_cohorts | | 4.56±0.05ms | 4.00±0.06ms | 0.88 | cohorts.ERA5MonthHourRechunked.time_find_group_cohorts | | 3.31±0.08ms | 2.79±0.01ms | 0.84 | cohorts.ERA5DayOfYear.time_find_group_cohorts | | 8.15±0.05ms | 6.78±0.02ms | 0.83 | cohorts.OISST.time_find_group_cohorts | | 659±10μs | 519±6μs | 0.79 | cohorts.PerfectBlockwiseResampling.time_find_group_cohorts | | 663±20μs | 487±5μs | 0.73 | cohorts.PerfectMonthly.time_find_group_cohorts | | 2.75±0.03ms | 1.93±0.03ms | 0.7 | cohorts.ERA5Google.time_find_group_cohorts | ```
1 parent e26fb39 commit 3e0653f

File tree

1 file changed

+2
-4
lines changed

1 file changed

+2
-4
lines changed

flox/core.py

+2-4
Original file line numberDiff line numberDiff line change
@@ -244,12 +244,11 @@ def _compute_label_chunk_bitmask(labels, chunks, nlabels):
244244

245245
labels = np.broadcast_to(labels, shape[-labels.ndim :])
246246

247-
rows = []
248247
cols = []
249248
# Add one to handle the -1 sentinel value
250249
label_is_present = np.zeros((nlabels + 1,), dtype=bool)
251250
ilabels = np.arange(nlabels)
252-
for idx, region in enumerate(slices_from_chunks(chunks)):
251+
for region in slices_from_chunks(chunks):
253252
# This is a quite fast way to find unique integers, when we know how many there are
254253
# inspired by a similar idea in numpy_groupies for first, last
255254
# instead of explicitly finding uniques, repeatedly write True to the same location
@@ -259,10 +258,9 @@ def _compute_label_chunk_bitmask(labels, chunks, nlabels):
259258
# skip the -1 sentinel by slicing
260259
# Faster than np.argwhere by a lot
261260
uniques = ilabels[label_is_present[:-1]]
262-
rows.append(np.full_like(uniques, idx))
263261
cols.append(uniques)
264262
label_is_present[:] = False
265-
rows_array = np.concatenate(rows)
263+
rows_array = np.repeat(np.arange(nchunks), tuple(len(col) for col in cols))
266264
cols_array = np.concatenate(cols)
267265
data = np.broadcast_to(np.array(1, dtype=np.uint8), rows_array.shape)
268266
bitmask = csc_array((data, (rows_array, cols_array)), dtype=bool, shape=(nchunks, nlabels))

0 commit comments

Comments
 (0)