Is it safe to do BatchLookup + BatchDelete in a loop?

[mscastanho]

I have a situation in which I have a large hash map with entries containing timestamps on each of them. I run a goroutine on userspace that periodically runs a loop to iterate over all entries in the map and delete only those whose timestamp is older than a given threshold.

The code looks something like this:

	var keys [1000]filter.Tuple5
	var vals [1000]uint64

	oldEntries := make([]filter.Tuple5, 0, 100)
	oldestTimestamp := time.Now().Add(-timeout).UnixNano()

	var cursor ebpf.MapBatchCursor
	for {
		cnt, err := m.BatchLookup(&cursor, keys[:], vals[:], nil)
		if err != nil {
			if errors.Is(err, ebpf.ErrKeyNotExist) {
				break
			}
		}

		for i, v := range vals[:cnt] {
			if int64(v) < oldestTimestamp {
				oldEntries = append(oldEntries, keys[i])
			}
		}

		if n, err := m.BatchDelete(oldEntries[:], nil); err != nil {
			slog.Error("failed to delete old entries from map", "error", err.Error())
		} 

		oldEntries = oldEntries[:0]
	}

But I'm wondering: is it safe to delete entries while I'm iterating over the map? Note I can't use BatchLookupAndDelete because I need to selectively delete entries from the map.

I'm observing some hangs in production where this loop sometimes gets stuck inside the call to BatchLookup and I suspect it may be related to the interaction between lookups and deletions in the loop above. Here's a flamegraph captured using pprof during one of these hangs:

[mscastanho]

Turns out the code was wrong (my bad 🙈 ) . The possible error values returned by BatchLookup were not being properly handled, which caused the loop to run forever on an error different than ErrKeyNotExist.

After closer inspection, BatchLookup was returning map batch lookup: no space left on device (batch size too small?), which is thrown by this code:

ebpf/map.go

Lines 1242 to 1246 in c64ffee

	n, sysErr := m.batchLookupCmd(cmd, cursor, count, keysOut, valueBuf.Pointer(), opts)
	if errors.Is(sysErr, unix.ENOSPC) {
	// Hash tables return ENOSPC when the size of the batch is smaller than
	// any bucket.
	return n, fmt.Errorf("%w (batch size too small?)", sysErr)

According to 9d02bd3:

The batch lookup API for hash maps returns ENOSPC when a bucket doesn't
fit into the provided batch. This is unfortunate since user space has no
way to figure out what the largest bucket size is. This makes the API
very awkward to use.

Which looks like what's happening here. I was using a batch of size 1000, but sounds like this was not large enough.
For now I have just increased the batch size to 10k, but I suspect this issue will probably resurface in the future.

Any suggestions on how to avoid this batch vs bucket size mismatch issue?