80% of measurement requests today use limit: 1, we can optimise for that.
On every request today:
- Filter all probes.
- Shuffle all.
- Group by asn+city.
- Build a list by picking a probe from every group until all groups exhaust.
- Pick the required number of probes from the beginning. Just first probe in most of the cases.
Instead:
On every probes list update:
- Build probe groups by asn+city.
- Sort groups desc by size.
On every request:
- Filter groups by checking its first probe (since probes are the same in one group).
- Pick the required number of probes from groups using random round-robin pick.
I'm oversimplifying here, but the idea looks valid for me.
80% of measurement requests today use
limit: 1, we can optimise for that.On every request today:
Instead:
On every probes list update:
On every request:
I'm oversimplifying here, but the idea looks valid for me.