You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+48-1Lines changed: 48 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,14 +2,56 @@
2
2
3
3
Efficient range reservoir sampling from massive sorted key-value datasets. Deterministic, stable, range-friendly, low read amplification. Cython-accelerated hot inner loop (siphash + encoding) with a small Python surface.
4
4
5
-
> Algorithm implementation coming. This is the repo skeleton.
5
+
## The problem
6
+
7
+
You have a huge sorted-by-integer-key dataset, potentially trillions of items on disk. You want `n` uniformly-sampled `(key, value)` pairs from a sub-range `[begin, end)` without pulling the whole thing. Naive every-Nth sampling is biased. Random shuffling on disk kills range access. You want both: uniformity and efficient range reads.
8
+
9
+
`kh57` gives you:
10
+
11
+
-**deterministic** - same keys, same salt, same sample.
12
+
-**stable** - appending new keys outside the queried range does not change which keys inside get sampled.
13
+
-**range-friendly** - scan only what the query needs.
14
+
-**low read amplification** - total reads stay within ~2x of `n`.
15
+
16
+
## How it works
17
+
18
+
Each key is hashed with SipHash-2-4 (uniform 64-bit output). The `bit_length` of that hash becomes the key's "level": level 63 holds roughly half the keys, level 62 a quarter, and so on down. Every item is stored under a compound sort key `(level << 57) | key` - top 7 bits are the level id, bottom 57 bits are the original key. This preserves original key order within a level, and level order across levels.
19
+
20
+
To sample from `[begin, end)`: walk levels from sparsest to densest, `range_scan` each level's slice, take full levels while they fit the quota, reservoir-sample the boundary level for the remainder, stop. Each level is a deterministic uniform subset of the range, so the union is a uniform sample.
6
21
7
22
## Install
8
23
9
24
```bash
10
25
pip install kh57
11
26
```
12
27
28
+
Requires Python 3.12+.
29
+
30
+
## Usage
31
+
32
+
```python
33
+
from kh57 import kh57, sample, MemBackend
34
+
35
+
backend = MemBackend()
36
+
for key inrange(1_000_000):
37
+
encoded = kh57(key).to_bytes(8, "big")
38
+
backend.put(encoded, str(key).encode())
39
+
40
+
# 500 uniform samples from the [100_000, 200_000) range
41
+
result = sample(backend, 500, begin=100_000, end=200_000)
42
+
```
43
+
44
+
Any sorted-by-bytes key-value store can be a backend - just implement the `Backend` protocol (`get`, `put`, `delete`, `range_scan`). `MemBackend` is the reference in-memory adapter; RocksDB / LMDB adapters can be added out-of-tree.
45
+
46
+
## Public API
47
+
48
+
-`kh57(key: int) -> int` - encode a 57-bit non-negative key into a 64-bit sort key.
0 commit comments