Skip to content

Commit 3d7312b

Browse files
committed
Update README with pitch, usage, and attribution
1 parent ee0e500 commit 3d7312b

1 file changed

Lines changed: 48 additions & 1 deletion

File tree

README.md

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,56 @@
22

33
Efficient range reservoir sampling from massive sorted key-value datasets. Deterministic, stable, range-friendly, low read amplification. Cython-accelerated hot inner loop (siphash + encoding) with a small Python surface.
44

5-
> Algorithm implementation coming. This is the repo skeleton.
5+
## The problem
6+
7+
You have a huge sorted-by-integer-key dataset, potentially trillions of items on disk. You want `n` uniformly-sampled `(key, value)` pairs from a sub-range `[begin, end)` without pulling the whole thing. Naive every-Nth sampling is biased. Random shuffling on disk kills range access. You want both: uniformity and efficient range reads.
8+
9+
`kh57` gives you:
10+
11+
- **deterministic** - same keys, same salt, same sample.
12+
- **stable** - appending new keys outside the queried range does not change which keys inside get sampled.
13+
- **range-friendly** - scan only what the query needs.
14+
- **low read amplification** - total reads stay within ~2x of `n`.
15+
16+
## How it works
17+
18+
Each key is hashed with SipHash-2-4 (uniform 64-bit output). The `bit_length` of that hash becomes the key's "level": level 63 holds roughly half the keys, level 62 a quarter, and so on down. Every item is stored under a compound sort key `(level << 57) | key` - top 7 bits are the level id, bottom 57 bits are the original key. This preserves original key order within a level, and level order across levels.
19+
20+
To sample from `[begin, end)`: walk levels from sparsest to densest, `range_scan` each level's slice, take full levels while they fit the quota, reservoir-sample the boundary level for the remainder, stop. Each level is a deterministic uniform subset of the range, so the union is a uniform sample.
621

722
## Install
823

924
```bash
1025
pip install kh57
1126
```
1227

28+
Requires Python 3.12+.
29+
30+
## Usage
31+
32+
```python
33+
from kh57 import kh57, sample, MemBackend
34+
35+
backend = MemBackend()
36+
for key in range(1_000_000):
37+
encoded = kh57(key).to_bytes(8, "big")
38+
backend.put(encoded, str(key).encode())
39+
40+
# 500 uniform samples from the [100_000, 200_000) range
41+
result = sample(backend, 500, begin=100_000, end=200_000)
42+
```
43+
44+
Any sorted-by-bytes key-value store can be a backend - just implement the `Backend` protocol (`get`, `put`, `delete`, `range_scan`). `MemBackend` is the reference in-memory adapter; RocksDB / LMDB adapters can be added out-of-tree.
45+
46+
## Public API
47+
48+
- `kh57(key: int) -> int` - encode a 57-bit non-negative key into a 64-bit sort key.
49+
- `recover(h: int) -> tuple[int, int]` - inverse, returns `(level, key)`.
50+
- `uniform_hash(key: int) -> int` - SipHash-2-4 with the default salt.
51+
- `sample(backend, n, begin=None, end=None, *, rng=None) -> list[tuple[int, bytes]]`.
52+
- `Backend` - Protocol.
53+
- `MemBackend` - in-memory reference implementation.
54+
1355
## Dev
1456

1557
```bash
@@ -18,8 +60,13 @@ make build # build cython extensions in-place
1860
make test # run tests
1961
make lint # ruff check
2062
make format # ruff format + fix
63+
make wheels # build manylinux wheels via docker
2164
```
2265

66+
## Credits
67+
68+
The algorithm was designed by Karen Hambardzumyan (mahnerak) in 2023.
69+
2370
## License
2471

2572
MIT

0 commit comments

Comments
 (0)