Commit 1e7a9a9
mongo: partition snapshots for numeric and string _id collections (#4487)
## Summary
Extends the MongoDB connector so initial snapshots **parallelize for
numeric and string `_id` collections**, not just ObjectID. Previously
any non-ObjectID `_id` fell back to a single serial full-table
partition, making large naturally-keyed collections (app-store numeric
IDs, package-name strings) too slow to snapshot inside the oplog
retention window.
Based on `v0.36.29` (our deployed release).
It improves initial sync (MongoDb -> Clickhouse from 10 hours to 3 hours
on 8mln collection).
## Approach
`GetQRepPartitions` dispatches on the collection's min/max `_id` BSON
type:
- **ObjectID** — unchanged (uniform division of the 12-byte keyspace).
- **Int32/Int64** — uniform min/max division, reusing
`utils.PartitionHelper` to emit non-overlapping `IntPartitionRange`
partitions (`$gte`/`$lte`).
- **String** — sampling-based quantile boundaries via `$sample` (honours
read preference), anchored by the real min/max, emitted as a new
`StringPartitionRange` proto type with contiguous half-open ranges
(`$gte`/`$lt`).
- **Mixed-type / Double / empty** — safe full-table fallback.
## Changes
- `protos/flow.proto` — add `StringPartitionRange` + `string_range`
oneof field.
- `connectors/mongo/qrep_partition.go` — type dispatcher; numeric +
string builders; pure, unit-tested `computeStringBoundaries`.
- `connectors/mongo/qrep.go` — `IntRange`/`StringRange` cases in
`toRangeFilter`.
- `connectors/utils/monitoring/monitoring.go` — handle `StringRange` in
`addPartitionToQRepRun` (avoids the `unknown range type` error path).
## Testing
- `go build ./...`, `go vet` clean.
- New unit tests for `computeStringBoundaries`
(even/clustered/sparse/duplicate/out-of-range/empty + contiguity),
`toRangeFilter` (Int/String/ObjectID/unsupported), and `_id` type
detection — all passing.
## Notes / limitations
- Numeric uses uniform value division; monitor partition balance on
first run for sparse ID spaces.
- Double `_id` and mixed-type collections fall back to full-table by
design.
- Requires regenerating protobuf bindings (`./generate-protos.sh`)
before building images, as `flow/generated` is gitignored.
---------
Co-authored-by: Ilia Demianenko <ilia.demianenko@clickhouse.com>1 parent 4ee56fc commit 1e7a9a9
4 files changed
Lines changed: 567 additions & 87 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | 29 | | |
38 | 30 | | |
39 | | - | |
| 31 | + | |
40 | 32 | | |
41 | 33 | | |
42 | 34 | | |
| |||
70 | 62 | | |
71 | 63 | | |
72 | 64 | | |
73 | | - | |
| 65 | + | |
74 | 66 | | |
75 | 67 | | |
76 | | - | |
| 68 | + | |
77 | 69 | | |
78 | 70 | | |
79 | 71 | | |
| |||
234 | 226 | | |
235 | 227 | | |
236 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
237 | 252 | | |
238 | 253 | | |
239 | 254 | | |
| |||
0 commit comments