mongo: partition snapshots for numeric and string _id collections by andreyzhelnin-st · Pull Request #1 · andreyzhelnin-st/peerdb

andreyzhelnin-st · 2026-06-23T16:12:20Z

Summary

Extends the MongoDB connector so initial snapshots parallelize for numeric and string _id collections, not just ObjectID. Previously any non-ObjectID _id fell back to a single serial full-table partition, making large naturally-keyed collections (app-store numeric IDs, package-name strings) too slow to snapshot inside the oplog retention window.

Based on v0.36.29 (our deployed release).

It improves initial sync (MongoDb -> Clickhouse from 10 hours to 3 hours on 8mln collection)

Approach

GetQRepPartitions dispatches on the collection's min/max _id BSON type:

ObjectID — unchanged (uniform division of the 12-byte keyspace).
Int32/Int64 — uniform min/max division, reusing utils.PartitionHelper to emit non-overlapping IntPartitionRange partitions ($gte/$lte).
String — sampling-based quantile boundaries via $sample (honours read preference), anchored by the real min/max, emitted as a new StringPartitionRange proto type with contiguous half-open ranges ($gte/$lt).
Mixed-type / Double / empty — safe full-table fallback.

Changes

protos/flow.proto — add StringPartitionRange + string_range oneof field.
connectors/mongo/qrep_partition.go — type dispatcher; numeric + string builders; pure, unit-tested computeStringBoundaries.
connectors/mongo/qrep.go — IntRange/StringRange cases in toRangeFilter.
connectors/utils/monitoring/monitoring.go — handle StringRange in addPartitionToQRepRun (avoids the unknown range type error path).

Testing

go build ./..., go vet clean.
New unit tests for computeStringBoundaries (even/clustered/sparse/duplicate/out-of-range/empty + contiguity), toRangeFilter (Int/String/ObjectID/unsupported), and _id type detection — all passing.

Notes / limitations

Numeric uses uniform value division; monitor partition balance on first run for sparse ID spaces.
Double _id and mixed-type collections fall back to full-table by design.
Requires regenerating protobuf bindings (./generate-protos.sh) before building images, as flow/generated is gitignored.

With this, it's safe to unpin `jackc/pgx` dependency and bump its version which is also done in the same PR. The selected protocol version is `3.0` which is he one we were selecting by default values with the pinned version: https://github.com/jackc/pgx/blob/4e4eaedb47b7b3cfba0a1b0a9e6a3f015764f046/pgconn/config.go#L472-L474 With the dependency update, the way of decoding macaddr fields is updated to more recent client library versions, concretely after: jackc/pgx@08c9bb1 According to that change, it was a bug to rely on Stringer representation for composite types as macaddr. The changes in this PR do precisely that, move back to using the stringer interface in the type conversion: PeerDB-io@e4b97ab

What resources are used and what test are run are given as input parameters. :memo: This is an experiment to use Tilt in CI workflows.

The Postgres-destination `IsQRepPartitionSynced` check matches the already-synced metadata row by `partitionID` alone, ignoring `flowJobName`. Since MySQL/Mongo initial loads give every table the same constant partition ID ("full-table-partition-id"), once the first table syncs, all other tables in the mirror are wrongly skipped as "already synced" and land empty. This PR adds a filter for flow name in `IsQRepPartitionSynced`

…rDB-io#4442) Test binlog `INCIDENT_EVENT` e2e with a MySQL debug image (PeerDB-io#4441) in a testcontainer. Tweaking the args brought memory usage from 300MB -> 130MB on my machine. Test is ran for mysql-pos and mysql-gtid (but not maria), also excluding ch-cluster as we're only testing the source. --------- Co-authored-by: Dmitrii Tunikov <dmitrii.tunikov@clickhouse.com> Co-authored-by: Dmitrii Tunikov <106737620+dtunikov@users.noreply.github.com>

PeerDB-io#4440) When custom partition key is not specified, determine default partition key based on table's primary key. If the primary key(s) starts with an integer or temporal type, perform parallel snapshot instead of full table snapshot. Testing: - add unit & e2e tests - validated parallel snapshotting locally

Works for replica identity `default` and `index`. Intentionally excluding `full`. Handles the most common case but given false positives/negatives keeping it to internal logs.

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [anthropics/claude-code-action](https://redirect.github.com/anthropics/claude-code-action) ([changelog](https://redirect.github.com/anthropics/claude-code-action/compare/4d7e1f0cd85743fdc93b1c8040ab54395da024e2..80b31826338489861333dc17217865dfe8085cdc)) | action | digest | `4d7e1f0` → `80b3182` | | linuxserver/openssh-server | service | digest | `d7ba00e` → `835c1f3` | | [otel/opentelemetry-collector-contrib](https://redirect.github.com/open-telemetry/opentelemetry-collector-releases) | service | minor | `0.153.0` → `0.154.0` | --- ### Release Notes <details> <summary>open-telemetry/opentelemetry-collector-releases (otel/opentelemetry-collector-contrib)</summary> ### [`v0.154.0`](https://redirect.github.com/open-telemetry/opentelemetry-collector-releases/blob/HEAD/CHANGELOG.md#v01540) [Compare Source](https://redirect.github.com/open-telemetry/opentelemetry-collector-releases/compare/v0.153.0...v0.154.0) ##### 🚩 Deprecations 🚩 - `jmxreceiver`: Remove deprecated JMX receiver from contrib distribution ([#1519](https://redirect.github.com/open-telemetry/opentelemetry-collector-releases/issues/1519)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Etc/UTC) - Branch creation - "after 5pm on monday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PeerDB-io/peerdb).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…eters (PeerDB-io#4478) - Declare different mariadb versions (start with `maria-lts`) and provide associated image, parameters and env vars as a separate object configuration. - Add a dedicated entry for gtid and pos MariaDB entries including changes in Go test code to support GTID in MariaDB tests. :memo: This change requires changes in branch protection conditions.

Follows PeerDB-io#4474, With these changes, Tilt environment on CI is fully functional. We also move to use e2e invocation directly with `gotestsum`. Additionally, internal test results ingestion is moved to an action and shared between traditional flow.yml based and Tilt based CI. Finally, this workflow is set-up so it can be invoked from other workflows as follow: ```yaml jobs: pg-ch-e2e: uses: ./.github/workflows/tilt-flow.yml with: test_matcher: TestGenericCH_PG ancillary_services: postgres clickhouse secrets: inherit ```

PeerDB-io#4436) ## Problem Linear: [DBI-810](https://linear.app/clickhouse/issue/DBI-810/b2-cdc-path-performs-no-character-set-transcoding-latin1gbksjis-text) Binlog row events carry string/ENUM/SET data as **raw bytes in each column's own character set**. Unlike the snapshot path — which runs `SET NAMES utf8mb4` so the server transcodes before sending — the CDC path got the bytes verbatim and cast them straight to Go strings (`qvalue_convert.go`). Result: for any non-utf8 column (`latin1` was the MySQL server default through 5.7, plus `gbk`/`sjis`/etc.), every non-ASCII character reached the destination as **invalid-UTF-8 mojibake on CDC while the snapshot was correct** — the same source value landing in two different encodings. ENUM/SET labels were equally raw. The collation needed to fix this was already present in `TABLE_MAP` metadata but unused for decoding. ## Fix - Resolve an `x/text` decoder per column from the `TABLE_MAP` collation metadata (`CollationMap()` for character columns, `EnumSetCollationMap()` for enum/set). - Transcode string values **and** ENUM/SET label tables to UTF-8. - `utf8`/`utf8mb3`/`utf8mb4`/`ascii`/`binary` resolve to a nil decoder → original **zero-cost passthrough**, so the common case is unaffected. - Collation id → charset name is sourced from `information_schema.COLLATIONS` and cached on the connector (covers MySQL + MariaDB without a hardcoded ~300-entry table). - Graceful degradation: unknown collation / unsupported charset / missing binlog row metadata → **warn once and pass bytes through** (no worse than today), rather than halting the mirror. Note: MySQL's `latin1` is Windows-1252 (not ISO-8859-1) — handled explicitly. ## Tests - `charset_test.go`: unit tests for the transcoders (latin1/gbk/sjis/euckr + passthrough), and a before/after assertion that a latin1 value was mojibake and is now correctly decoded. - `Test_MySQL_Charset_Consistency` (e2e): latin1 + gbk columns, asserts snapshot and CDC rows are **byte-identical** and correctly decoded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…4484) ## What Switch the **Stable Docker images** and **Update Deployment tags** workflows from the `push: tags` trigger to `release: published`. ## Why Tagging and publishing the GitHub Release were decoupled. Tags were pushed independently, so the release-drafter draft could sit unpublished even though Docker images were already built: - `v0.36.28` sat as a draft for 4 days (images built 06-15, published 06-19). - `v0.36.29` was never published — images built 06-19, draft only published manually on 06-23. Triggering off `release: published` makes **publishing the draft the single source of truth**: GitHub natively creates the `vX.Y.Z` tag at the release's target commit, which fires both workflows. ## Notes - `stable-docker.yml` needs no other change — `github.ref_name` resolves to the tag name on both event types, so `SHA_SHORT: stable-${{ github.ref_name }}` still yields `stable-vX.Y.Z` and `actions/checkout` checks out the tag automatically. - `update-docker-compose-stable.yaml` keeps `workflow_dispatch`; its `releases/latest` lookup returns the just-published release. - **Behavior change:** bare `git push origin vX.Y.Z` no longer triggers builds — you must publish a release. - **GITHUB_TOKEN caveat:** `release: published` fires on human publishes (the current flow). If release-drafter is later set to auto-publish via the default `GITHUB_TOKEN`, that bot event would not trigger these downstream workflows (GitHub's recursion guard). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

After PeerDB-io#4478, the last row in flow CI definition for the version matrix was not `latest` anymore but `stable`. This row is used by the Tilt environment to select the default target version for ClickHouse. `stable` is not a valid tag hence Tilt environments broke

Postgres destination mirrors store metadata on the destination PG peer instead of catalog. We handle this logic for PG to PG mirrors but not MySQL to PG or Mongo to PG. This PR extends this logic to fix CDC for those mirrors too, by only checking if destination is PG regardless of what source is

Was only temporarily used after joining ClickHouse. The path also has SNS sender that's not exposed in UI and seems to have been a part of the decommissioned Cloud offering, will remove that separately if there aren't signs it's used elsewhere.

Resolves DBI-833

Trailing `\x00` is the most common case of parse failures and also the most silly one. Closes DBI-859

Confirmed not used outside decommissioned Cloud

This PR contains the following updates: | Package | Update | Change | |---|---|---| | postgres | digest | `96d56f7` → `1b1689b` | --- ### Configuration 📅 **Schedule**: (in timezone Etc/UTC) - Branch creation - "after 5pm on monday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PeerDB-io/peerdb).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Maria binlog encodes vectors as MYSQL_TYPE_VARCHAR and gives us `string` while MySQL comes as `[]byte` Resolves DBI-826

…ge (PeerDB-io#4490) ## Problem After a mirror resync on a clustered ClickHouse destination, the old local shard table (e.g. `itunes_apps_shard`) is left behind as an orphan. During resync, `RenameTables` does: 1. `EXCHANGE TABLES itunes_apps AND itunes_apps_resync` — swaps the two Distributed tables 2. `DROP TABLE itunes_apps_resync` — drops the now-old Distributed shell But this only handles the Distributed layer. The local shard that the original `itunes_apps` pointed to (`itunes_apps_shard`) has no Distributed table referencing it anymore and is never dropped, leaving it as dead storage. ## Fix Before the `EXCHANGE`, resolve the shard table name from the original Distributed table's `engine_full` definition using the existing `getDistributedShardTable` helper. After the exchange and Distributed drop succeed, explicitly drop that shard table. If the shard lookup fails (e.g. non-Distributed engine, single-node deployment), we log a warning and continue — the swap itself is not blocked. ## Test plan - [ ] Run a mirror resync on a clustered ClickHouse destination - [ ] Confirm `itunes_apps_shard` (or equivalent old shard) is gone after resync completes - [ ] Confirm `itunes_apps` Distributed table still points to the new timestamped shard and data is intact

…rDB-io#4457) Resolves DBI-856

* Handle status terminating during resync-drop-flow. * Add user facing logs for waiting replication slot creation warning. * Streaming replication protocol doesn't support timeouts, so best we can do is user warning

Add parallel snapshotting support for MySQL when default primary key is a UUID string column. We best-effort detect uuid by evaluating the min and max values in the table; if both min and max value match uuid format (and have a _consistent hex casing_), we can guarantee correctness (see description for `DetectUuidWithHexCasing` for a bit more details). Original plan was to sample first and last N records, but it doesn't add a ton of value here, so just checking min/max values for simplicity. If string column is deemed non-UUID, we fallback to full table partition for now; will follow-up with partitioning for arbitrary string. Testing: add unit and integration tests to ensure uuids PKs are parallel snapshotted while arbitrary string PKs still correctly replicate with full table partition. Fixes: DBI-868

add metric to count column type change events for MySQL and postgres

- DBI-809 - handle transaction payload event (MySQL can send it if binlog_transaction_compression=ON) - handle missing GTID events (MariadbGTIDEvent, GtidTaggedLogEvent)

Parse out status vars from query event and pass the flag to the tidb parser. Add an integration test that can work with binlog more freely as we can't really observe query parsing from e2e (besides the 5.7-pos configuration that doesn't have binlog metadata to fall back on, but we need a maria test as well). As a bonus, the integration test runs in 1.2 seconds. Closes DBI-821

Already stamped on every metric

emit metric for gh-ost MySQL migrations to track it's usage

Follow-up to PeerDB-io#4436 Optimize building two extra maps per event on go-mysql side when re-encoding won't be needed (common case)

- Expanded unit tests for GTID decoding, including case description refactor. - Added e2e roundtrip gtid test. - Re-just test matrix in flow.yaml (left mariadb-pos in to test the failure of the newly added tests) Follows: PeerDB-io#4478 Closes: https://linear.app/clickhouse/issue/DBI-829

The MongoDB connector only parallelized initial snapshots when _id was a native ObjectID; numeric or string _id collections fell back to a single serial full-table partition. For large collections keyed by natural ids (e.g. app-store numeric ids or package-name strings) this makes the snapshot too slow to finish within the oplog retention window. Dispatch in GetQRepPartitions on the collection's min/max _id BSON type: - ObjectID: unchanged (uniform division of the 12-byte keyspace). - Int32/Int64: uniform min/max division, reusing utils.PartitionHelper to emit non-overlapping IntPartitionRange partitions. - String: sampling-based quantile boundaries ($sample, honouring the configured read preference) anchored by the real min/max, emitted as a new StringPartitionRange proto type with contiguous half-open ranges. Mixed-type, Double, or empty collections fall back to full-table. Add IntRange/StringRange cases to toRangeFilter and StringRange handling to monitoring.addPartitionToQRepRun. computeStringBoundaries is pure and unit-tested alongside the range-to-filter conversions.

- Add end_inclusive to StringPartitionRange proto (aligns with MySQL PR PeerDB-io#4482 adding the same field; avoids future merge conflict) - Replace sentinel-string approach with end_inclusive=true on the last string partition, using $lte instead of $lt for its MongoDB filter - Sort string _id samples via MongoDB $sort stage instead of Go sort, preserving the collection's collation order - Filter boundary samples by exact equality instead of <=/>= comparison, trusting MongoDB's min/max guarantees - Add e2e tests for string and numeric _id partitioned snapshot flows - Document the double _id gap limitation on numericPartitions (follow-up)

- Add utils.FullTablePartition() helper and use it in place of the local fullTablePartitions() in qrep_partition.go and the inline construction in qrep.go - Remove slices.Compact() from computeStringBoundaries: the seen map already deduplicates interior values, and the quantile index formula is strictly increasing when len(interior) >= numPartitions, so no consecutive duplicates can occur in the else branch - Tighten e2e partition count assertions to exact counts (10) instead of >1 for string and numeric _id tests - Delete Test_Snapshot_Non_ObjectID_Falls_Back_To_Single_Partition: string _id collections are now partitioned rather than falling back to full-table, so the test expectation no longer holds

Notify user when clickpipes user is dropped.

Add a user-facing log when a large txn result in a PullRecord sync interval to get extended. Logged once per sync_interval, but throttled at 1 minute in case customer has a very short sync intervals that makes the log excessively noisy. Fixes: DBI-681

Load tags metadata when calling getFlowsForMetrics() to allow context-ware metrics to emit pipeName when recording metrics in ScheduledTasks.

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/cache](https://redirect.github.com/actions/cache) ([changelog](https://redirect.github.com/actions/cache/compare/27d5ce7f107fe9357f9df03efb73ab90386fccae..caa296126883cff596d87d8935842f9db880ef25)) | action | digest | `27d5ce7` → `caa2961` | | [actions/setup-go](https://redirect.github.com/actions/setup-go) ([changelog](https://redirect.github.com/actions/setup-go/compare/4a3601121dd01d1626a1e23e37211e3254c1c06c..924ae3a1cded613372ab5595356fb5720e22ba16)) | action | digest | `4a36011` → `924ae3a` | | [anthropics/claude-code-action](https://redirect.github.com/anthropics/claude-code-action) ([changelog](https://redirect.github.com/anthropics/claude-code-action/compare/80b31826338489861333dc17217865dfe8085cdc..a92e7c70a4da9793dc164451d829089dc057a464)) | action | digest | `80b3182` → `a92e7c7` | | [aws-actions/configure-aws-credentials](https://redirect.github.com/aws-actions/configure-aws-credentials) ([changelog](https://redirect.github.com/aws-actions/configure-aws-credentials/compare/e7f100cf4c008499ea8adda475de1042d6975c7b..254c19bd240aabef8777f48595e9d2d7b972184b)) | action | digest | `e7f100c` → `254c19b` | | elasticsearch | service | digest | `715c5d5` → `e182d81` | | [golangci/golangci-lint-action](https://redirect.github.com/golangci/golangci-lint-action) ([changelog](https://redirect.github.com/golangci/golangci-lint-action/compare/82606bf257cbaff209d206a39f5134f0cfbfd2ee..ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a)) | action | digest | `82606bf` → `ba0d7d2` | | linuxserver/openssh-server | service | digest | `835c1f3` → `67d4c3a` | | redpandadata/redpanda | service | digest | `394cee6` → `3e72ea3` | | [release-drafter/release-drafter](https://redirect.github.com/release-drafter/release-drafter) ([changelog](https://redirect.github.com/release-drafter/release-drafter/compare/ed4bc48ec97379be2258e7b7ac2624a3e26ab809..4d75298e00d9e34c483e5ff8c68d0ea1c1940c1e)) | action | digest | `ed4bc48` → `4d75298` | --- ### Configuration 📅 **Schedule**: (in timezone Etc/UTC) - Branch creation - "after 5pm on monday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PeerDB-io/peerdb).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (in timezone Etc/UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PeerDB-io/peerdb).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

## Summary - keep the mysql-debug Dockerfile Renovate updates on the MySQL 8.0 series - document why the debug fixture should not be automatically moved across MySQL release series

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | alpine | final | minor | `3.23` → `3.24` | | [docker/dockerfile](https://redirect.github.com/moby/buildkit) | syntax | minor | `1.24` → `1.25` | | golang | stage | digest | `f1ddd9f` → `3ad5730` | | mysql | final | pinDigest | → `7dcddc0` | | [node](https://redirect.github.com/nodejs/node) | final | digest | `21f403a` → `a0b9bf0` | --- ### Configuration 📅 **Schedule**: (in timezone Etc/UTC) - Branch creation - "after 5pm on monday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/PeerDB-io/peerdb).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Mainlining https://github.com/PeerDB-io/peerdb/releases/tag/v0.36.32

Changes: * Incorporate PeerDB-io#4516 making Maria a separate DB in the matrix * Spawn out the maria-roots of e2e suites * Introduce TestIntegraion.../mysql and .../mariadb naming pattern for integration tests, filter from tilt with a matcher. MySQL-only are TestMySQLOnlyIntegration... to avoid running them from maria targets as parent tests are first ran to discover subtests. Unit tests are out of scope for tilt as they don't need any infra Also GitHub seems to not handle indentation diffs very well, most of the line changes are just subtest indentation --------- Co-authored-by: Pablo Francisco Pérez Hidalgo <pablo.perez@clickhouse.com>

Avoids stuck mirror being silent in error logs

E2E's `CompareTableSchemas` was comparing everything with an `||`, which is not a lot of comparing. Switch all to more strict `RequireEqualTableSchemas`. After trivial cleanups, one test that continued failing was `Test_Schema_Changes_Cutoff_Bug, validating schema changes handling: - in PG we test that the changes are only picked up after we get a row with that column and a schema delta (and a lazy RelationMessage right before) after PeerDB-io#2717 - in MySQL we issue a schema delta the moment we see the DDL so the row is optional This is a product decision but funny that a test revealed it. Eagle eye is codex and main investigator is claude with a bunch of nudges

## Summary Fixes PeerDB-io#4473. This updates the Postgres <15 fallback normalize statements to use the same per-batch parameters as the PG15+ `MERGE` path: ```sql _peerdb_batch_id = $1 AND _peerdb_destination_table_name = $2 ```

pfcoperez and others added 3 commits June 22, 2026 15:22

feat(tilt): Add setup peerdb peers in Tilt (PeerDB-io#4444)

b50e65a

Initial attempt to run Tilt environment in CI (PeerDB-io#4474)

222bd34

What resources are used and what test are run are given as input parameters. :memo: This is an experiment to use Tilt in CI workflows.

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from ef3cdaf to 4ff837b Compare June 23, 2026 16:18

andreyzhelnin-st changed the base branch from release/v0.36.27 to release/v0.36.29 June 23, 2026 16:18

andreyzhelnin-st self-assigned this Jun 23, 2026

Amogh-Bharadwaj and others added 21 commits June 23, 2026 23:26

Log a warning when a replica idenity column changes (PeerDB-io#4480)

78b4413

Works for replica identity `default` and `index`. Intentionally excluding `full`. Handles the most common case but given false positives/negatives keeping it to internal logs.

Handle error cases from parsing filepos (PeerDB-io#4462)

624d450

Resolves DBI-833

Support null-terminated strings in MySQL DDL (PeerDB-io#4459)

aa72725

Trailing `\x00` is the most common case of parse failures and also the most silly one. Closes DBI-859

Remove SNS and telemetry sender infra (PeerDB-io#4492)

85f91a7

Confirmed not used outside decommissioned Cloud

Support MariaDB VECTOR during CDC (PeerDB-io#4454)

c21a707

Maria binlog encodes vectors as MYSQL_TYPE_VARCHAR and gives us `string` while MySQL comes as `[]byte` Resolves DBI-826

Decode BINARY(N) as Bytes on the binlog-metadata add-column path (Pee…

8514199

…rDB-io#4457) Resolves DBI-856

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from acc2b74 to 4ff837b Compare June 25, 2026 07:32

dtunikov added 2 commits June 25, 2026 13:15

observe mysql/postgres column type changes (PeerDB-io#4488)

f43a670

add metric to count column type change events for MySQL and postgres

fix(mysql): handle transaction payload event (PeerDB-io#4433)

29306ca

- DBI-809 - handle transaction payload event (MySQL can send it if binlog_transaction_compression=ON) - handle missing GTID events (MariadbGTIDEvent, GtidTaggedLogEvent)

ilidemi and others added 8 commits June 25, 2026 23:10

Dedup sourcePeerType (PeerDB-io#4499)

256631d

Already stamped on every metric

add online_schema_migrations metric (PeerDB-io#4497)

bb70d4d

emit metric for gh-ost MySQL migrations to track it's usage

Optimize encoding maps (PeerDB-io#4494)

d1b3142

Follow-up to PeerDB-io#4436 Optimize building two extra maps per event on go-mysql side when re-encoding won't be needed (common case)

mongo: reduce e2e test doc count to 100 for faster local runs

8dac193

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from 7c14f4d to 24e30e2 Compare June 26, 2026 15:15

andreyzhelnin-st force-pushed the mongo-id-partitioning branch from 24e30e2 to b89e2fc Compare June 26, 2026 15:17

andreyzhelnin-st and others added 16 commits June 26, 2026 17:47

fix(lint): restore stdlib import group separator in qrep_partition.go

c21475e

error classification: add error code 192 (PeerDB-io#4507)

633a773

Notify user when clickpipes user is dropped.

Add tag to support propagating pipe name (PeerDB-io#4508)

ce58262

Load tags metadata when calling getFlowsForMetrics() to allow context-ware metrics to emit pipeName when recording metrics in ScheduledTasks.

Constrain mysql-debug Renovate updates (PeerDB-io#4512)

9145ee5

## Summary - keep the mysql-debug Dockerfile Renovate updates on the MySQL 8.0 series - document why the debug fixture should not be automatically moved across MySQL release series

Disable PG CDC store for CH destinations by default (PeerDB-io#4515)

a231c19

Mainlining https://github.com/PeerDB-io/peerdb/releases/tag/v0.36.32

LogFlowError on mysql ConnectionActive failure (PeerDB-io#4521)

4c85a9d

Avoids stuck mirror being silent in error logs

Fix claude review on fork PRs (PeerDB-io#4502)

a072710

Classify CH access denied (PeerDB-io#4528)

8306ce3

Merge remote-tracking branch 'origin/main' into mongo-id-partitioning

1a11c54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mongo: partition snapshots for numeric and string _id collections#1

mongo: partition snapshots for numeric and string _id collections#1
andreyzhelnin-st wants to merge 51 commits into
release/v0.36.29from
mongo-id-partitioning

andreyzhelnin-st commented Jun 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

andreyzhelnin-st commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Approach

Changes

Testing

Notes / limitations

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

andreyzhelnin-st commented Jun 23, 2026 •

edited

Loading