You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Signal deprecation of the Snowflake, BigQuery, ElasticSearch, Kafka
(incl. Confluent/Redpanda), Azure Event Hubs, and Google Pub/Sub
*destination* connectors. No code is removed; they remain fully functional.
Actively-maintained paths: Postgres -> ClickHouse / ClickHouse Cloud /
Postgres.
- README: deprecation notice (above the fold) + connector status matrix;
BigQuery is listed as a maintained source and a deprecated destination.
- docs/deprecated-connectors.md: pin + fork migration guide.
- CONTRIBUTING.md: links to the migration guide.
- UI: "Deprecated" badge in the peer-creation grid. BigQuery is dual-role,
so it is marked destination-only ("Deprecated*" + footnote) since it is
still a valid source peer.
Refs #4465
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thanks for your interest in contributing to PeerDB! Bug reports, feature requests, and pull requests are all welcome. If you have a question, feel free to drop by our [Slack](https://slack.peerdb.io/).
4
+
5
+
## Deprecated connectors
6
+
7
+
Several destination connectors (Snowflake, BigQuery, ElasticSearch, Kafka including Confluent and Redpanda, Azure Event Hubs, and Google Pub/Sub) are deprecated and no longer actively maintained. They remain fully functional, and no code is being removed.
8
+
9
+
If you depend on one of these connectors, see the [deprecated connectors migration guide](docs/deprecated-connectors.md) for how to pin to a release or fork the relevant connector code.
Copy file name to clipboardExpand all lines: README.md
+22-1Lines changed: 22 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,8 @@ At PeerDB, we are building a fast, simple and the most cost effective way to str
17
17
18
18
We support different modes of streaming - log based (CDC), cursor based (timestamp or integer) and XMIN based. Performance wise, we are 10x faster than existing tools. Features wise, we support native Postgres features such as comprehensive set of data-types incl. jsonb/arrays/geospatial, efficiently streaming TOAST columns, schema changes and so on.
19
19
20
+
> **Connector deprecation notice:** The Snowflake, BigQuery, ElasticSearch, Kafka (including Confluent and Redpanda variants), Azure Event Hubs, and Google Pub/Sub **destination** connectors are deprecated and no longer actively maintained. They remain fully functional in the current and all prior releases, and no code is being removed. The actively-maintained paths going forward are Postgres → ClickHouse, Postgres → ClickHouse Cloud, and Postgres → Postgres. This deprecation applies to the **destination** role only — BigQuery remains a supported **source**. See the [connector status matrix](#status) and the [deprecated connectors migration guide](docs/deprecated-connectors.md) for how to pin to a release or fork the relevant code.
21
+
20
22
## Get started
21
23
22
24
```bash
@@ -89,7 +91,26 @@ You can use Postgres’ ecosystem to manage your ETL —
89
91
90
92
## Status
91
93
92
-
We have expanded our connector ecosystem to support multiple source connectors beyond Postgres, including MySQL and MongoDB. You can check the status of connectors [here](https://docs.peerdb.io/sql/commands/supported-connectors)
94
+
We have expanded our connector ecosystem to support multiple source connectors beyond Postgres, including MySQL and MongoDB. You can check the status of connectors [here](https://docs.peerdb.io/sql/commands/supported-connectors).
95
+
96
+
| Connector | Direction (source/destination) | Status |
The following **destination** connectors are deprecated and no longer actively maintained:
4
+
5
+
- Snowflake
6
+
- BigQuery (destination role only — see below)
7
+
- ElasticSearch
8
+
- Kafka (including Confluent and Redpanda variants)
9
+
- Azure Event Hubs
10
+
- Google Pub/Sub
11
+
12
+
These connectors remain **fully functional** in the current and all prior releases. No code is being removed, and they will **not** be automatically removed from existing deployments. The actively-maintained paths going forward are Postgres → ClickHouse, Postgres → ClickHouse Cloud, and Postgres → Postgres. Sources (Postgres, MySQL, MongoDB) and the S3 destination are not affected by this deprecation.
13
+
14
+
> **BigQuery is deprecated only as a destination.** BigQuery remains a **supported source** for QRep mirrors and is not deprecated in that role. The migration guidance below applies to BigQuery's use as a *destination*.
15
+
16
+
If you rely on a deprecated destination, you have two supported options to keep using it: pin to a release that includes it, or fork the relevant connector code.
17
+
18
+
## How to pin to the last actively-maintained release
19
+
20
+
Deprecated connectors continue to ship in current and prior releases, so the simplest way to keep using one is to pin your deployment to a specific release rather than tracking the latest mainline.
21
+
22
+
-**Docker / docker-compose:** pin the PeerDB image tags to an explicit release tag instead of a moving tag like `latest`. In your `docker-compose.yml`, replace floating tags with the latest release tag that still includes the connector you depend on.
23
+
-**Helm chart:** pin the chart version and the image tags in your `values.yaml` to that same release.
24
+
25
+
Pin to the latest release tag before any future removal so that you have the most recent fixes while retaining the connector. Once pinned, validate your pipelines and avoid upgrading past a release that no longer carries the connector you need.
26
+
27
+
## How to fork
28
+
29
+
Because no code is being removed, you can fork PeerDB and carry the relevant connector code yourself. The connectors are self-contained enough that forking and maintaining a single connector is practical:
30
+
31
+
1. Fork the repository.
32
+
2. Keep the connector's flow code, its UI form and helper, and (optionally) its end-to-end tests so you can continue validating it.
33
+
3. Rebase your fork periodically onto upstream `main` to pick up fixes to shared infrastructure.
34
+
35
+
The sections below list the exact code paths to carry for each deprecated connector.
36
+
37
+
### Snowflake
38
+
39
+
-`flow/connectors/snowflake/`
40
+
-`ui/components/PeerForms/SnowflakeForm.tsx`
41
+
-`ui/app/peers/create/[peerType]/helpers/sf.ts`
42
+
-`flow/e2e/snowflake*.go`
43
+
44
+
### BigQuery (destination only)
45
+
46
+
> Deprecated only as a **destination**. BigQuery is still a supported **source**, so keep this code if you use BigQuery as a source.
0 commit comments