Skip to content

Commit 477439e

Browse files
Deprecate non-ClickHouse / non-Postgres destination connectors (#4491)
Implements the code/config deliverables for #4465 — signaling the deprecation of the non-ClickHouse / non-Postgres **destination** connectors. No connector code is removed; deprecated connectors stay fully functional. **Deprecated destinations:** Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent + Redpanda), Azure Event Hubs, Google Pub/Sub, and S3. Actively-maintained paths: Postgres → ClickHouse / ClickHouse Cloud / Postgres. ## Changes - **README** — deprecation notice placed just after the intro (above the fold) + a connector status matrix. BigQuery is shown as a maintained **source** and deprecated **destination**. - **docs/deprecated-connectors.md** (new) — pin + fork migration guide, per-connector paths. - **CONTRIBUTING.md** (new) — links to the migration guide. - **UI** (`peer-types` route + `SelectSource`) — uniform "Deprecated" badge in the peer-creation grid. Since peers are direction-agnostic and BigQuery is a valid source, BigQuery is marked "Deprecated\*" with a footnote: *"Deprecated as a destination only."* - **GitHub automation** — issue form with a destination-connector dropdown + deprecation notice; `.github/labeler.yml` + `deprecated-connector.yml` workflow that labels deprecated-connector PRs and posts a single de-duplicated notice (never auto-closes). ## Notes / follow-ups (not in this PR) - The `deprecated` label must be created once in repo **Settings → Labels** (labels aren't defined in-repo). - Post-merge: apply the `deprecated` label + standard comment to existing open non-CH destination issues/PRs (do not auto-close). - BigQuery is deprecated **only as a destination** — it remains a supported source. ## Verification - Brought the full dev stack up (after `generate-protos.sh`); verified `/api/peer-types` returns the correct per-connector `deprecated`/`deprecatedRole` data and `/peers/create` renders the badges + footnote without errors. - All issue-form / labeler / workflow YAML validated. - GitHub automation validated end-to-end on a fork (see the pinned summary comment). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 3cd6c9c commit 477439e

12 files changed

Lines changed: 606 additions & 34 deletions

File tree

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
name: Bug report
2+
description: Report a problem with PeerDB
3+
title: "[Bug]: "
4+
labels: ["bug"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
Thanks for taking the time to file a bug report! Please fill out the fields below so we can reproduce and triage the issue.
10+
11+
- type: textarea
12+
id: description
13+
attributes:
14+
label: Description
15+
description: A clear and concise description of what the bug is, including the expected behavior and what actually happened.
16+
placeholder: When I run a CDC mirror to ..., I expected ... but instead ...
17+
validations:
18+
required: true
19+
20+
- type: textarea
21+
id: steps_to_reproduce
22+
attributes:
23+
label: Steps to reproduce
24+
description: Step-by-step instructions to reproduce the behavior.
25+
placeholder: |
26+
1. Create a peer with ...
27+
2. Create a mirror with ...
28+
3. Observe ...
29+
validations:
30+
required: true
31+
32+
- type: input
33+
id: version
34+
attributes:
35+
label: PeerDB version
36+
description: The PeerDB version (or Docker image tag / commit SHA) you are running.
37+
placeholder: e.g. v0.30.0 or stable-dev
38+
validations:
39+
required: true
40+
41+
- type: dropdown
42+
id: source_connector
43+
attributes:
44+
label: Source connector
45+
description: If this issue relates to a source connector, select it here. Leave blank if not applicable.
46+
options:
47+
- Postgres
48+
- MySQL
49+
- MongoDB
50+
- BigQuery
51+
- Other
52+
validations:
53+
required: false
54+
55+
- type: markdown
56+
attributes:
57+
value: |
58+
### ⚠️ Deprecated destination connectors
59+
60+
The following destination connectors are **deprecated**: **Snowflake**, **BigQuery**, **ElasticSearch**, **Kafka**, **Redpanda**, **Confluent**, **Azure Event Hubs**, **Google Pub/Sub**, and **S3**.
61+
62+
These destinations are no longer actively maintained, but remain functional. Bugs affecting only these destinations are unlikely to be prioritized. Note: BigQuery is deprecated **only as a destination** — it remains a supported source. See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md) for details and migration guidance.
63+
64+
- type: dropdown
65+
id: destination_connector
66+
attributes:
67+
label: Destination connector
68+
description: If this issue relates to a destination connector, select it here (see the deprecation notice above). Leave blank if not applicable.
69+
options:
70+
- ClickHouse
71+
- ClickHouse Cloud
72+
- Postgres
73+
- S3
74+
- Snowflake
75+
- BigQuery
76+
- ElasticSearch
77+
- Kafka
78+
- Redpanda
79+
- Confluent
80+
- Azure Event Hubs
81+
- Google Pub/Sub
82+
- Other
83+
validations:
84+
required: false

.github/ISSUE_TEMPLATE/config.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
blank_issues_enabled: false
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: Feature request
2+
description: Suggest a feature or improvement for PeerDB
3+
title: "[Feature]: "
4+
labels: ["enhancement"]
5+
body:
6+
- type: markdown
7+
attributes:
8+
value: |
9+
Thanks for suggesting an improvement! Please describe the problem you are trying to solve and what you would like to happen.
10+
11+
- type: textarea
12+
id: problem
13+
attributes:
14+
label: Problem / motivation
15+
description: What problem are you trying to solve, and what is the use case?
16+
placeholder: When replicating ..., I need ... but currently ...
17+
validations:
18+
required: true
19+
20+
- type: textarea
21+
id: proposal
22+
attributes:
23+
label: Proposed solution
24+
description: Describe what you would like to happen.
25+
validations:
26+
required: true
27+
28+
- type: textarea
29+
id: alternatives
30+
attributes:
31+
label: Alternatives considered
32+
description: Any alternative solutions or workarounds you have considered.
33+
validations:
34+
required: false
35+
36+
- type: dropdown
37+
id: source_connector
38+
attributes:
39+
label: Source connector
40+
description: If this request relates to a source connector, select it here. Leave blank if not applicable.
41+
options:
42+
- Postgres
43+
- MySQL
44+
- MongoDB
45+
- BigQuery
46+
- Other
47+
validations:
48+
required: false
49+
50+
- type: markdown
51+
attributes:
52+
value: |
53+
### ⚠️ Deprecated destination connectors
54+
55+
The following destination connectors are **deprecated** and no longer actively maintained: **Snowflake**, **BigQuery**, **ElasticSearch**, **Kafka**, **Redpanda**, **Confluent**, **Azure Event Hubs**, **Google Pub/Sub**, and **S3**. New features for these destinations are unlikely to be prioritized. Note: BigQuery is deprecated **only as a destination** — it remains a supported source. See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md).
56+
57+
- type: dropdown
58+
id: destination_connector
59+
attributes:
60+
label: Destination connector
61+
description: If this request relates to a destination connector, select it here (see the deprecation notice above). Leave blank if not applicable.
62+
options:
63+
- ClickHouse
64+
- ClickHouse Cloud
65+
- Postgres
66+
- S3
67+
- Snowflake
68+
- BigQuery
69+
- ElasticSearch
70+
- Kafka
71+
- Redpanda
72+
- Confluent
73+
- Azure Event Hubs
74+
- Google Pub/Sub
75+
- Other
76+
validations:
77+
required: false

.github/labeler.yml

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Configuration for actions/labeler@v5.
2+
# Applies the `deprecated` label to PRs that touch deprecated destination connectors:
3+
# Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent/Redpanda), Azure Event Hubs, Google Pub/Sub, S3.
4+
# ClickHouse and Postgres (the actively-maintained destinations) are intentionally excluded.
5+
deprecated:
6+
- changed-files:
7+
- any-glob-to-any-file:
8+
# Destination-only connectors — the whole connector is deprecated.
9+
- flow/connectors/snowflake/**
10+
- flow/connectors/elasticsearch/**
11+
- flow/connectors/eventhub/**
12+
- flow/connectors/kafka/**
13+
- flow/connectors/pubsub/**
14+
- flow/connectors/s3/**
15+
# End-to-end tests
16+
- flow/e2e/snowflake*.go
17+
- flow/e2e/elasticsearch*.go
18+
- flow/e2e/eventhub*.go
19+
- flow/e2e/kafka*.go
20+
- flow/e2e/pubsub*.go
21+
- flow/e2e/s3_*.go
22+
# UI peer forms
23+
- ui/components/PeerForms/SnowflakeForm.tsx
24+
- ui/components/PeerForms/ElasticsearchConfigForm.tsx
25+
- ui/components/PeerForms/KafkaConfig.tsx
26+
- ui/components/PeerForms/PubSubConfig.tsx
27+
- ui/components/PeerForms/S3Form.tsx
28+
- ui/components/PeerForms/Eventhubs/**
29+
# UI peer-create helpers. The literal `[` in the [peerType] dynamic
30+
# route segment is escaped as `[[]` so minimatch treats it literally
31+
# rather than as a character class.
32+
- ui/app/peers/create/[[]peerType]/helpers/sf.ts
33+
- ui/app/peers/create/[[]peerType]/helpers/es.ts
34+
- ui/app/peers/create/[[]peerType]/helpers/eh.ts
35+
- ui/app/peers/create/[[]peerType]/helpers/ka.ts
36+
- ui/app/peers/create/[[]peerType]/helpers/ps.ts
37+
- ui/app/peers/create/[[]peerType]/helpers/s3.ts
38+
# BigQuery is dual-role: only its DESTINATION is deprecated, while its
39+
# source is actively maintained and lives in the same directory. List
40+
# the destination-specific files explicitly so BigQuery *source*
41+
# changes (source.go, source_schema.go, qrep_object_pull.go,
42+
# bigquery_source_test.go) are never auto-labeled as deprecated.
43+
# (A `!`-negation / `all-globs-to-all-files` block was tried first but
44+
# silently disables the whole labeler config in actions/labeler v5.)
45+
- flow/connectors/bigquery/bigquery.go
46+
- flow/connectors/bigquery/merge_stmt_generator.go
47+
- flow/connectors/bigquery/qrep.go
48+
- flow/connectors/bigquery/qrep_avro_sync.go
49+
- flow/connectors/bigquery/clustering.go
50+
- flow/e2e/bigquery_test.go
51+
- flow/e2e/bigquery_qrep_test.go
52+
- ui/components/PeerForms/BigqueryConfig.tsx
53+
- ui/app/peers/create/[[]peerType]/helpers/bq.ts
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
name: Deprecated connector
2+
3+
# - Labels PRs that touch deprecated destination connectors (actions/labeler).
4+
# - On bug reports, applies the `deprecated` label only when the selected
5+
# destination connector (issue-form dropdown) is a deprecated one.
6+
# - Posts a single informational notice on any issue/PR carrying the
7+
# `deprecated` label. It never closes the issue or PR.
8+
#
9+
# NOTE: the `deprecated` label must be created once in the repository settings
10+
# (Settings -> Labels) — labels are not defined in this repo.
11+
12+
on:
13+
pull_request_target:
14+
types: [opened, synchronize, reopened]
15+
pull_request:
16+
types: [labeled]
17+
issues:
18+
types: [opened, edited, labeled]
19+
20+
permissions:
21+
contents: read
22+
23+
jobs:
24+
# Apply path-based labels (incl. `deprecated`) to PRs per .github/labeler.yml.
25+
label:
26+
if: github.event_name == 'pull_request_target'
27+
runs-on: ubuntu-latest
28+
permissions:
29+
contents: read
30+
pull-requests: write
31+
steps:
32+
- uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5.0.0
33+
with:
34+
sync-labels: false
35+
36+
# On a new/edited bug report, apply `deprecated` only when the chosen
37+
# destination connector (issue-form dropdown) is a deprecated one. Issues
38+
# that select an actively-maintained destination are left untouched.
39+
triage-issue:
40+
if: github.event_name == 'issues' && (github.event.action == 'opened' || github.event.action == 'edited')
41+
runs-on: ubuntu-latest
42+
permissions:
43+
issues: write
44+
steps:
45+
- uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
46+
with:
47+
script: |
48+
const issue = context.payload.issue;
49+
const body = issue.body || '';
50+
// Extract the "Destination connector" dropdown value from the form body.
51+
const match = body.match(/###\s*Destination connector\s*\r?\n+\s*(.+)/);
52+
const selection = match ? match[1].trim() : '';
53+
const deprecated = new Set([
54+
'Snowflake', 'BigQuery', 'ElasticSearch', 'Kafka', 'Redpanda',
55+
'Confluent', 'Azure Event Hubs', 'Google Pub/Sub', 'S3',
56+
]);
57+
if (!deprecated.has(selection)) {
58+
core.info(`Destination "${selection || '(none)'}" is not a deprecated connector; not labeling.`);
59+
return;
60+
}
61+
if ((issue.labels || []).some((l) => (l.name || l) === 'deprecated')) {
62+
core.info('Issue already labeled deprecated.');
63+
return;
64+
}
65+
await github.rest.issues.addLabels({
66+
owner: context.repo.owner,
67+
repo: context.repo.repo,
68+
issue_number: issue.number,
69+
labels: ['deprecated'],
70+
});
71+
core.info(`Labeled issue #${issue.number} deprecated (destination: ${selection}).`);
72+
73+
# Post a one-time notice when the `deprecated` label is present on an issue/PR.
74+
# Depends on the prior jobs so it observes labels they applied within this same
75+
# run (a GITHUB_TOKEN label write does not itself emit a `labeled` event).
76+
# `if: always()` keeps it running when an upstream job is skipped.
77+
notice:
78+
needs: [label, triage-issue]
79+
if: always()
80+
runs-on: ubuntu-latest
81+
permissions:
82+
issues: write
83+
pull-requests: write
84+
steps:
85+
- uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
86+
with:
87+
script: |
88+
const marker = '<!-- deprecated-connector-notice -->';
89+
const number =
90+
context.eventName === 'issues'
91+
? context.payload.issue?.number
92+
: context.payload.pull_request?.number;
93+
if (!number) {
94+
core.info('No issue/PR number in payload; nothing to do.');
95+
return;
96+
}
97+
98+
// Re-fetch labels so we observe any applied by the label/triage jobs
99+
// earlier in this run (rather than the stale event payload).
100+
const { data: labels } = await github.rest.issues.listLabelsOnIssue({
101+
owner: context.repo.owner,
102+
repo: context.repo.repo,
103+
issue_number: number,
104+
});
105+
if (!labels.some((l) => l.name === 'deprecated')) {
106+
core.info('No deprecated label present; nothing to do.');
107+
return;
108+
}
109+
110+
// Guard against duplicate comments using the hidden marker.
111+
const comments = await github.paginate(github.rest.issues.listComments, {
112+
owner: context.repo.owner,
113+
repo: context.repo.repo,
114+
issue_number: number,
115+
});
116+
if (comments.some((c) => c.body && c.body.includes(marker))) {
117+
core.info('Deprecated-connector notice already posted; skipping.');
118+
return;
119+
}
120+
121+
const body = [
122+
marker,
123+
'### ⚠️ Deprecated destination connector',
124+
'',
125+
'This issue or pull request relates to a **deprecated** destination connector ' +
126+
'(Snowflake, BigQuery, ElasticSearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub, or S3).',
127+
'',
128+
'These destinations are **no longer actively maintained**, but remain functional. ' +
129+
'We are unlikely to prioritize new work here.',
130+
'',
131+
'> Note: BigQuery is deprecated **only as a destination** — it remains a supported source.',
132+
'',
133+
'If you depend on one of these connectors, we recommend:',
134+
'',
135+
'- **Pin** to a known-good PeerDB version so behavior stays stable.',
136+
'- **Fork** the repository if you need to carry your own changes.',
137+
'',
138+
'See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md) for details and migration guidance.',
139+
].join('\n');
140+
141+
await github.rest.issues.createComment({
142+
owner: context.repo.owner,
143+
repo: context.repo.repo,
144+
issue_number: number,
145+
body,
146+
});
147+
core.info(`Posted deprecated-connector notice on #${number}.`);

CONTRIBUTING.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Contributing to PeerDB
2+
3+
Thanks for your interest in contributing to PeerDB! Bug reports, feature requests, and pull requests are all welcome. If you have a question, feel free to drop by our [Slack](https://slack.peerdb.io/).
4+
5+
## Deprecated connectors
6+
7+
Several destination connectors (Snowflake, BigQuery, ElasticSearch, Kafka including Confluent and Redpanda, Azure Event Hubs, Google Pub/Sub, and S3) are deprecated and no longer actively maintained. They remain fully functional, and no code is currently being removed. (BigQuery is deprecated only as a destination — it remains a supported source.)
8+
9+
If you depend on one of these connectors, see the [deprecated connectors migration guide](docs/deprecated-connectors.md) for how to pin to a release or fork the relevant connector code.

0 commit comments

Comments
 (0)