-
Notifications
You must be signed in to change notification settings - Fork 198
Deprecate non-ClickHouse / non-Postgres destination connectors #4491
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
2dca815
Deprecate non-ClickHouse / non-Postgres destinations: docs + UI
Jeremyyang920 0e50c1c
Add deprecated-connector GitHub automation
Jeremyyang920 21ff46e
Address review (ilidemi)
Jeremyyang920 1e94bbf
Deprecate QRep and XMin mirror types
Jeremyyang920 c402b9c
ci: re-trigger Lint UI after rebasing onto main
Jeremyyang920 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: Bug report | ||
| description: Report a problem with PeerDB | ||
| title: "[Bug]: " | ||
| labels: ["bug"] | ||
| body: | ||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| Thanks for taking the time to file a bug report! Please fill out the fields below so we can reproduce and triage the issue. | ||
|
|
||
| - type: textarea | ||
| id: description | ||
| attributes: | ||
| label: Description | ||
| description: A clear and concise description of what the bug is, including the expected behavior and what actually happened. | ||
| placeholder: When I run a CDC mirror to ..., I expected ... but instead ... | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: steps_to_reproduce | ||
| attributes: | ||
| label: Steps to reproduce | ||
| description: Step-by-step instructions to reproduce the behavior. | ||
| placeholder: | | ||
| 1. Create a peer with ... | ||
| 2. Create a mirror with ... | ||
| 3. Observe ... | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: input | ||
| id: version | ||
| attributes: | ||
| label: PeerDB version | ||
| description: The PeerDB version (or Docker image tag / commit SHA) you are running. | ||
| placeholder: e.g. v0.30.0 or stable-dev | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: dropdown | ||
| id: source_connector | ||
| attributes: | ||
| label: Source connector | ||
| description: If this issue relates to a source connector, select it here. Leave blank if not applicable. | ||
| options: | ||
| - Postgres | ||
| - MySQL | ||
| - MongoDB | ||
| - BigQuery | ||
| - Other | ||
| validations: | ||
| required: false | ||
|
|
||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| ### ⚠️ Deprecated destination connectors | ||
|
|
||
| The following destination connectors are **deprecated**: **Snowflake**, **BigQuery**, **ElasticSearch**, **Kafka**, **Redpanda**, **Confluent**, **Azure Event Hubs**, **Google Pub/Sub**, and **S3**. | ||
|
|
||
| These destinations are no longer actively maintained, but remain functional. Bugs affecting only these destinations are unlikely to be prioritized. Note: BigQuery is deprecated **only as a destination** — it remains a supported source. See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md) for details and migration guidance. | ||
|
|
||
| - type: dropdown | ||
| id: destination_connector | ||
| attributes: | ||
| label: Destination connector | ||
| description: If this issue relates to a destination connector, select it here (see the deprecation notice above). Leave blank if not applicable. | ||
| options: | ||
| - ClickHouse | ||
| - ClickHouse Cloud | ||
| - Postgres | ||
| - S3 | ||
| - Snowflake | ||
| - BigQuery | ||
| - ElasticSearch | ||
| - Kafka | ||
| - Redpanda | ||
| - Confluent | ||
| - Azure Event Hubs | ||
| - Google Pub/Sub | ||
| - Other | ||
| validations: | ||
| required: false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| blank_issues_enabled: false | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| name: Feature request | ||
| description: Suggest a feature or improvement for PeerDB | ||
| title: "[Feature]: " | ||
| labels: ["enhancement"] | ||
| body: | ||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| Thanks for suggesting an improvement! Please describe the problem you are trying to solve and what you would like to happen. | ||
|
|
||
| - type: textarea | ||
| id: problem | ||
| attributes: | ||
| label: Problem / motivation | ||
| description: What problem are you trying to solve, and what is the use case? | ||
| placeholder: When replicating ..., I need ... but currently ... | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: proposal | ||
| attributes: | ||
| label: Proposed solution | ||
| description: Describe what you would like to happen. | ||
| validations: | ||
| required: true | ||
|
|
||
| - type: textarea | ||
| id: alternatives | ||
| attributes: | ||
| label: Alternatives considered | ||
| description: Any alternative solutions or workarounds you have considered. | ||
| validations: | ||
| required: false | ||
|
|
||
| - type: dropdown | ||
| id: source_connector | ||
| attributes: | ||
| label: Source connector | ||
| description: If this request relates to a source connector, select it here. Leave blank if not applicable. | ||
| options: | ||
| - Postgres | ||
| - MySQL | ||
| - MongoDB | ||
| - BigQuery | ||
| - Other | ||
| validations: | ||
| required: false | ||
|
|
||
| - type: markdown | ||
| attributes: | ||
| value: | | ||
| ### ⚠️ Deprecated destination connectors | ||
|
|
||
| The following destination connectors are **deprecated** and no longer actively maintained: **Snowflake**, **BigQuery**, **ElasticSearch**, **Kafka**, **Redpanda**, **Confluent**, **Azure Event Hubs**, **Google Pub/Sub**, and **S3**. New features for these destinations are unlikely to be prioritized. Note: BigQuery is deprecated **only as a destination** — it remains a supported source. See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md). | ||
|
|
||
| - type: dropdown | ||
| id: destination_connector | ||
| attributes: | ||
| label: Destination connector | ||
| description: If this request relates to a destination connector, select it here (see the deprecation notice above). Leave blank if not applicable. | ||
| options: | ||
| - ClickHouse | ||
| - ClickHouse Cloud | ||
| - Postgres | ||
| - S3 | ||
| - Snowflake | ||
| - BigQuery | ||
| - ElasticSearch | ||
| - Kafka | ||
| - Redpanda | ||
| - Confluent | ||
| - Azure Event Hubs | ||
| - Google Pub/Sub | ||
| - Other | ||
| validations: | ||
| required: false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Configuration for actions/labeler@v5. | ||
| # Applies the `deprecated` label to PRs that touch deprecated destination connectors: | ||
| # Snowflake, BigQuery, ElasticSearch, Kafka (incl. Confluent/Redpanda), Azure Event Hubs, Google Pub/Sub, S3. | ||
| # ClickHouse and Postgres (the actively-maintained destinations) are intentionally excluded. | ||
| deprecated: | ||
| - changed-files: | ||
| - any-glob-to-any-file: | ||
| # Destination-only connectors — the whole connector is deprecated. | ||
| - flow/connectors/snowflake/** | ||
| - flow/connectors/elasticsearch/** | ||
| - flow/connectors/eventhub/** | ||
| - flow/connectors/kafka/** | ||
| - flow/connectors/pubsub/** | ||
| - flow/connectors/s3/** | ||
| # End-to-end tests | ||
| - flow/e2e/snowflake*.go | ||
| - flow/e2e/elasticsearch*.go | ||
| - flow/e2e/eventhub*.go | ||
| - flow/e2e/kafka*.go | ||
| - flow/e2e/pubsub*.go | ||
| - flow/e2e/s3_*.go | ||
| # UI peer forms | ||
| - ui/components/PeerForms/SnowflakeForm.tsx | ||
| - ui/components/PeerForms/ElasticsearchConfigForm.tsx | ||
| - ui/components/PeerForms/KafkaConfig.tsx | ||
| - ui/components/PeerForms/PubSubConfig.tsx | ||
| - ui/components/PeerForms/S3Form.tsx | ||
| - ui/components/PeerForms/Eventhubs/** | ||
| # UI peer-create helpers. The literal `[` in the [peerType] dynamic | ||
| # route segment is escaped as `[[]` so minimatch treats it literally | ||
| # rather than as a character class. | ||
| - ui/app/peers/create/[[]peerType]/helpers/sf.ts | ||
| - ui/app/peers/create/[[]peerType]/helpers/es.ts | ||
| - ui/app/peers/create/[[]peerType]/helpers/eh.ts | ||
| - ui/app/peers/create/[[]peerType]/helpers/ka.ts | ||
| - ui/app/peers/create/[[]peerType]/helpers/ps.ts | ||
| - ui/app/peers/create/[[]peerType]/helpers/s3.ts | ||
| # BigQuery is dual-role: only its DESTINATION is deprecated, while its | ||
| # source is actively maintained and lives in the same directory. List | ||
| # the destination-specific files explicitly so BigQuery *source* | ||
| # changes (source.go, source_schema.go, qrep_object_pull.go, | ||
| # bigquery_source_test.go) are never auto-labeled as deprecated. | ||
| # (A `!`-negation / `all-globs-to-all-files` block was tried first but | ||
| # silently disables the whole labeler config in actions/labeler v5.) | ||
| - flow/connectors/bigquery/bigquery.go | ||
| - flow/connectors/bigquery/merge_stmt_generator.go | ||
| - flow/connectors/bigquery/qrep.go | ||
| - flow/connectors/bigquery/qrep_avro_sync.go | ||
|
Jeremyyang920 marked this conversation as resolved.
|
||
| - flow/connectors/bigquery/clustering.go | ||
| - flow/e2e/bigquery_test.go | ||
| - flow/e2e/bigquery_qrep_test.go | ||
| - ui/components/PeerForms/BigqueryConfig.tsx | ||
| - ui/app/peers/create/[[]peerType]/helpers/bq.ts | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| name: Deprecated connector | ||
|
|
||
| # - Labels PRs that touch deprecated destination connectors (actions/labeler). | ||
| # - On bug reports, applies the `deprecated` label only when the selected | ||
| # destination connector (issue-form dropdown) is a deprecated one. | ||
| # - Posts a single informational notice on any issue/PR carrying the | ||
| # `deprecated` label. It never closes the issue or PR. | ||
| # | ||
| # NOTE: the `deprecated` label must be created once in the repository settings | ||
| # (Settings -> Labels) — labels are not defined in this repo. | ||
|
|
||
| on: | ||
| pull_request_target: | ||
| types: [opened, synchronize, reopened] | ||
| pull_request: | ||
| types: [labeled] | ||
| issues: | ||
| types: [opened, edited, labeled] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| # Apply path-based labels (incl. `deprecated`) to PRs per .github/labeler.yml. | ||
| label: | ||
| if: github.event_name == 'pull_request_target' | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| pull-requests: write | ||
| steps: | ||
| - uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5.0.0 | ||
| with: | ||
| sync-labels: false | ||
|
|
||
| # On a new/edited bug report, apply `deprecated` only when the chosen | ||
| # destination connector (issue-form dropdown) is a deprecated one. Issues | ||
| # that select an actively-maintained destination are left untouched. | ||
| triage-issue: | ||
| if: github.event_name == 'issues' && (github.event.action == 'opened' || github.event.action == 'edited') | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| issues: write | ||
| steps: | ||
| - uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 | ||
| with: | ||
| script: | | ||
| const issue = context.payload.issue; | ||
| const body = issue.body || ''; | ||
| // Extract the "Destination connector" dropdown value from the form body. | ||
| const match = body.match(/###\s*Destination connector\s*\r?\n+\s*(.+)/); | ||
| const selection = match ? match[1].trim() : ''; | ||
| const deprecated = new Set([ | ||
| 'Snowflake', 'BigQuery', 'ElasticSearch', 'Kafka', 'Redpanda', | ||
| 'Confluent', 'Azure Event Hubs', 'Google Pub/Sub', 'S3', | ||
| ]); | ||
| if (!deprecated.has(selection)) { | ||
| core.info(`Destination "${selection || '(none)'}" is not a deprecated connector; not labeling.`); | ||
| return; | ||
| } | ||
| if ((issue.labels || []).some((l) => (l.name || l) === 'deprecated')) { | ||
| core.info('Issue already labeled deprecated.'); | ||
| return; | ||
| } | ||
| await github.rest.issues.addLabels({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: issue.number, | ||
| labels: ['deprecated'], | ||
| }); | ||
| core.info(`Labeled issue #${issue.number} deprecated (destination: ${selection}).`); | ||
|
|
||
| # Post a one-time notice when the `deprecated` label is present on an issue/PR. | ||
| # Depends on the prior jobs so it observes labels they applied within this same | ||
| # run (a GITHUB_TOKEN label write does not itself emit a `labeled` event). | ||
| # `if: always()` keeps it running when an upstream job is skipped. | ||
| notice: | ||
| needs: [label, triage-issue] | ||
| if: always() | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| issues: write | ||
| pull-requests: write | ||
| steps: | ||
| - uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 | ||
| with: | ||
| script: | | ||
| const marker = '<!-- deprecated-connector-notice -->'; | ||
| const number = | ||
| context.eventName === 'issues' | ||
| ? context.payload.issue?.number | ||
| : context.payload.pull_request?.number; | ||
| if (!number) { | ||
| core.info('No issue/PR number in payload; nothing to do.'); | ||
| return; | ||
| } | ||
|
|
||
| // Re-fetch labels so we observe any applied by the label/triage jobs | ||
| // earlier in this run (rather than the stale event payload). | ||
| const { data: labels } = await github.rest.issues.listLabelsOnIssue({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: number, | ||
| }); | ||
| if (!labels.some((l) => l.name === 'deprecated')) { | ||
| core.info('No deprecated label present; nothing to do.'); | ||
| return; | ||
| } | ||
|
|
||
| // Guard against duplicate comments using the hidden marker. | ||
| const comments = await github.paginate(github.rest.issues.listComments, { | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: number, | ||
| }); | ||
| if (comments.some((c) => c.body && c.body.includes(marker))) { | ||
| core.info('Deprecated-connector notice already posted; skipping.'); | ||
| return; | ||
| } | ||
|
|
||
| const body = [ | ||
| marker, | ||
| '### ⚠️ Deprecated destination connector', | ||
| '', | ||
| 'This issue or pull request relates to a **deprecated** destination connector ' + | ||
| '(Snowflake, BigQuery, ElasticSearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub, or S3).', | ||
| '', | ||
| 'These destinations are **no longer actively maintained**, but remain functional. ' + | ||
| 'We are unlikely to prioritize new work here.', | ||
| '', | ||
| '> Note: BigQuery is deprecated **only as a destination** — it remains a supported source.', | ||
| '', | ||
| 'If you depend on one of these connectors, we recommend:', | ||
| '', | ||
| '- **Pin** to a known-good PeerDB version so behavior stays stable.', | ||
| '- **Fork** the repository if you need to carry your own changes.', | ||
| '', | ||
| 'See the [deprecated connectors documentation](https://github.com/PeerDB-io/peerdb/blob/main/docs/deprecated-connectors.md) for details and migration guidance.', | ||
| ].join('\n'); | ||
|
|
||
| await github.rest.issues.createComment({ | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| issue_number: number, | ||
| body, | ||
| }); | ||
| core.info(`Posted deprecated-connector notice on #${number}.`); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Contributing to PeerDB | ||
|
|
||
| Thanks for your interest in contributing to PeerDB! Bug reports, feature requests, and pull requests are all welcome. If you have a question, feel free to drop by our [Slack](https://slack.peerdb.io/). | ||
|
|
||
| ## Deprecated connectors | ||
|
|
||
| Several destination connectors (Snowflake, BigQuery, ElasticSearch, Kafka including Confluent and Redpanda, Azure Event Hubs, Google Pub/Sub, and S3) are deprecated and no longer actively maintained. They remain fully functional, and no code is currently being removed. (BigQuery is deprecated only as a destination — it remains a supported source.) | ||
|
|
||
| If you depend on one of these connectors, see the [deprecated connectors migration guide](docs/deprecated-connectors.md) for how to pin to a release or fork the relevant connector code. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.