New Destination Connector: Dewey

[Gareth Ari Aye (lambdabaa)]

Problem

Dewey is a managed RAG service that ingests documents and runs the chunking, embedding, indexing, and retrieval pipeline server-side. Today, anyone wanting to load Airbyte source data into Dewey has to script the document upload themselves — there is no native destination, so RAG users on Dewey can't take advantage of Airbyte's source catalog for keeping their collections in sync.

Proposed Solution

Add a Python destination connector destination-dewey. Design summary:

• Pattern: closer to destination-vectara than destination-pinecone/destination-weaviate. Dewey owns embedding and chunking, so the connector does not pull in the airbyte_cdk.destinations.vector_db_based framework (no Embedder/Indexer/Writer/DocumentProcessor).
• One Dewey collection per Airbyte stream, configured via a stream_collections: { stream_name → collection_id } map, with optional auto_create_collections for first-time setups.
• Record → JSON document. Each record is uploaded as a JSON blob via POST /collections/:id/documents (multipart). Dewey accepts JSON natively, so no markdown synthesis is required. Optional text_fields projects only specific dot-paths into the indexed body; metadata_fields lifts other fields into Dewey's per-document metadata for query-time filtering.
• Sync modes:
  • overwrite: at sync start, list documents tagged airbyte_stream:<stream> and batch-delete them, then upload.
  • append: upload only.
  • append_dedup: at flush time, delete prior versions by airbyte_pk:<pk> tag, then upload.
• Auth: simple Bearer header with a project API key (dwy_live_... / dwy_test_...). Default base URL https://api.meetdewey.com/v1, overridable for self-hosted.

Implementation status

A working implementation is ready locally: 27 unit tests + 4 live integration tests passing end-to-end against the production Dewey API (check, append_dedup with PK-based replacement, overwrite). I will open the PR once this discussion is acknowledged.

Author

Personal contribution — happy to maintain.

[airbyte-support-bot]

Hi Gareth Ari Aye (@lambdabaa), thanks for the detailed proposal and for offering to contribute and maintain this!

We've escalated this to our team for review and tracking: airbytehq/oncall#12197. Note that community discussions and contributions do not have an SLA — review may take some time, and acceptance of a new community-maintained destination is a decision for the engineering team rather than something we can pre-approve here.

A few things that will help when you open the PR:

• Follow the contributor guide for new connectors. The relevant docs live under Contribute a Connector and Connector Metadata File. In particular, please make sure the connector ships with the required metadata.yaml, icon.svg, an airbyte-integrations/connectors/destination-dewey/README.md, and a user-facing docs/integrations/destinations/dewey.md page.
• Compare against existing patterns. It's worth a brief justification in the PR for why destination-dewey should use the vectara-style pattern instead of the airbyte_cdk.destinations.vector_db_based framework used by destination-pinecone, destination-weaviate, etc. Reviewers will want to make sure the chosen pattern is the best fit for a RAG-backend-as-a-service like Dewey.
• Sync mode semantics. Please document the tag-based delete behavior for overwrite and append_dedup (using airbyte_stream:<stream> and airbyte_pk:<pk> tags) clearly in the user-facing docs, since users will want to understand what gets deleted on each sync and how auto_create_collections interacts with their existing Dewey collections.
• Tests and CI. Make sure the unit tests run under the standard connector test scaffolding so CI can pick them up. Live integration tests typically need to be wired up via the existing connector secrets infrastructure if they're going to run in CI.
• Maintenance and support tier. Since you're proposing this as a personal contribution, please be explicit in the PR about ongoing maintenance expectations — the team will likely classify this as a community-supported connector unless agreed otherwise.

Feel free to open the PR whenever you're ready. We'll also share this with the connectors team via the internal tracking issue so they're aware before it lands.

---

Need more help? Join the Airbyte Community Slack for peer support and to chat with the connectors team about contributions.

---

Devin session

[Gareth Ari Aye (lambdabaa)]

PR for new connector: #77771