Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions quest_db/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,16 @@ This connector syncs high-performance time series data from QuestDB to your Five

Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template quest_db
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

- Syncs time series data from multiple QuestDB tables
Expand Down Expand Up @@ -45,22 +55,21 @@ The connector requires the following configuration parameters:

Configuration parameters:


- `tables` (required) - Comma-separated list of table names to sync (e.g., `SENSOR_DATA`, `MARKET_TICKS`)
- `tables` (required) - Comma-separated list of table names to sync (e.g., `SENSOR_DATA`, `MARKET_TICKS`)
- `host` (optional) - QuestDB server hostname or IP address (defaults to `localhost`)
- `port` (optional) - QuestDB REST API port (defaults to `9000`)
- `username` (optional) - QuestDB username for HTTP basic authentication
- `password` (optional) - QuestDB password for HTTP basic authentication
- `batch_size` (optional) - Number of records to fetch per API request (defaults to `1000`)
- `timestamp_column` (optional) - Column name for timestamp-based incremental sync (defaults to `timestamp`)

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

This connector does not require a `requirements.txt` file as it only uses standard library modules and the `requests` library, which is pre-installed in the Fivetran environment.

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

Expand Down
24 changes: 19 additions & 5 deletions rabbitmq/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,33 @@
# RabbitMQ Connector SDK Example
# RabbitMQ Connector Example

## Connector overview

This connector syncs messages from RabbitMQ queues to Fivetran. The connector retrieves messages from specified queues using RabbitMQ's `basic_get` method. It consumes the messages only after a successful sync to prevent duplicate reads.

## Requirements

- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
- Operating system:
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## Getting started

Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template rabbitmq
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

- Synchronizes messages from multiple RabbitMQ queues
- Captures complete message metadata including delivery tags, routing keys, headers, and timestamps
- Implements incremental sync with per-queue state tracking using delivery tags
Expand All @@ -24,7 +38,6 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co

## Configuration file


```json
{
"connection_url": "<YOUR_RABBITMQ_CONNECTION_URL>",
Expand All @@ -39,7 +52,7 @@ The configuration keys required for your connector are as follows:
- `queues` (required) - Comma-separated list of queue names to sync (for example: "orders,payments,notifications")
- `batch_size` (optional) - Number of messages to process per batch (defaults to 100)

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

Expand All @@ -51,9 +64,10 @@ pika==1.3.2

The connector uses the `pika` library (version 1.3.2) for connecting to and consuming messages from RabbitMQ.

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

The connector uses AMQP connection URLs for authentication. The connection URL includes credentials embedded in the URL format.

Example configurations:
Expand Down Expand Up @@ -83,7 +97,7 @@ The connector automatically handles AMQPS/TLS connections when using the `amqps:

## Data handling

**Important**: Messages are permanently removed from RabbitMQ after a successful sync. This connector consumes messages using `basic_ack`, which deletes them from the queue. Use this connector only with dedicated analytical queues (for example, dead-letter or audit queues); never use it on production operational queues.
> Note: Messages are permanently removed from RabbitMQ after a successful sync. This connector consumes messages using `basic_ack`, which deletes them from the queue. Use this connector only with dedicated analytical queues (for example, dead-letter or audit queues); never use it on production operational queues.

The connector performs message consumption and transformation. Refer to the `fetch_and_upsert_messages_batch` function for message handling:

Expand Down
54 changes: 33 additions & 21 deletions raven_db/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,24 @@ The connector is designed to handle large datasets efficiently through streaming

- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
- Operating system:
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## Getting started

Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template raven_db
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

- Connect to RavenDB clusters with certificate-based authentication
Expand All @@ -40,13 +50,14 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co
```

### Configuration parameters

- `ravendb_urls` (required): Your RavenDB Cloud URLs. For clusters, provide comma-separated URLs
- `database_name` (required): Name of the RavenDB database to sync from
- `certificate_base64` (required): Base64-encoded client certificate (PEM format)

Important: You must provide the certificate as a base64-encoded PEM certificate string.
> Note: You must provide the certificate as a base64-encoded PEM certificate string.

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

Expand All @@ -56,7 +67,7 @@ The connector requires the RavenDB Python client:
ravendb
```

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

Expand All @@ -68,11 +79,11 @@ To obtain your certificate:
3. Download the client certificate package.
4. Convert the certificate to base64 format:

- **macOS/Linux:**
- macOS/Linux:
```sh
base64 -i certificate.pem
```
- **Windows:**
- Windows:
```cmd
certutil -encode certificate.pem certificate.b64
```
Expand Down Expand Up @@ -114,20 +125,21 @@ The connector implements the following error-handling strategies:
This connector creates tables in your destination based on the collections you sync. The table structure follows this pattern:

### Example: `ORDERS` table (if syncing an Orders collection)
- Primary Key: `Id` (STRING - RavenDB document ID)

- Primary key: `Id` (STRING - RavenDB document ID)
- Columns:
- `Id` (STRING): Document unique identifier
- `OrderNumber` (STRING): Order number
- `CustomerName` (STRING): Customer name
- `TotalAmount` (FLOAT): Order total
- `Status` (STRING): Order status
- `Items` (JSON): Order items as JSON array
- `ShippingAddress_Street` (STRING): Flattened nested field
- `ShippingAddress_City` (STRING): Flattened nested field
- `LastModified` (UTC_DATETIME): Document last update timestamp from metadata

Note: The actual schema depends on your document structure. Nested objects are flattened with underscore separators, and arrays are stored as JSON strings.
- `Id` (STRING): Document unique identifier
- `OrderNumber` (STRING): Order number
- `CustomerName` (STRING): Customer name
- `TotalAmount` (FLOAT): Order total
- `Status` (STRING): Order status
- `Items` (JSON): Order items as JSON array
- `ShippingAddress_Street` (STRING): Flattened nested field
- `ShippingAddress_City` (STRING): Flattened nested field
- `LastModified` (UTC_DATETIME): Document last update timestamp from metadata

> Note: The actual schema depends on your document structure. Nested objects are flattened with underscore separators, and arrays are stored as JSON strings.

## Additional considerations

The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team.
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team.
Loading
Loading