From 1579f09b537f1e3ad6ec2a3bbbe7ce80f20d1bef Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Tue, 7 Apr 2026 23:22:08 +0530 Subject: [PATCH 01/19] docs: improve README files for batch 01 --- apache_hbase/README.md | 48 +++++----- apache_hive/using_pyhive/README.md | 36 +++++--- apache_hive/using_sqlalchemy/README.md | 39 +++++--- apache_pulsar/README.md | 32 +++++-- arango_db/README.md | 48 +++++++--- awardco/README.md | 40 +++++++-- aws_athena/using_boto3/README.md | 43 +++++---- aws_athena/using_sqlalchemy/README.md | 44 +++++---- aws_dynamo_db_authentication/README.md | 47 ++++++---- aws_rds_oracle/readme.md | 70 ++++++++------- betterstack/README.md | 16 +++- cassandra/README.md | 57 ++++++------ checkly/README.md | 119 +++++++++++++++---------- clerk/README.md | 28 +++++- clickhouse/README.md | 55 ++++++------ commonpaper/README.md | 41 +++++---- couchbase_capella/README.md | 48 +++++----- couchbase_magma/README.md | 47 ++++++---- courier/README.md | 16 +++- customer_thermometer/README.md | 18 +++- data_camp/README.md | 40 +++++---- dgraph/README.md | 16 +++- discord/README.md | 38 +++++--- documentdb/README.md | 51 ++++++----- docusign/README.md | 95 +++++++++++--------- dolphin_db/README.md | 28 ++++-- dragonfly_db/README.md | 16 +++- elastic_email/README.md | 18 +++- firebird_db/README.md | 94 ++++++++++--------- fleetio/README.md | 69 +++++++------- fred/README.md | 17 +++- gcp_pub_sub/README.md | 22 ++--- github/README.md | 16 +++- github_traffic/README.md | 87 ++++++++++++------ gnews/README.md | 41 ++++++--- 35 files changed, 974 insertions(+), 566 deletions(-) diff --git a/apache_hbase/README.md b/apache_hbase/README.md index cb5e6a9..54cb0b8 100644 --- a/apache_hbase/README.md +++ b/apache_hbase/README.md @@ -6,15 +6,25 @@ This connector shows how to sync data from Apache HBase databases using Fivetran ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template apache_hbase +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -26,18 +36,18 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se ## Configuration file -The connector requires the following configuration parameters: +The connector requires the following configuration parameters: -``` +```json { "hostname": "", "port": "", - "table_name": "", + "table_name": "", "column_family": "" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -47,7 +57,7 @@ This connector requires the happybase library to communicate with Apache HBase: happybase==1.2.0 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -61,8 +71,7 @@ Refer to the `execute_query_and_upsert` function, which implements batched data ## Data handling - -The connector processes data from HBase in the following way: +The connector processes data from HBase in the following way: - Creates a connection to the HBase server using the `happybase` library. - Scans the specified table with a filter based on the `created_at` timestamp. - Decodes and transforms each row into a structured format @@ -71,26 +80,25 @@ The connector processes data from HBase in the following way: ## Error handling -The connector implements error handling at multiple levels: +The connector implements error handling at multiple levels: - Connection errors: Captured in the `create_hbase_connection` function, raising a descriptive RuntimeError - Data processing errors: The `execute_query_and_upsert` function uses try-except blocks to handle missing columns in row data, logging warnings and continuing execution without failing the entire sync -## Tables Created +## Tables created - -The connector creates one table: +The connector creates one table: - profile_table The schema of the created table is as follows: -``` +```json { "table": "profile_table", "primary_key": ["id"], "columns": { "id": "STRING", - "created_at": "UTC_DATETIME", - }, + "created_at": "UTC_DATETIME" + } } ``` diff --git a/apache_hive/using_pyhive/README.md b/apache_hive/using_pyhive/README.md index 492d0e6..0f19111 100644 --- a/apache_hive/using_pyhive/README.md +++ b/apache_hive/using_pyhive/README.md @@ -6,16 +6,26 @@ This connector shows how to fetch data from Apache Hive using the `PyHive` and ` ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template apache_hive +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Direct connection to Apache Hive data source. @@ -26,7 +36,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co This connector requires the following configuration parameters to establish a connection to your Hive instance: -``` +```json { "hostname": "YOUR_HIVE_HOSTNAME", "port": "", @@ -36,7 +46,7 @@ This connector requires the following configuration parameters to establish a co } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -48,11 +58,11 @@ thrift_sasl sasl ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector supports `CUSTOM` authentication for Apache Hive. You need to provide: +The connector supports `CUSTOM` authentication for Apache Hive. You need to provide: - `hostname`: The address of your Hive server - `port`: The port number Hive is listening on (typically 10000) - `username`: Your Hive username @@ -62,25 +72,25 @@ Authentication is handled in the `create_hive_connection` function. ## Data handling -The connector performs the following data handling operations: +The connector performs the following data handling operations: - Fetching: Data is retrieved from Apache Hive using SQL queries with timestamp-based filtering. - Processing: The `process_row` function converts raw Hive data into dictionary format suitable for Fivetran. - Column names are extracted and mapped to their values. -- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow. +- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow. - State management: The connector tracks the latest created timestamp to enable incremental syncs. ## Tables created The connector creates a table named `PEOPLE` with the following schema: -``` +```json { "table": "people", "primary_key": ["id"], "columns": { "id": "INT", "created_at": "UTC_DATETIME" - }, + } } ``` diff --git a/apache_hive/using_sqlalchemy/README.md b/apache_hive/using_sqlalchemy/README.md index 104ee13..dca1308 100644 --- a/apache_hive/using_sqlalchemy/README.md +++ b/apache_hive/using_sqlalchemy/README.md @@ -6,16 +6,26 @@ This connector demonstrates how to fetch data from Apache Hive using `SQLAlchemy ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template apache_hive +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Connection to Apache Hive data source using SQLAlchemy ORM. @@ -26,16 +36,16 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co This connector requires the following configuration parameters to establish a connection to your Hive instance: -``` +```json { "hostname": "", "port": "", "username": "", - "database": "", + "database": "" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -47,13 +57,14 @@ PyHive==0.7.0 thrift_sasl sasl ``` + `PyHive` is required for actual dialect implementation for Hive along with the `SQLAlchemy` ORM. The `thrift_sasl` and `sasl` packages are necessary for SASL authentication, which is commonly used with Hive. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following: +The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following: - `hostname`: The address of your Hive server - `port`: The port number Hive is listening on (typically 10000) - `username`: Your Hive username @@ -63,7 +74,7 @@ Authentication is handled in the `create_hive_connection` function. ## Data handling -The connector performs the following data handling operations: +The connector performs the following data handling operations: - Fetching: Data is retrieved from Apache Hive using SQLAlchemy with raw SQL queries and stream options. - Processing: The `process_row` function converts raw Hive data into a dictionary format suitable for Fivetran. - Column names are extracted and mapped to their values. @@ -74,17 +85,17 @@ The connector performs the following data handling operations: The connector creates a table named `PEOPLE` with the following schema: -``` +```json { "table": "people", "primary_key": ["id"], "columns": { "id": "INT", "created_at": "UTC_DATETIME" - }, + } } ``` ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/).. +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/). diff --git a/apache_pulsar/README.md b/apache_pulsar/README.md index bf34257..cdd665f 100644 --- a/apache_pulsar/README.md +++ b/apache_pulsar/README.md @@ -1,11 +1,13 @@ # Apache Pulsar Connector Example ## Connector overview -This connector demonstrates how to fetch data from Apache Pulsar topics and sync it to a destination using the Fivetran Connector SDK. It supports multiple topics and uses Pulsar's Reader API to consume messages with proper checkpointing for incremental syncs. + +This connector demonstrates how to fetch data from Apache Pulsar topics and sync it to a destination using the Fivetran Connector SDK. It supports multiple topics and uses Pulsar's Reader API to consume messages with proper checkpointing for incremental syncs. The connector is designed for organizations using Apache Pulsar who need to sync streaming data to their destination for analytics, such as clickstream events, payment transactions, and application logs. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -13,9 +15,21 @@ The connector is designed for organizations using Apache Pulsar who need to sync - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template apache_pulsar +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Supports syncing multiple Pulsar topics simultaneously, each to its own table - Uses Pulsar's Reader API with checkpointing to track progress and resume from last position - Automatically creates warehouse tables with proper data types @@ -24,9 +38,10 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Uses robust error handling with detailed logging ## Configuration file + The `configuration.json` file contains the connection details for your Apache Pulsar cluster. Update the following keys with your Pulsar cluster details: -``` +```json { "service_url": "", "tenant": "", @@ -43,30 +58,36 @@ Configuration parameters: - `topics` (required) - Comma-separated list of topic names to sync. - `auth_token` (optional) - Optional authentication token for secured Pulsar clusters. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies the Python libraries required by the connector. For this Apache Pulsar connector, the following library is required: ``` pulsar-client>=3.4.0 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector supports authentication using Apache Pulsar authentication tokens. For local standalone Pulsar instances, authentication is typically not required. For cloud or secured clusters, you can provide an authentication token in the `auth_token` configuration parameter. To obtain an authentication token, refer to your Pulsar provider's documentation (e.g., DataStax Astra Streaming, StreamNative Cloud). ## Pagination + The connector processes messages in batches to prevent memory overflow. It reads up to `__MAX_MESSAGES_PER_TOPIC` (default: `1000`) messages per topic per sync. The connector uses a timeout mechanism (`__READ_TIMEOUT_MS`, default: `5000ms`) to detect when all available messages have been read. This approach ensures efficient memory usage while maintaining good throughput. ## Data handling + Each Pulsar message is parsed and transformed into a structured record before being upserted to the destination. The connector creates a separate table for each topic, with the table name normalized from the topic name (replacing hyphens and dots with underscores). Message payloads are parsed as JSON when possible; otherwise, they are stored as raw strings or base64-encoded data. Refer to the `parse_message` function in `connector.py`. ## Error handling + The connector implements error handling at multiple levels. Configuration validation ensures all required parameters are present before sync starts. Connection errors are caught and raised with descriptive messages. Individual message processing errors are logged as warnings and the connector continues processing subsequent messages. Timeout exceptions are used to detect when all available messages have been consumed. ## Tables created + The connector creates one table per Pulsar topic. Each table has the following schema: | Column | Type | Description | @@ -82,7 +103,8 @@ The connector creates one table per Pulsar topic. Each table has the following s | `sequence_id` | INT | Message sequence ID | | `synced_at` | UTC_DATETIME | When Fivetran synced this message | -Fore more details, refer to the `schema` function in `connector.py`. +For more details, refer to the `schema` function in `connector.py`. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/arango_db/README.md b/arango_db/README.md index e070ca9..7945220 100644 --- a/arango_db/README.md +++ b/arango_db/README.md @@ -1,11 +1,13 @@ # ArangoDB Connector Example ## Connector overview + This connector demonstrates how to sync data from ArangoDB, a native multi-model database that combines document, graph, and key-value capabilities. The connector syncs three collections from a travel dataset: airports (document collection), flights (edge collection for graph relationships), and points-of-interest (document collection). This example showcases ArangoDB's unique multi-model architecture and is ideal for use cases involving interconnected data like social networks, recommendation engines, and travel planning systems. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -13,9 +15,21 @@ The connector syncs three collections from a travel dataset: airports (document - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template arango_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Syncs document collections (airports, points-of-interest) and edge collections (flights) - Demonstrates ArangoDB's multi-model capabilities combining documents and graphs - Implements batch processing with checkpointing for large datasets @@ -23,9 +37,10 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Preserves ArangoDB's native fields including `_key`, `_from`, and `_to` for graph relationships ## Configuration file + The connector requires the following configuration parameters: -``` +```json { "host": "", "database": "", @@ -36,23 +51,25 @@ The connector requires the following configuration parameters: Configuration parameters: -- `host` (requried): Your ArangoDB host. -- `database` (requried): Your ArangoDB database name. -- `username` (requried): Your ArangoDB username. -- `password` (requried): Your ArangoDB password. +- `host` (required): Your ArangoDB host. +- `database` (required): Your ArangoDB database name. +- `username` (required): Your ArangoDB username. +- `password` (required): Your ArangoDB password. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector requires the `python-arango` package to connect to ArangoDB databases. ``` python-arango ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector uses username and password authentication to connect to ArangoDB. The credentials are specified in the configuration file and passed with the `ArangoClient.db()` method. Refer to the `connect_to_arangodb()` function in `connector.py`. To set up authentication: @@ -63,18 +80,21 @@ To set up authentication: 4. Ensure the user has `READ` permissions on the collections that need to be synced. ## Pagination + The connector implements offset-based pagination using ArangoDB's `skip` and `limit` parameters. Each collection is synced in batches of 1,000 records (defined by `__CHECKPOINT_BATCH_SIZE`). The connector tracks the current offset in the state for each collection, allowing it to resume from the correct position after interruptions. Refer to the `sync_collection_batches()` function in `connector.py` for pagination logic details. ## Data handling -The connector processes each ArangoDB collection independently and upserts documents as-is to the destination tables. -Document collections (airports, points-of-interest) contain standard fields like name, location coordinates, and metadata. Edge collections (flights) include special ArangoDB fields (`_from`, `_to`) that define graph relationships between airports. +The connector processes each ArangoDB collection independently and upserts documents as-is to the destination tables. + +Document collections (airports, points-of-interest) contain standard fields like name, location coordinates, and metadata. Edge collections (flights) include special ArangoDB fields (`_from`, `_to`) that define graph relationships between airports. All ArangoDB system fields (`_key`, `_id`, `_rev`) are preserved in the destination for data lineage. The schema definition includes only the primary key (`_key`); Fivetran infers all other columns and data types automatically. Refer to the `schema()` and `sync_collection()` functions in `connector.py` for data handling details. ## Error handling + The connector implements comprehensive error handling with the following strategies: - Connection failures are caught and logged with detailed error messages before raising a `RuntimeError` - Collection access errors are caught at the collection level with specific error context @@ -84,13 +104,15 @@ The connector implements comprehensive error handling with the following strateg Refer to the `connect_to_arangodb()`, `sync_collection()`, and `update()` functions in `connector.py` for error handling details. ## Tables created + The connector creates the following tables: -| Table Name | Type | Primary Key | Description | +| Table name | Type | Primary key | Description | |----------------------|---------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `AIRPORTS` | Document Collection | `_key` | Airport information including name, city, state, country, coordinates (lat, long), and VIP status. All ArangoDB system fields (`_key`, `_id`, `_rev`) are preserved. | -| `FLIGHTS` | Edge Collection | `_key` | Flight connections between airports (graph relationships). Includes ArangoDB graph fields (`_from`, `_to`) plus flight details like date, times, carrier, flight number, tail number, and distance. | -| `POINTS_OF_INTEREST` | Document Collection | `_key` | Travel destination information including title, type, description, coordinates (latitude, longitude), URL, article, contact information (phone, email), and `last_edit` timestamp. | +| `AIRPORTS` | Document collection | `_key` | Airport information including name, city, state, country, coordinates (lat, long), and VIP status. All ArangoDB system fields (`_key`, `_id`, `_rev`) are preserved. | +| `FLIGHTS` | Edge collection | `_key` | Flight connections between airports (graph relationships). Includes ArangoDB graph fields (`_from`, `_to`) plus flight details like date, times, carrier, flight number, tail number, and distance. | +| `POINTS_OF_INTEREST` | Document collection | `_key` | Travel destination information including title, type, description, coordinates (latitude, longitude), URL, article, contact information (phone, email), and `last_edit` timestamp. | ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/awardco/README.md b/awardco/README.md index 18a8db7..032bbda 100644 --- a/awardco/README.md +++ b/awardco/README.md @@ -1,14 +1,17 @@ # Awardco Users Connector Example -Awardco is an employee recognition platform. The Awardco API typically exposes RESTful JSON endpoints to manage resources such as users and recognition-related entities. -Authentication is performed via an API key supplied in request headers. Common behaviors include paginated responses, timestamp fields for change tracking, and conventional HTTP status codes. +Awardco is an employee recognition platform. The Awardco API typically exposes RESTful JSON endpoints to manage resources such as users and recognition-related entities. + +Authentication is performed via an API key supplied in request headers. Common behaviors include paginated responses, timestamp fields for change tracking, and conventional HTTP status codes. Refer to [Awardco documentation](https://www.awardco.com/) for additional details and production configurations. ## Connector overview + This example connector uses the Fivetran Connector SDK to sync Awardco user data into your destination. It performs incremental syncs based on a timestamp cursor, upserts user rows into a single `USER` table, and emits checkpoints for reliable resumption. It also supports paginated requests and can pass a timestamp filter to the Awardco API so only changed records are returned (avoiding full table scans). There is a local mock mode for offline development and testing. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -16,20 +19,33 @@ This example connector uses the Fivetran Connector SDK to sync Awardco user data - Linux: Ubuntu 20.04+ / Debian 10+ / Amazon Linux 2+ (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template awardco +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Incremental sync based on `updated_at` timestamp - Pagination support for large datasets - Automatic retry with exponential backoff for transient failures - Configurable timestamp query parameter for API compatibility ## Configuration file + The connector reads configuration values from `configuration.json` (string values only). Example: -``` +```json { "api_key": "", "base_url": "", @@ -37,8 +53,6 @@ Example: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - Configuration parameters: - `api_key` (required): Awardco API key. - `base_url` (required): Awardco API base URL. Do not include a trailing `/api` because the connector appends the endpoint path (for example, `/api/users`). @@ -48,13 +62,17 @@ Configuration parameters: - Validation — `awardco-users-connector/connector.py:70-81` - Main config usage in update — `awardco-users-connector/connector.py:198-199` +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. + ## Requirements file + This connector does not require any additional packages beyond those provided by the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -This connector uses an API key. The request includes the key in the `apiKey` header when calling the Awardco API. + +This connector uses an API key. The request includes the key in the `apiKey` header when calling the Awardco API. To obtain the API key, do the following: @@ -62,18 +80,21 @@ To obtain the API key, do the following: 2. Set `api_key` in `configuration.json`. 3. Set `base_url` (for example, `https://api.awardco.com`) in `configuration.json`. -Note: Refer to the `make_request_with_retry()` and `fetch_users()` functions in `connector.py` for implementation details. See also `awardco-users-connector/connector.py:58` and `awardco-users-connector/connector.py:72`. +> Note: Refer to the `make_request_with_retry()` and `fetch_users()` functions in `connector.py` for implementation details. See also `awardco-users-connector/connector.py:58` and `awardco-users-connector/connector.py:72`. ## Pagination + This connector implements simple page-based pagination. The `update` loop requests pages from the users endpoint and processes each page in turn. If the Awardco API supports a query parameter for returning only records updated since a timestamp, the connector will pass the checkpointed `last_sync_time` on every page request to avoid full-table scans. ## Data handling + - Table mapping: All user records are upserted into the `USER` table. - Primary key: `employeeId`. - Incremental field: `updated_at` (the maximum value is stored as `last_sync_time`). - State management: The connector writes a checkpoint with `last_sync_time` after processing each page. ## Error handling + The connector validates the configuration and wraps network and processing steps in a try/except, surfacing failures with a clear error message. Logging uses the [Connector SDK logger](https://fivetran.com/docs/connector-sdk/technical-reference/connector-sdk-code/connector-sdk-logs) for observability. Recommendations: @@ -82,7 +103,7 @@ Recommendations: ## Tables created -The connector creates the `USER` table. +The connector creates the `USER` table. ### USER @@ -92,4 +113,5 @@ The connector creates the `USER` table. After a debug run, validate the DuckDB output (default: `warehouse.db`) and check operation counts in the CLI summary. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/aws_athena/using_boto3/README.md b/aws_athena/using_boto3/README.md index c8e94ba..4255962 100644 --- a/aws_athena/using_boto3/README.md +++ b/aws_athena/using_boto3/README.md @@ -1,6 +1,7 @@ # AWS Athena Connector Example - Using Boto3 ## Connector overview + This example demonstrates how to sync data from Amazon Athena into a Fivetran destination table using the Fivetran Connector SDK and the `boto3` AWS SDK for Python. The connector executes a query against Athena, polls for the result, and streams the resulting rows using `op.upsert()`. This pattern is ideal for: @@ -8,22 +9,32 @@ This pattern is ideal for: - Working with large result sets using paginated tokens (`NextToken`). - Handling cloud data access via AWS credentials. -Note: You must provide your AWS credentials and an S3 staging location for Athena query results. - +> Note: You must provide your AWS credentials and an S3 staging location for Athena query results. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template aws_athena +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Connects to AWS Athena using `boto3`. - Starts a query execution and polls until complete. - Uses `get_query_results()` to fetch paginated results using NextToken. @@ -31,8 +42,8 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se - Uses `op.upsert()` to sync rows into the customers table. - Uses `op.checkpoint()` to persist state. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -45,38 +56,38 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` boto3 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -AWS authentication is handled via the `boto3.client()` function using the credentials provided in the configuration file. Make sure these credentials are secure and limited in scope (e.g., read-only access). +AWS authentication is handled via the `boto3.client()` function using the credentials provided in the configuration file. Make sure these credentials are secure and limited in scope (e.g., read-only access). ## Pagination + The connector handles paginated Athena results using `NextToken`: - It skips the first row (header). - If `NextToken` is present, it fetches the next page. - Continues until all rows are processed. - ## Data handling + - Query string: `SELECT * FROM test_rows`. - Response rows are transformed and emitted to the `CUSTOMERS` table. - Supported Athena data types include `VarCharValue`, `BigIntValue`, `DoubleValue`, `BooleanValue`. - Null fields are handled and converted to Python `None`. - ## Error handling + - If Athena query fails, the sync ends and prints the failure state. - The connector checks for terminal statuses: `SUCCEEDED`, `FAILED`, or `CANCELLED`. - All paginated results are upserted directly as they are retrieved. @@ -84,8 +95,8 @@ The connector handles paginated Athena results using `NextToken`: - Adding timeout logic for long-running queries. - Adding retry handling for transient AWS errors. - ## Tables created + The connector creates a `CUSTOMERS` table: ```json @@ -98,6 +109,6 @@ The connector creates a `CUSTOMERS` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/aws_athena/using_sqlalchemy/README.md b/aws_athena/using_sqlalchemy/README.md index 6a74ac5..53888f6 100644 --- a/aws_athena/using_sqlalchemy/README.md +++ b/aws_athena/using_sqlalchemy/README.md @@ -1,36 +1,47 @@ # AWS Athena Connector Example - Using SQLAlchemy and PyAthena ## Connector overview + This connector demonstrates how to use SQLAlchemy with PyAthena to connect to Amazon Athena, execute a query, and stream the results to Fivetran using the Connector SDK. It shows how to work with paginated result sets using `fetchmany()` and how to construct Athena-compatible connection strings using credentials and S3 staging configuration. This pattern is helpful when: - You want to connect to Athena using SQLAlchemy ORM abstraction. -- You prefer PyAthena’s REST-based driver over raw `boto3` usage. +- You prefer PyAthena's REST-based driver over raw `boto3` usage. - You need to work with large Athena query results efficiently. -Note: You must provide your AWS credentials and an S3 staging location for Athena query results. - +> Note: You must provide your AWS credentials and an S3 staging location for Athena query results. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template aws_athena +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Connects to Athena using SQLAlchemy + PyAthena REST dialect. - Streams query results in chunks using `fetchmany()`. - Emits each row using `op.upsert()` into the `CUSTOMERS` table. - Performs state checkpointing to support resumable syncs. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -43,10 +54,10 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` @@ -54,33 +65,34 @@ PyAthena sqlalchemy ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + Authentication is handled via the credentials passed into the SQLAlchemy connection string: ``` awsathena+rest://:@athena..amazonaws.com:443/?s3_staging_dir= ``` - ## Pagination + - The result set is streamed in chunks using `result.fetchmany(2)`. - This helps prevent memory overload for large Athena results. - ## Data handling + - Query executed: `SELECT * FROM test_rows`. - Assumes the table contains: `customer_id`, `first_name`, `last_name`, `email`. - Each row is upserted as a dictionary using `op.upsert()`. ## Error handling + - Query results are paginated safely via `fetchmany()`. - If the connection string is invalid or credentials are wrong, the connector will raise an error. - Consider wrapping query execution in a `try/except` block for production use. - ## Tables created + The connector creates a `CUSTOMERS` table: ```json @@ -93,6 +105,6 @@ The connector creates a `CUSTOMERS` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/aws_dynamo_db_authentication/README.md b/aws_dynamo_db_authentication/README.md index 9ba2b7e..9a32ba7 100644 --- a/aws_dynamo_db_authentication/README.md +++ b/aws_dynamo_db_authentication/README.md @@ -1,6 +1,7 @@ # AWS DynamoDB Connector Using IAM Role Authentication ## Connector overview + This connector demonstrates how to sync data from Amazon DynamoDB using the Fivetran Connector SDK and AWS IAM role-based authentication. It authenticates using assumed IAM roles via STS, discovers all tables, extracts their primary keys dynamically, and syncs each table using a parallel scan. This example highlights: @@ -11,29 +12,39 @@ This example highlights: Refer to [Boto3 DynamoDB Docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html) for more detail on AWS client behavior. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template aws_dynamo_db_authentication +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Uses `boto3` STS to assume a role for DynamoDB access. - Discovers all available tables using `list_tables()`. -- Dynamically builds schema by querying each table’s key schema. +- Dynamically builds schema by querying each table's key schema. - Uses `aws_dynamodb_parallel_scan` to perform fast, concurrent scans. - Syncs paginated records via `op.upsert()`. - Checkpoints sync state using `op.checkpoint()`. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -45,10 +56,10 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` @@ -56,36 +67,36 @@ aws_dynamodb_parallel_scan==1.1.0 boto3==1.37.4 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector authenticates by: - Using `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to create an STS client. - Calling `sts.assume_role()` with the `ROLE_ARN`. -- Using the assumed role’s credentials to access DynamoDB. - +- Using the assumed role's credentials to access DynamoDB. ## Pagination + Pagination is handled via: -- `aws_dynamodb_parallel_scan.get_paginator(`) +- `aws_dynamodb_parallel_scan.get_paginator(` - Parallel scanning with `TotalSegments=4` and `Limit=10` per page. - ## Data handling + - Schema is inferred by calling `describe_table()` on each table. - Keys and values from each DynamoDB record are normalized using `map_item()`. - Nested arrays are converted to strings. - All tables are synced in parallel and checkpointed. - ## Error handling + - Errors during schema discovery or sync are logged with `log.severe()`. - Exceptions are re-raised to surface failures in the connector. - You can extend error handling with retry/backoff for robustness. - ## Tables created + The connector creates a `CUSTOMERS` table: ```json @@ -96,6 +107,6 @@ The connector creates a `CUSTOMERS` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/aws_rds_oracle/readme.md b/aws_rds_oracle/readme.md index 1313381..7e34c3f 100644 --- a/aws_rds_oracle/readme.md +++ b/aws_rds_oracle/readme.md @@ -6,7 +6,7 @@ This connector demonstrates how to sync records from an AWS RDS Oracle database ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) @@ -16,12 +16,22 @@ This connector demonstrates how to sync records from an AWS RDS Oracle database Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to install the SDK, configure your environment, and run the example locally. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template aws_rds_oracle +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features -- Direct connection to AWS RDS Oracle database -- Incremental data extraction based on the `LAST_UPDATED` column -- Support for primary key identification -- Easy configuration via JSON file +- Direct connection to AWS RDS Oracle database +- Incremental data extraction based on the `LAST_UPDATED` column +- Support for primary key identification +- Easy configuration via JSON file ## Configuration file @@ -29,15 +39,15 @@ The connector requires the following configuration parameters in `configuration. ```json { -"host": "", -"port": "", -"service_name": "", -"user": "", -"password": "" + "host": "", + "port": "", + "service_name": "", + "user": "", + "password": "" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -47,7 +57,7 @@ The connector requires the `oracledb` package to connect to Oracle databases. oracledb==3.3.0 ``` -Note: The `fivetran_connector_sdk` and `requests` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -55,29 +65,29 @@ This connector uses username and password authentication to connect to Oracle. T To set up authentication: -- Create an Oracle user with appropriate permissions to access the required tables. -- Provide the username and password in the `configuration.json` file. -- Ensure the user has `SELECT` permissions on the tables that need to be synced. +- Create an Oracle user with appropriate permissions to access the required tables. +- Provide the username and password in the `configuration.json` file. +- Ensure the user has `SELECT` permissions on the tables that need to be synced. ## Pagination -The connector handles pagination by filtering rows with `_build_incremental_query()` and processing results in batches through `_iterate_records()`.The Oracle cursor uses `cursor.fetchmany()` with a configured `arraysize`, so the connector efficiently streams rows without requiring manual page tokens. +The connector handles pagination by filtering rows with `_build_incremental_query()` and processing results in batches through `_iterate_records()`. The Oracle cursor uses `cursor.fetchmany()` with a configured `arraysize`, so the connector efficiently streams rows without requiring manual page tokens. ## Data handling The connector handles data as follows (see the `update()` function): -1. Connects to the Oracle database using the provided configuration. -2. Retrieves the last sync timestamp from state (or uses a default date for the initial sync). -3. Executes a SQL query to fetch records modified after the last sync timestamp. -4. Transforms each database record into the target schema format. -5. Performs upsert operations for each record. -6. Maintains state with the latest processed timestamp. +1. Connects to the Oracle database using the provided configuration. +2. Retrieves the last sync timestamp from state (or uses a default date for the initial sync). +3. Executes a SQL query to fetch records modified after the last sync timestamp. +4. Transforms each database record into the target schema format. +5. Performs upsert operations for each record. +6. Maintains state with the latest processed timestamp. ## Error handling - `validate_configuration()` verifies all required configuration keys before any work begins and raises descriptive errors when values are missing -- `connect_oracle()` catches invalid port values +- `connect_oracle()` catches invalid port values - `update()` function logs connection failures with `log.severe()` before re-raising the exception. - During syncs, the connector wraps Oracle interactions in a `try/finally` block to ensure the database connection is closed - Progress checkpoints are created with `op.checkpoint()` to prevent re-processing if an error occurs mid-batch. @@ -88,9 +98,9 @@ The connector is configured to sync the following table (see the `TABLES` list i ```json { -"table": "FIVETRAN_LOGMINER_TEST", -"primary_key": ["ID"], -"columns": { + "table": "FIVETRAN_LOGMINER_TEST", + "primary_key": ["ID"], + "columns": { "ID": "INT", "NAME": "STRING", "LAST_UPDATED": "UTC_DATETIME" @@ -102,11 +112,11 @@ You can add more tables to the `TABLES` list in `connector.py` as needed. ## Additional files -This connector doesn’t require any additional files, apart from the standard ones listed below: +This connector doesn't require any additional files, apart from the standard ones listed below: -- `connector.py` contains the `update()` and `schema()` implementations for this example. -- `configuration.json` provides a placeholder configuration you can copy and fill with your Oracle credentials. -- `requirements.txt` declares the Oracle driver dependency used at runtime. +- `connector.py` contains the `update()` and `schema()` implementations for this example. +- `configuration.json` provides a placeholder configuration you can copy and fill with your Oracle credentials. +- `requirements.txt` declares the Oracle driver dependency used at runtime. ## Additional considerations diff --git a/betterstack/README.md b/betterstack/README.md index e59c12f..d32171d 100644 --- a/betterstack/README.md +++ b/betterstack/README.md @@ -16,6 +16,16 @@ This connector syncs uptime monitoring data from Better Stack's Uptime API. It r Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template betterstack +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Incremental syncing based on `updated_at` timestamps for all endpoints @@ -35,13 +45,13 @@ The connector requires the following configuration parameter: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector uses only the standard library and SDK-provided packages. No additional dependencies are required in `requirements.txt`. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -91,7 +101,7 @@ The connector implements comprehensive error handling (refer to the `make_api_re The connector creates the following tables (refer to the `schema()` function): -| Table Name | Primary Key | Description | +| Table name | Primary key | Description | |------------|-------------|-------------| | `MONITORS` | `id` | Website/API uptime monitors with configuration including URL, monitor type, check frequency, regions, and status | | `STATUS_PAGES` | `id` | Public-facing status pages with company info, subdomain, custom domain, theme, and display settings | diff --git a/cassandra/README.md b/cassandra/README.md index 0a8221c..76b95b3 100644 --- a/cassandra/README.md +++ b/cassandra/README.md @@ -1,23 +1,32 @@ # Cassandra Database Example -## Connector Overview +## Connector overview -This connector integrates Cassandra databases with Fivetran, syncing data from Cassandra clusters to your destination. It connects to a Cassandra instance, efficiently retrieves data using pagination, and handles incremental updates based on the `created_at` timestamp column. - -The connector is designed to handle large datasets efficiently through streaming and pagination techniques, making it suitable for production environments with significant data volumes. It includes functionality for creating test environments with dummy data for development and testing purposes. +This connector integrates Cassandra databases with Fivetran, syncing data from Cassandra clusters to your destination. It connects to a Cassandra instance, efficiently retrieves data using pagination, and handles incremental updates based on the `created_at` timestamp column. +The connector is designed to handle large datasets efficiently through streaming and pagination techniques, making it suitable for production environments with significant data volumes. It includes functionality for creating test environments with dummy data for development and testing purposes. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +## Getting started -## Getting Started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template cassandra +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -28,12 +37,11 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Support for large datasets through pagination techniques - Detailed logging for monitoring and troubleshooting - -## Configuration File +## Configuration file The connector requires the following configuration parameters: -``` +```json { "hostname": "", "username": "", @@ -49,10 +57,9 @@ The connector requires the following configuration parameters: - keyspace: The Cassandra keyspace to connect to - port: The port number for the Cassandra server -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - -## Requirements File +## Requirements file The connector requires the Cassandra Python driver and dateutil for timestamp parsing: @@ -61,29 +68,26 @@ cassandra-driver python-dateutil ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector uses PlainTextAuthProvider for authentication with Cassandra. Provide the following credentials in the configuration: +The connector uses PlainTextAuthProvider for authentication with Cassandra. Provide the following credentials in the configuration: - username: A valid Cassandra user with read permissions on the specified keyspace - password: The corresponding password for the user - ## Pagination -The connector implements efficient pagination when retrieving data from Cassandra: +The connector implements efficient pagination when retrieving data from Cassandra: - Uses Cassandra's native pagination capabilities with the fetch_size parameter - Default page size is set to 100 records but can be adjusted - Performs upserts one record at a time, avoiding excessive memory usage - Handles checkpointing every 1000 records to maintain state during long-running syncs +## Data handling -## Data Handling - -The connector processes data with the following approach: +The connector processes data with the following approach: - Connects to the specified Cassandra keyspace - Retrieves records incrementally based on the `created_at` timestamp - Transforms Cassandra row objects into dictionaries for Fivetran @@ -95,10 +99,9 @@ The connector processes data with the following approach: - name (text → STRING) - created_at (timestamp → UTC_DATETIME) +## Error handling -## Error Handling - -The connector implements the following error handling strategies: +The connector implements the following error handling strategies: - Validates configuration parameters before attempting connection - Provides detailed error messages for connection failures - Handles timezone-related errors by ensuring consistent timezone awareness @@ -106,10 +109,10 @@ The connector implements the following error handling strategies: - Gracefully handles pagination issues that may occur with large datasets - Implements regular checkpointing to minimize data loss in case of failures - ## Tables created ### `SAMPLE_TABLE` + The `SAMPLE_TABLE` contains sample data from the Cassandra database. The schema for the table is defined as follows: ```json @@ -123,12 +126,10 @@ The `SAMPLE_TABLE` contains sample data from the Cassandra database. The schema } ``` - -## Additional Files +## Additional files - `adding_dummy_data_to_cassandra.py`: This python file contains functions to add dummy data to the Cassandra database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Cassandra table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Cassandra database. - -## Additional Considerations +## Additional considerations The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/checkly/README.md b/checkly/README.md index ec0c183..4c995bb 100644 --- a/checkly/README.md +++ b/checkly/README.md @@ -1,22 +1,35 @@ # Checkly API Connector Example ## Connector overview + The [Checkly](https://www.checklyhq.com/) custom connector for Fivetran fetches monitoring check data and performance analytics from the Checkly API and syncs it to your destination. This connector supports multiple endpoints including check configurations and browser analytics. The connector implements Bearer token authentication, handles pagination automatically, and separates aggregated and non-aggregated analytics data, following Fivetran best practices for reliability, security, and maintainability. - ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template checkly +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Fetches all check configurations from Checkly API via `/v1/checks` endpoint - Supports both API and Browser check types with full metadata - Automatically fetches analytics data for Browser checks via `/v1/analytics/browser-checks/{check_id}` endpoint @@ -27,6 +40,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Comprehensive error handling with graceful degradation for failed analytics requests ## Configuration file + The connector requires the following configuration parameters in the configuration.json file. This configuration is uploaded to Fivetran and defines how the connector authenticates with and queries the Checkly API. ```json @@ -38,31 +52,34 @@ The connector requires the following configuration parameters in the configurati } ``` -**Required Parameters:** +Required parameters: - `api_key`: Your Checkly API key with read permissions for accessing checks and analytics data - `account_id`: Your Checkly account identifier for API authentication -**Optional Parameters:** +Optional parameters: - `aggregation_interval`: Time interval for aggregating analytics data in minutes (default: 60). Must be a positive integer between 1 and 43200 - `quick_range`: Time range for analytics data collection (default: `last24Hours`) - - Available options: `last24Hours`, `last7Days`, `last30Days`, `thisWeek`, `thisMonth`, `lastWeek`, `lastMonth` + - Available options: `last24Hours`, `last7Days`, `last30Days`, `thisWeek`, `thisMonth`, `lastWeek`, `lastMonth` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector example uses standard libraries provided by Python and does not require any additional packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses Bearer Token authentication with the Checkly API. Authentication is handled through the `validate_configuration` function which validates credentials before any API calls are made. You'll need to obtain the following credentials: -1. **API Key**: A Checkly API key with read permissions for checks and analytics endpoints -2. **Account ID**: Your Checkly account identifier for proper API access +1. API key: A Checkly API key with read permissions for checks and analytics endpoints +2. Account ID: Your Checkly account identifier for proper API access + +### Steps to obtain credentials -#### Steps to obtain credentials 1. Log into your Checkly dashboard at https://app.checklyhq.com/. 2. Navigate to **Account Settings > API Keys**. 3. Create a new API key with appropriate read permissions. @@ -70,6 +87,7 @@ You'll need to obtain the following credentials: 5. Add both values to your `configuration.json` file. ## Pagination + The connector implements page-based pagination using the Checkly API's `limit` and `page` parameters. Pagination is handled automatically in the `get_checks_data` function which: - Fetches data in batches of 100 records per page (defined by `PAGE_SIZE` constant) @@ -80,52 +98,59 @@ The connector implements page-based pagination using the Checkly API's `limit` a The pagination loop automatically increments the page number and constructs the appropriate API URL for each request until all check data is retrieved. ## Data handling + The connector processes Checkly data through several transformation steps implemented across multiple functions: -- **Check Data Retrieval**: Fetches check configurations from `/v1/checks` endpoint with automatic pagination -- **Data Flattening**: Converts nested JSON objects into flat SQL-compatible structures -- **Analytics Processing**: For browser checks, fetches analytics data from `/v1/analytics/browser-checks/{check_id}` endpoint -- **Metrics Separation**: Separates analytics into aggregated and non-aggregated metrics tables using predefined metric constants (`AGGREGATED_METRICS` and `NON_AGGREGATED_METRICS`) -- **Array Handling**: Converts string arrays to comma-separated values and complex arrays to JSON strings for optimal database storage -- **Rate Limiting**: Implements configurable delays between API calls to respect Checkly's rate limits +- Check data retrieval: Fetches check configurations from `/v1/checks` endpoint with automatic pagination +- Data flattening: Converts nested JSON objects into flat SQL-compatible structures +- Analytics processing: For browser checks, fetches analytics data from `/v1/analytics/browser-checks/{check_id}` endpoint +- Metrics separation: Separates analytics into aggregated and non-aggregated metrics tables using predefined metric constants (`AGGREGATED_METRICS` and `NON_AGGREGATED_METRICS`) +- Array handling: Converts string arrays to comma-separated values and complex arrays to JSON strings for optimal database storage +- Rate limiting: Implements configurable delays between API calls to respect Checkly's rate limits ## Error handling + The connector implements comprehensive error handling strategies across multiple functions: -- **API Rate Limiting**: Automatically handles HTTP 429 responses with retry delays -- **Configuration Validation**: Validates all required configuration parameters and optional parameters with sensible defaults before any API calls -- **Exception Handling**: Catches and logs specific `requests.exceptions.RequestException` while continuing processing where possible -- **Graceful Degradation**: If analytics data fails for a specific check, the connector logs the error and continues processing other checks rather than failing the entire sync -- **Request Timeouts**: Implements 30-second timeouts for all API requests to prevent hanging connections -- **Retry Logic**: Built-in retry mechanism specifically for rate-limited requests (HTTP 429 status codes) -- **Validation Logic**: Ensures `quick_range` values are within allowed options and `aggregation_interval` is a positive integer +- API rate limiting: Automatically handles HTTP 429 responses with retry delays +- Configuration validation: Validates all required configuration parameters and optional parameters with sensible defaults before any API calls +- Exception handling: Catches and logs specific `requests.exceptions.RequestException` while continuing processing where possible +- Graceful degradation: If analytics data fails for a specific check, the connector logs the error and continues processing other checks rather than failing the entire sync +- Request timeouts: Implements 30-second timeouts for all API requests to prevent hanging connections +- Retry logic: Built-in retry mechanism specifically for rate-limited requests (HTTP 429 status codes) +- Validation logic: Ensures `quick_range` values are within allowed options and `aggregation_interval` is a positive integer All errors are logged appropriately using the Fivetran SDK logging system while maintaining sync reliability. ## Tables created -The connector creates three main tables in your data warehouse as defined in the `schema` function - -### 1. **checks** -- **Purpose**: Contains all check configurations and metadata from the Checkly API -- **Primary Key**: `id` (check identifier) -- **Content**: Check settings, URLs, schedules, alert configurations, and all flattened check properties -- **Data Source**: `/v1/checks` endpoint with full pagination support - -### 2. **browser_checks_analytics_aggregated** -- **Purpose**: Contains aggregated analytics metrics for browser checks over specified time periods -- **Primary Key**: `_fivetran_id` (hash of all the column values) -- **Content**: Statistical aggregations including averages, percentiles (p50, p90, p95, p99), min/max values, standard deviations for performance metrics -- **Metrics**: Response times, Core Web Vitals (FCP, LCP, CLS, TBT), TTFB, error counts (console, network, user script, document errors) with full statistical breakdowns -- **Data Source**: `/v1/analytics/browser-checks/{check_id}` endpoint using `AGGREGATED_METRICS` constants - -### 3. **browser_checks_analytics_non_aggregated** -- **Purpose**: Contains raw analytics metrics for browser checks as individual data points -- **Primary Key**: `_fivetran_id` (hash of all the column values) -- **Content**: Individual measurement data points for each check execution -- **Metrics**: Raw response times, Core Web Vitals measurements, TTFB values, error counts per execution -- **Data Source**: `/v1/analytics/browser-checks/{check_id}` endpoint using `NON_AGGREGATED_METRICS` constants + +The connector creates three main tables in your data warehouse as defined in the `schema` function. + +### checks + +- Purpose: Contains all check configurations and metadata from the Checkly API +- Primary key: `id` (check identifier) +- Content: Check settings, URLs, schedules, alert configurations, and all flattened check properties +- Data source: `/v1/checks` endpoint with full pagination support + +### browser_checks_analytics_aggregated + +- Purpose: Contains aggregated analytics metrics for browser checks over specified time periods +- Primary key: `_fivetran_id` (hash of all the column values) +- Content: Statistical aggregations including averages, percentiles (p50, p90, p95, p99), min/max values, standard deviations for performance metrics +- Metrics: Response times, Core Web Vitals (FCP, LCP, CLS, TBT), TTFB, error counts (console, network, user script, document errors) with full statistical breakdowns +- Data source: `/v1/analytics/browser-checks/{check_id}` endpoint using `AGGREGATED_METRICS` constants + +### browser_checks_analytics_non_aggregated + +- Purpose: Contains raw analytics metrics for browser checks as individual data points +- Primary key: `_fivetran_id` (hash of all the column values) +- Content: Individual measurement data points for each check execution +- Metrics: Raw response times, Core Web Vitals measurements, TTFB values, error counts per execution +- Data source: `/v1/analytics/browser-checks/{check_id}` endpoint using `NON_AGGREGATED_METRICS` constants All tables use the `op.upsert()` operation to insert or update data, ensuring data consistency across sync runs. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/clerk/README.md b/clerk/README.md index 5a373c4..d995d0f 100644 --- a/clerk/README.md +++ b/clerk/README.md @@ -1,11 +1,13 @@ # Clerk API Connector ## Connector overview + This connector integrates with the Clerk API to synchronize user data into your destination. It fetches user records from Clerk and flattens nested data structures into normalized tables, making it easy to analyze user data in your data warehouse. The connector supports incremental sync using timestamp-based cursors and handles pagination automatically to process large datasets efficiently. -Clerk is a user authentication and management platform that provides APIs for managing users, their email addresses, phone numbers, social accounts, and authentication methods. +Clerk is a user authentication and management platform that provides APIs for managing users, their email addresses, phone numbers, social accounts, and authentication methods. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -13,9 +15,21 @@ Clerk is a user authentication and management platform that provides APIs for ma - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template clerk +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Fetches user data from the Clerk API `/v1/users` endpoint - Incremental sync support using `created_at_after` parameter to fetch only newly created users (defaults to January 1, 1990 for the initial sync) - Automatic pagination using offset-based pagination (limit and offset parameters) @@ -26,6 +40,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Memory-efficient processing using generator functions to avoid loading all the data at once ## Configuration file + The configuration file contains the API key required to authenticate with the Clerk API. ```json @@ -37,14 +52,16 @@ The configuration file contains the API key required to authenticate with the Cl Configuration parameters: - `api_key` (required): Your Clerk API secret key. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector example uses standard libraries provided by Python and does not require any additional packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector uses API key authentication with bearer tokens. The API key is passed in the `Authorization` header as `Bearer `. To obtain your Clerk API key: @@ -84,6 +101,7 @@ Data type inference: - Fivetran automatically infers data types for all other columns based on the data ## Error handling + The connector implements comprehensive error handling with retry logic. Retry strategy: @@ -101,9 +119,10 @@ Error categories: Timeout: All API requests have a 30-second timeout to prevent hanging connections. ## Tables created + The connector creates 8 tables in your destination: -| Table Name | Type | Primary Key | Foreign Key | Description | +| Table name | Type | Primary key | Foreign key | Description | |------------|------|-------------|-------------|-------------| | `USER` | Main table | `id` | - | Contains flattened user profile data including metadata fields. Nested objects like `public_metadata`, `private_metadata`, `unsafe_metadata` are flattened into columns. | | `USER_EMAIL_ADDRESS` | Child table | `id` | `user_id` → `USER.id` | Contains email addresses with verification details. | @@ -115,4 +134,5 @@ The connector creates 8 tables in your destination: | `USER_ENTERPRISE_ACCOUNT` | Child table | `id` | `user_id` → `USER.id` | Contains enterprise account connections and SSO details. | ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/clickhouse/README.md b/clickhouse/README.md index 9c4df42..146335a 100644 --- a/clickhouse/README.md +++ b/clickhouse/README.md @@ -1,21 +1,30 @@ # ClickHouse Database Example -## Connector Overview +## Connector overview This connector shows how to pull data from ClickHouse databases. It connects to a specified ClickHouse instance, extracts data from tables, and syncs it to your destination. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +## Getting started -## Getting Started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template clickhouse +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -24,12 +33,11 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Tracks sync progress using state and checkpointing - Supports incremental syncs +## Configuration file -## Configuration File - -The connector requires the following configuration parameters: +The connector requires the following configuration parameters: -``` +```json { "hostname": "", "username": "", @@ -43,10 +51,9 @@ The connector requires the following configuration parameters: - password: Password for authentication - database: The ClickHouse database to connect to -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - -## Requirements File +## Requirements file The connector requires the clickhouse_connect Python library for connecting to ClickHouse: @@ -54,44 +61,40 @@ The connector requires the clickhouse_connect Python library for connecting to C clickhouse_connect==0.8.17 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector uses basic username and password authentication to connect to ClickHouse. These credentials are specified in the configuration file. The connector uses TLS/SSL for secure connections (indicated by the secure=True parameter in the client configuration). To obtain credentials: +The connector uses basic username and password authentication to connect to ClickHouse. These credentials are specified in the configuration file. The connector uses TLS/SSL for secure connections (indicated by the secure=True parameter in the client configuration). To obtain credentials: 1. Create an account in the ClickHouse Cloud console 2. Click on the "Connect" button for your cluster 3. Choose the python Language client to get the connection details 4. Use the connection details to fill in the configuration file - ## Pagination -The connector uses ClickHouse's streaming query capabilities to efficiently process large datasets without loading all data into memory. This is implemented using the `client.query_rows_stream()` method which upserts rows as they are retrieved from the database. +The connector uses ClickHouse's streaming query capabilities to efficiently process large datasets without loading all data into memory. This is implemented using the `client.query_rows_stream()` method which upserts rows as they are retrieved from the database. +## Data handling -## Data Handling - -The connector: +The connector: - Establishes a connection to the ClickHouse database. - Inserts data into dummy table for testing purposes. - Retrieves column metadata for the specified table. - Executes queries with incremental filtering based on a timestamp column (`created_at`). - Upserts data into destination table. +## Error handling -## Error Handling - -The connector implements error handling for: +The connector implements error handling for: - Connection failures: Raises informative exceptions if the connector fails to connect to ClickHouse - Configuration validation: Checks for required configuration parameters before attempting to connect - ## Tables created ### `TEST_TABLE` + The `TEST_TABLE` table contains dummy data with the following schema: ```json @@ -105,12 +108,10 @@ The `TEST_TABLE` table contains dummy data with the following schema: } ``` - -## Additional Files +## Additional files `clickhouse_dummy_data_generator.py`: This python file contains functions to add dummy data to the Clickhouse database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Clickhouse table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Clickhouse database. - -## Additional Considerations +## Additional considerations The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/commonpaper/README.md b/commonpaper/README.md index 30530a7..e8b1dd2 100644 --- a/commonpaper/README.md +++ b/commonpaper/README.md @@ -10,23 +10,33 @@ This connector fetches and syncs agreement data from the Common Paper API. It pr ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template commonpaper +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -* Fetches agreement data from Common Paper API -* Supports incremental syncs using updated_at timestamps -* Handles list values by converting them to JSON strings -* Maintains sync state for resumable operations -* Simple authentication using API key +- Fetches agreement data from Common Paper API +- Supports incremental syncs using updated_at timestamps +- Handles list values by converting them to JSON strings +- Maintains sync state for resumable operations +- Simple authentication using API key ## Configuration file @@ -43,13 +53,13 @@ Configuration parameters: - `api_key`: Your Common Paper API key - `initial_sync_timestamp`: ISO format timestamp for the initial sync (optional, defaults to current time) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file The connector does not use any additional Python packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -72,15 +82,16 @@ The connector implements basic error handling: - Logs errors using the Fivetran logging system - Uses checkpoints to ensure sync progress is saved -## Tables Created +## Tables created The connector creates a single table: ### agreements + Primary key: `id` This table contains all agreement data from Common Paper, with nested structures stored as JSON strings. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 54020b9..6bccdcc 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -2,24 +2,32 @@ ## Connector overview - -This connector example demonstrates how to sync data from Couchbase Capella using the Connector SDK. It connects to a Couchbase Capella instance, executes SQL++ (N1QL) queries to fetch data from a specific bucket, scope, and collection, and efficiently streams the data to destination table while implementing best practices for handling large datasets. -This connector is built for Couchbase Capella. It supports buckets using either the Magma or Couchstore storage engines. +This connector example demonstrates how to sync data from Couchbase Capella using the Connector SDK. It connects to a Couchbase Capella instance, executes SQL++ (N1QL) queries to fetch data from a specific bucket, scope, and collection, and efficiently streams the data to destination table while implementing best practices for handling large datasets. +This connector is built for Couchbase Capella. It supports buckets using either the Magma or Couchstore storage engines. > For syncing data from a Magma bucket on self-managed Couchbase Server, please refer to the [Couchbase Magma connector example](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/examples/source_examples/couchbase_magma) - ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template couchbase_capella +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -33,7 +41,7 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se The connector requires the following configuration parameters to connect to your Couchbase Capella instance: -``` +```json { "username": "YOUR_COUCHBASE_USERNAME", "password": "YOUR_COUCHBASE_PASSWORD", @@ -44,7 +52,7 @@ The connector requires the following configuration parameters to connect to your } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -54,7 +62,7 @@ The connector requires the Couchbase Python SDK: couchbase==4.3.6 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -62,7 +70,7 @@ The connector authenticates with Couchbase using a username and password authent ## Data handling -The connector handles data processing through the following steps: +The connector handles data processing through the following steps: - Establishes a connection to the Couchbase cluster using the provided credentials. - Executes a `SQL++` query against the specified collection. - Streams the query results to avoid loading the entire dataset into memory @@ -71,25 +79,25 @@ The connector handles data processing through the following steps: ## Error handling -The connector implements error handling in several critical functions: +The connector implements error handling in several critical functions: - In `create_couchbase_client`: Catches any exception during cluster connection and raises a meaningful error message - In `execute_query_and_upsert`: Catches exceptions during query execution and data processing, raising descriptive runtime errors -## Tables Created +## Tables created The `schema()` function defines the structure of the destination table: -``` +```json { "table": "airline_table", "primary_key": ["id"], "columns": { - "id": "INT", - }, + "id": "INT" + } } ``` -The table contains `airline` information from the Couchbase `travel-sample` bucket. +The table contains `airline` information from the Couchbase `travel-sample` bucket. ## Additional considerations diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index 6083673..981a238 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -1,21 +1,32 @@ # Couchbase Magma Connector Example -This connector example demonstrates how to sync data from a self-managed Couchbase Server (local, on-premises, or self-hosted in the cloud) using the Magma storage engine with the Connector SDK. + +This connector example demonstrates how to sync data from a self-managed Couchbase Server (local, on-premises, or self-hosted in the cloud) using the Magma storage engine with the Connector SDK. It connects to a Couchbase instance, runs SQL++ (N1QL) queries to retrieve data from a specified Magma bucket, scope, and collection, and streams the results efficiently to a destination table—following best practices for handling large datasets. ->For syncing data from a Magma bucket on Couchbase Capella, please refer to the [Couchbase Capella connector example](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/examples/source_examples/couchbase_capella). +> For syncing data from a Magma bucket on Couchbase Capella, please refer to the [Couchbase Capella connector example](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/examples/source_examples/couchbase_capella). ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template couchbase_magma +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Connect to Couchbase instances using secure authentication @@ -27,7 +38,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co The connector requires the following configuration parameters to connect to your Couchbase instance: -``` +```json { "username": "YOUR_COUCHBASE_USERNAME", "password": "YOUR_COUCHBASE_PASSWORD", @@ -35,12 +46,12 @@ The connector requires the following configuration parameters to connect to your "bucket_name": "YOUR_COUCHBASE_BUCKET_NAME", "scope": "YOUR_COUCHBASE_SCOPE_NAME", "collection": "YOUR_COUCHBASE_COLLECTION_NAME", - "use_tls": "", # Optional, defaults to false, required if couchbase instance requires TLS - "cert_path": "PATH_TO_YOUR_TLS_CERTIFICATE" # Optional, required if use_tls is true + "use_tls": "", + "cert_path": "PATH_TO_YOUR_TLS_CERTIFICATE" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -50,7 +61,7 @@ The connector requires the Couchbase Python SDK: couchbase==4.3.6 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -58,7 +69,7 @@ The connector authenticates with Couchbase using a username and password authent ## Data handling -The connector handles data processing through the following steps: +The connector handles data processing through the following steps: - Establishes a connection to the Couchbase cluster using the provided credentials. - Executes a `SQL++` query against the specified collection. - Streams the query results to avoid loading the entire dataset into memory @@ -67,7 +78,7 @@ The connector handles data processing through the following steps: ## Error handling -The connector implements error handling in several critical functions: +The connector implements error handling in several critical functions: - In `create_couchbase_client`: Catches any exception during cluster connection and raises a meaningful error message. - In `execute_query_and_upsert`: Catches exceptions during query execution and data processing, raising descriptive runtime errors. @@ -75,18 +86,18 @@ The connector implements error handling in several critical functions: The `schema()` function defines the structure of the destination table: -``` +```json { "table": "airline_table", "primary_key": ["id"], "columns": { "id": "INT", - "created_at": "UTC_DATETIME", - }, + "created_at": "UTC_DATETIME" + } } ``` -The table contains `airline` information from the Couchbase `travel-sample` bucket. +The table contains `airline` information from the Couchbase `travel-sample` bucket. ## Additional considerations diff --git a/courier/README.md b/courier/README.md index 68faac8..b446cd5 100644 --- a/courier/README.md +++ b/courier/README.md @@ -16,6 +16,16 @@ This connector fetches data from the Courier API, a multi-channel notification p Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template courier +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Syncs audit events with timestamp-based incremental updates for compliance tracking @@ -38,13 +48,13 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require any additional packages beyond the pre-installed libraries in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -64,7 +74,7 @@ The connector implements different pagination strategies based on the endpoint ( - Cursor-based pagination – Used for the audiences, messages, and notifications endpoints. The connector processes pages sequentially using cursor tokens returned in the API response and checkpoints after each page (refer to the `fetch_audiences()`, `fetch_messages()`, and `fetch_notifications()` functions). - Next URL-based pagination – Used for the tenants endpoint. The connector follows the `next_url` field in the API response to retrieve subsequent pages (refer to the `fetch_tenants()` function). - Offset pagination – Used for lists, brands, and audit events endpoints. These endpoints return paginated results with a `more` indicator in the paging object. - + ## Data handling The connector processes data as follows: diff --git a/customer_thermometer/README.md b/customer_thermometer/README.md index 9c95a27..385b24e 100644 --- a/customer_thermometer/README.md +++ b/customer_thermometer/README.md @@ -2,7 +2,7 @@ ## Connector overview -The [Customer Thermometer](https://www.customerthermometer.com/) custom Fivetran connector fetches customer feedback data from the Customer Thermometer API and syncs it to your destination. This connector supports multiple endpoints including comments, blast results, recipient lists, thermometers, and feedback metrics. +The [Customer Thermometer](https://www.customerthermometer.com/) custom Fivetran connector fetches customer feedback data from the Customer Thermometer API and syncs it to your destination. This connector supports multiple endpoints including comments, blast results, recipient lists, thermometers, and feedback metrics. The connector implements API key authentication, parses XML responses, and is stateless, following Fivetran best practices for reliability, security, and maintainability. @@ -18,6 +18,16 @@ The connector implements API key authentication, parses XML responses, and is st Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template customer_thermometer +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features The connector supports the following features: @@ -44,13 +54,13 @@ The connector requires API key authentication for the Customer Thermometer API. - `api_key`: (Required) Your Customer Thermometer API key - `from_date`: (Optional) Start date for initial data retrieval in YYYY-MM-DD format. If not provided, incremental sync will start from EPOCH (1970-01-01) for the initial sync, then use state-based incremental sync for subsequent runs. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector example uses standard libraries provided by Python and does not require any additional packages. -Note: The `fivetran_connector_sdk` and `requests` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -107,4 +117,4 @@ All tables include flattened versions of complex nested objects where applicable ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/data_camp/README.md b/data_camp/README.md index 098bc02..28523f4 100644 --- a/data_camp/README.md +++ b/data_camp/README.md @@ -18,19 +18,29 @@ The connector implements bearer token authentication with comprehensive retry lo Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template data_camp +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features The connector supports the following features: -- **Multiple content types**: Supports courses, projects, tracks, practices, assessments, and custom tracks -- **Bearer token authentication**: Secure authentication using DataCamp API access tokens -- **Flexible server configuration**: Optional base_url parameter supports live production and mock/test environments -- **Retry logic with exponential backoff**: Automatic retries for failed API requests with exponential backoff -- **Data flattening**: Comprehensive flattening of nested objects and arrays into relational structures -- **Breakout tables**: Separate tables for nested relationships (chapters, topics, content) -- **State management**: Maintains sync state with checkpointing for reliable syncs -- **Error handling**: Individual record error handling without stopping entire sync -- **Comprehensive logging**: Detailed logging for troubleshooting and monitoring with retry attempt tracking +- Multiple content types: Supports courses, projects, tracks, practices, assessments, and custom tracks +- Bearer token authentication: Secure authentication using DataCamp API access tokens +- Flexible server configuration: Optional base_url parameter supports live production and mock/test environments +- Retry logic with exponential backoff: Automatic retries for failed API requests with exponential backoff +- Data flattening: Comprehensive flattening of nested objects and arrays into relational structures +- Breakout tables: Separate tables for nested relationships (chapters, topics, content) +- State management: Maintains sync state with checkpointing for reliable syncs +- Error handling: Individual record error handling without stopping entire sync +- Comprehensive logging: Detailed logging for troubleshooting and monitoring with retry attempt tracking ## Configuration file @@ -50,13 +60,13 @@ The connector requires bearer token authentication for the DataCamp LMS Catalog - Leave blank to use the default DataCamp production server: `https://lms-catalog-api.datacamp.com` - You can specify a URL for testing or mock environments. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector example uses the standard libraries provided by Python and does not require any additional packages. -Note: The `fivetran_connector_sdk` and `requests` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -97,9 +107,9 @@ The connector processes JSON data from the DataCamp API and transforms it into s - Processes each endpoint with specific configuration parameters The `update` function processes each content type with a specific configuration: -- **Endpoint configuration**: URL, table name, and primary key information from `__ENDPOINTS` constant -- **Flatten parameters**: Content-specific flattening options (skip_keys, flatten_topic, flatten_licenses, etc.) -- **Breakout configuration**: Optional child table specifications for nested arrays +- Endpoint configuration: URL, table name, and primary key information from `__ENDPOINTS` constant +- Flatten parameters: Content-specific flattening options (skip_keys, flatten_topic, flatten_licenses, etc.) +- Breakout configuration: Optional child table specifications for nested arrays Data is delivered to Fivetran using upsert operations for each record, ensuring data consistency and enabling incremental updates. @@ -109,7 +119,7 @@ The connector implements comprehensive error handling strategies following Fivet - Failed records don't stop the entire sync process - HTTP errors are caught and logged with severity levels -- Automatic retries with exponential backoff +- Automatic retries with exponential backoff - Configurable timeout (30 seconds default) for API requests to prevent hanging - Uses Fivetran's logging framework with retry attempt tracking - Checkpoints progress after each endpoint to enable recovery diff --git a/dgraph/README.md b/dgraph/README.md index f69c855..336fa48 100644 --- a/dgraph/README.md +++ b/dgraph/README.md @@ -22,6 +22,16 @@ The connector showcases best practices for syncing graph data: Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template dgraph +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - True incremental sync using timestamp-based filtering on updatedAt/createdAt fields @@ -49,13 +59,13 @@ Configuration parameters: - `dgraph_url` (required) - The base URL of your Dgraph instance (e.g., `http://localhost:8080` or `https://your-instance.dgraph.io`). Must start with `http://` or `https://`. - `api_key` (required) - Authentication token for accessing the Dgraph API. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector uses only the Python standard library and SDK-provided packages. No additional dependencies are required beyond what is pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -127,7 +137,7 @@ All errors are logged using the SDK logging framework with appropriate severity The connector creates the following tables in the destination warehouse. The SDK automatically infers column data types from the synced data. -| Table | Primary Key | Description | +| Table | Primary key | Description | |-------|-------------|-------------| | `product` | `product_id` | Product catalog with SKU, name, price, inventory status, and foreign key to category. Includes counts of related attributes and products. Contains timestamps for creation, updates, and sync tracking. | | `category` | `category_id` | Product categories with hierarchical parent relationships. Includes name, description, and timestamps for creation, updates, and sync tracking. | diff --git a/discord/README.md b/discord/README.md index f67317f..edd04f4 100644 --- a/discord/README.md +++ b/discord/README.md @@ -15,7 +15,7 @@ This connector is particularly well-suited for the following AI/ML use cases: ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) @@ -25,6 +25,16 @@ This connector is particularly well-suited for the following AI/ML use cases: Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template discord +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Multi-guild support: Automatically discovers and processes all guilds the bot has access to (refer to the `fetch_user_guilds` function) @@ -57,17 +67,17 @@ The connector requires the following configuration parameters in `configuration. - `sync_messages` (optional): Specifies whether to sync message data (default: `true`) - `message_limit` (optional): Maximum messages per channel to sync (default: `1000`) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file -The `requirements.txt` file specifies the Python libraries required by the connector. This connector uses only the standard library and pre-installed packages +The `requirements.txt` file specifies the Python libraries required by the connector. This connector uses only the standard library and pre-installed packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -This connector uses Discord bot token authentication (Refer to `validate_configuration` function, and `_normalize_bot_token` function). To set up authentication: +This connector uses Discord bot token authentication (refer to `validate_configuration` function, and `_normalize_bot_token` function). To set up authentication: 1. Create a Discord application: - Go to [Discord Developer Portal](https://discord.com/developers/applications). @@ -75,8 +85,8 @@ This connector uses Discord bot token authentication (Refer to `validate_configu - Navigate to the Bot section in the left sidebar. 2. Create a bot: - Click Add Bot if no bot exists. - - Make a note of the bot token. - > **Warning:** The bot token is a sensitive credential that grants access to your Discord bot. **Do not share this token with anyone or post it publicly.** Keep it secure and never check it into version control or expose it in public repositories. + - Make a note of the bot token. + > Note: The bot token is a sensitive credential that grants access to your Discord bot. Do not share this token with anyone or post it publicly. Keep it secure and never check it into version control or expose it in public repositories. 3. Set bot permissions: - In the Bot section, scroll down to Privileged Gateway Intents. - Enable Server Members Intent (required for member data). @@ -89,7 +99,7 @@ This connector uses Discord bot token authentication (Refer to `validate_configu ## Pagination -The connector handles pagination automatically for all Discord API endpoints (Refer to `fetch_channel_messages` function): +The connector handles pagination automatically for all Discord API endpoints (refer to `fetch_channel_messages` function): - Channels: Fetches all channels in a single request - Members: Uses Discord's member endpoint with 1000 member limit per request @@ -99,7 +109,7 @@ The connector processes data in batches and checkpoints progress every 50 record ## Data handling -The connector processes and normalizes Discord data for optimal analysis (Refer to `process_guild_data`, `process_channel_data`, `process_member_data`, and `process_message_data` functions): +The connector processes and normalizes Discord data for optimal analysis (refer to `process_guild_data`, `process_channel_data`, `process_member_data`, and `process_message_data` functions): - JSON serialization: Complex objects (mentions, attachments, embeds) are stored as JSON strings - Timestamp normalization: All timestamps are converted to ISO format with UTC timezone @@ -107,7 +117,7 @@ The connector processes and normalizes Discord data for optimal analysis (Refer - Null handling: Gracefully handles missing or null values from the API - Schema evolution: Automatically adapts to new Discord API fields -### Data Processing Pipeline +### Data processing pipeline 1. Fetch: Retrieve data from Discord API with rate limit handling 2. Normalize: Convert Discord data format to Fivetran schema @@ -125,7 +135,7 @@ The connector implements comprehensive error handling strategies (refer to `make - Data validation: Graceful handling of malformed API responses - State recovery: Checkpoint-based state management for sync resumption -### Error Types Handled +### Error types handled - 429 rate limited: Waits for retry-after header duration - 500-504 server errors: Exponential backoff retry strategy @@ -138,18 +148,22 @@ The connector implements comprehensive error handling strategies (refer to `make The connector creates four main tables with the following structure: ### GUILD + Primary key: `id` Contains complete Discord server information including settings, features, member counts, and metadata. -### CHANNEL +### CHANNEL + Primary key: `id` All channel types (text, voice, category, etc.) with permissions, settings, and configuration. ### MEMBER + Primary key: `user_id` and `guild_id` (composite primary key) User profiles with roles, join dates, activity status, and server-specific information. ### MESSAGE + Primary key: `id` Complete message data including content, attachments, embeds, reactions, and metadata. diff --git a/documentdb/README.md b/documentdb/README.md index 72d0342..ab465e8 100644 --- a/documentdb/README.md +++ b/documentdb/README.md @@ -2,17 +2,17 @@ ## Connector overview -This connector integrates AWS DocumentDB with Fivetran, syncing data from DocumentDB collections to your destination. It connects to a DocumentDB cluster, efficiently retrieves data using pagination, and handles incremental updates based on the `updated_at` timestamp field. +This connector integrates AWS DocumentDB with Fivetran, syncing data from DocumentDB collections to your destination. It connects to a DocumentDB cluster, efficiently retrieves data using pagination, and handles incremental updates based on the `updated_at` timestamp field. -Note: Fivetran already provides a native Java-based DocumentDB connector that supports SaaS deployment models. This Python-based connector is specifically intended for Hybrid Deployment (HD) scenarios, as the native connector does not support Hybrid Deployment. For SaaS deployments, we recommend using the native DocumentDB connector available at https://fivetran.com/docs/connectors/databases/documentdb. +> Note: Fivetran already provides a native Java-based DocumentDB connector that supports SaaS deployment models. This Python-based connector is specifically intended for Hybrid Deployment (HD) scenarios, as the native connector does not support Hybrid Deployment. For SaaS deployments, we recommend using the native DocumentDB connector available at https://fivetran.com/docs/connectors/databases/documentdb. -Important Network Consideration: DocumentDB clusters are typically deployed in private subnets and do not allow direct connections from outside the VPC. If you need to connect from outside the VPC (such as from your local development environment), you can set up an EC2 instance in the same VPC as your DocumentDB cluster using the [AWS automatic connectivity guide](https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-ec2-auto.html#auto-connect-ec2.process). Once the EC2 instance is set up, you can create a TCP tunnel using socat: `socat TCP-LISTEN:27017,fork TCP::27017` to forward traffic from your local machine to the DocumentDB cluster through the EC2 instance. +> Note: DocumentDB clusters are typically deployed in private subnets and do not allow direct connections from outside the VPC. If you need to connect from outside the VPC (such as from your local development environment), you can set up an EC2 instance in the same VPC as your DocumentDB cluster using the [AWS automatic connectivity guide](https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-ec2-auto.html#auto-connect-ec2.process). Once the EC2 instance is set up, you can create a TCP tunnel using socat: `socat TCP-LISTEN:27017,fork TCP::27017` to forward traffic from your local machine to the DocumentDB cluster through the EC2 instance. The connector is designed to handle large datasets efficiently through streaming and pagination techniques, making it suitable for production environments with significant data volumes. It supports MongoDB-compatible operations since DocumentDB is MongoDB-compatible. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) @@ -20,7 +20,17 @@ The connector is designed to handle large datasets efficiently through streaming ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template documentdb +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -36,7 +46,7 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se The connector requires the following configuration parameters: -``` +```json { "hostname": "", "username": "", @@ -52,7 +62,7 @@ The connector requires the following configuration parameters: - database: The DocumentDB database name to connect to - port: The port number for the DocumentDB cluster (default: 27017) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -63,11 +73,11 @@ pymongo python-dateutil ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector uses SSL authentication with DocumentDB. Provide the following credentials in the configuration: +The connector uses SSL authentication with DocumentDB. Provide the following credentials in the configuration: - username: A valid DocumentDB user with read permissions on the specified database - password: The corresponding password for the user @@ -76,7 +86,7 @@ The connector automatically configures SSL settings required for DocumentDB conn ## Pagination -The connector implements efficient pagination when retrieving data from DocumentDB: +The connector implements efficient pagination when retrieving data from DocumentDB: - Uses MongoDB cursor-based pagination with batch_size parameter - Default batch size is set to 100 documents but can be adjusted - Performs upserts one document at a time, avoiding excessive memory usage @@ -84,7 +94,7 @@ The connector implements efficient pagination when retrieving data from Document ## Data handling -The connector processes data with the following approach: +The connector processes data with the following approach: - Connects to the specified DocumentDB database - Retrieves documents incrementally based on the `updated_at` timestamp - Transforms DocumentDB documents into dictionaries for Fivetran @@ -102,7 +112,7 @@ The connector processes data with the following approach: ## Error handling -The connector implements the following error handling strategies: +The connector implements the following error handling strategies: - Validates configuration parameters before attempting connection - Provides detailed error messages for connection failures - Handles timezone-related errors by ensuring consistent timezone awareness @@ -116,22 +126,23 @@ The connector implements the following error handling strategies: This connector creates the following tables in your destination: ### USERS -- Primary Key: `_id` -- Columns: + +- Primary key: `_id` +- Columns: - `_id` (STRING): Document unique identifier - `created_at` (UTC_DATETIME): Document creation timestamp - `updated_at` (UTC_DATETIME): Document last update timestamp - `metadata` (JSON): Additional user metadata as JSON object ### ORDERS -- Primary Key: `_id` + +- Primary key: `_id` - Columns: - `_id` (STRING): Document unique identifier - `created_at` (UTC_DATETIME): Order creation timestamp - `updated_at` (UTC_DATETIME): Order last update timestamp - `items` (JSON): Order items as JSON array - ## Prerequisites This connector requires that the following prerequisites exist in your DocumentDB instance: @@ -145,9 +156,9 @@ This connector requires that the following prerequisites exist in your DocumentD Example document structure: ```json { - "_id": ObjectId("..."), - "created_at": ISODate("2023-01-01T00:00:00Z"), - "updated_at": ISODate("2023-01-01T00:00:00Z"), + "_id": "ObjectId(...)", + "created_at": "ISODate(2023-01-01T00:00:00Z)", + "updated_at": "ISODate(2023-01-01T00:00:00Z)", "metadata": {"source": "web"} } ``` @@ -156,7 +167,7 @@ Example document structure: The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. -## DocumentDB-specific notes +### DocumentDB-specific notes - DocumentDB is MongoDB-compatible, so this connector uses the PyMongo driver - SSL is required for DocumentDB connections and is automatically configured diff --git a/docusign/README.md b/docusign/README.md index 910ba3b..17088d4 100644 --- a/docusign/README.md +++ b/docusign/README.md @@ -4,29 +4,38 @@ This connector extracts data from the Docusign eSignature API. It is designed to sync key objects related to the electronic signature process, including envelopes, recipients, documents (including their binary content), audit events, and templates. The extracted data can be used in a destination to enable analytics for Sales, Legal, Operations, and other teams tracking contract lifecycles, signature status, and compliance. -## Contributor +## Accreditation This example was contributed by [Arpit Kumar Khatri](https://github.com/ArpitKhatri1). The connector was developed as part of the [AI Accelerate](https://ai-accelerate.devpost.com/) hackathon. ## Requirements - - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86\_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86\_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. -## Features +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template docusign +``` - - Extracts core Docusign resources such as envelopes and templates and their related child objects. - - Incremental sync based on timestamp tracking. - - Pagination and performance: uses offset-based pagination with sensible batch sizes to efficiently iterate large result sets. - - Resiliency and partial-failure handling: per-envelope sub-resource failures are logged and skipped so a single broken item does not stop the entire sync. +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + +## Features + +- Extracts core Docusign resources such as envelopes and templates and their related child objects. +- Incremental sync based on timestamp tracking. +- Pagination and performance: uses offset-based pagination with sensible batch sizes to efficiently iterate large result sets. +- Resiliency and partial-failure handling: per-envelope sub-resource failures are logged and skipped so a single broken item does not stop the entire sync. ## Configuration file @@ -39,28 +48,29 @@ The configuration for this connector is defined in `configuration.json`. "account_id": "" } ``` + Configuration parameters: - - access_token (required) - The OAuth2 access token for authenticating with the Docusign API. - - base_url (required) - The base URL for the Docusign API (e.g., `https://demo.docusign.net/restapi` for demo or `https://na3.docusign.net/restapi` for production). - - account_id (required) - The Account ID for your Docusign account. +- access_token (required) - The OAuth2 access token for authenticating with the Docusign API. +- base_url (required) - The base URL for the Docusign API (e.g., `https://demo.docusign.net/restapi` for demo or `https://na3.docusign.net/restapi` for production). +- account_id (required) - The Account ID for your Docusign account. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector uses the `requests` library, which is pre-installed in the Fivetran environment. No additional libraries are required, so the `requirements.txt` file can be empty. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication This connector authenticates using an OAuth2 bearer token. The token is provided in `configuration.json` and used in the `get_docusign_headers` function to create the required `Authorization` header for all API requests. To obtain credentials: -1. Configure an OAuth integration within your Docusign account (e.g., using JWT Grant or Authorization Code Grant). -2. Generate a valid `access_token`. -3. Find your `account_id` and correct `base_url` (e.g., `demo.docusign.net` or `na3.docusign.net`) from your Docusign Admin panel. -4. Add these three values to the `configuration.json` file. +1. Configure an OAuth integration within your Docusign account (e.g., using JWT Grant or Authorization Code Grant). +2. Generate a valid `access_token`. +3. Find your `account_id` and correct `base_url` (e.g., `demo.docusign.net` or `na3.docusign.net`) from your Docusign Admin panel. +4. Add these three values to the `configuration.json` file. ## Pagination @@ -81,7 +91,7 @@ Data is processed within the `update` function and its various `fetch_*` helper ## Error handling -- API request errors: The `make_api_request` function uses `response.raise_for_status()` to automatically raise an exception for HTTP 4xx (client) or 5xx (server) errors. +- API request errors: The `make_api_request` function uses `response.raise_for_status()` to automatically raise an exception for HTTP 4xx (client) or 5xx (server) errors. - Resilient child object fetching: All functions that fetch data for a specific envelope (e.g., `fetch_audit_events`, `fetch_document_content`, `fetch_recipients_for_envelope`) are wrapped in `try...except` blocks. If an API call for one envelope's sub-resource fails, the error is logged using `log.warning` and an empty list or `None` is returned. This prevents a single failed document or recipient from stopping the entire sync. - Global error handling: The entire `update` function is wrapped in a `try...except Exception as e` block. Any unhandled exception will be caught and raised as a `RuntimeError`, which signals to Fivetran that the sync has failed. @@ -91,9 +101,8 @@ This connector creates the following tables in the destination, as defined in th ### ENVELOPES - ```json - { - +```json +{ "table": "envelopes", "primary_key": [ "envelope_id" @@ -115,8 +124,8 @@ This connector creates the following tables in the destination, as defined in th ### RECIPIENTS - ```json - { +```json +{ "table": "recipients", "primary_key": [ "envelope_id", @@ -136,8 +145,8 @@ This connector creates the following tables in the destination, as defined in th ### ENHANCED_RECIPIENTS - ```json - { +```json +{ "table": "enhanced_recipients", "primary_key": [ "envelope_id", @@ -159,9 +168,9 @@ This connector creates the following tables in the destination, as defined in th ``` ### AUDIT_EVENTS - - ```json - { + +```json +{ "table": "audit_events", "primary_key": [ "envelope_id", @@ -180,9 +189,9 @@ This connector creates the following tables in the destination, as defined in th ``` ### ENVELOPE_NOTIFICATIONS - - ```json - { + +```json +{ "table": "envelope_notifications", "primary_key": [ "envelope_id", @@ -200,8 +209,8 @@ This connector creates the following tables in the destination, as defined in th ### DOCUMENTS - ```json - { +```json +{ "table": "documents", "primary_key": [ "envelope_id", @@ -219,8 +228,8 @@ This connector creates the following tables in the destination, as defined in th ### DOCUMENT_CONTENTS - ```json - { +```json +{ "table": "document_contents", "primary_key": [ "envelope_id", @@ -236,8 +245,8 @@ This connector creates the following tables in the destination, as defined in th ### TEMPLATES - ```json - { +```json +{ "table": "templates", "primary_key": [ "template_id" @@ -254,9 +263,9 @@ This connector creates the following tables in the destination, as defined in th ``` ### CUSTOM_FIELDS - - ```json - { + +```json +{ "table": "custom_fields", "primary_key": [ "envelope_id", diff --git a/dolphin_db/README.md b/dolphin_db/README.md index 1c12894..0b73d4f 100644 --- a/dolphin_db/README.md +++ b/dolphin_db/README.md @@ -6,15 +6,25 @@ This connector syncs data from DolphinDB, a high-performance time series columna ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template dolphin_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -38,7 +48,7 @@ The connector requires the following configuration parameters for connecting to } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -54,7 +64,7 @@ sqlalchemy==1.4.54 The `pydolphindb` package is used to connect to the DolphinDB database, while `pandas` and `numpy` are used for data manipulation and processing. `SQLAlchemy` is included for database interaction. `dolphindb` is a pre-requisite for `pydolphindb`, which is a Python client for DolphinDB. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -95,4 +105,4 @@ The connector creates a single table named `trade` in the destination database. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/dragonfly_db/README.md b/dragonfly_db/README.md index 124eb6f..49bd95b 100644 --- a/dragonfly_db/README.md +++ b/dragonfly_db/README.md @@ -18,10 +18,20 @@ The connector is particularly valuable for organizations using DragonflyDB as a Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template dragonfly_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Only syncs new or modified keys using deterministic MD5 hashing -- Aautomatically detects and removes keys deleted from DragonflyDB +- Automatically detects and removes keys deleted from DragonflyDB - Automatic detection and timestamp-based incremental sync for RedisTimeSeries keys - Synchronizes all Redis-compatible data types: strings, hashes, lists, sets, and sorted sets - High-performance data extraction leveraging DragonflyDB's multi-threaded architecture @@ -70,7 +80,7 @@ Key pattern examples: - `user:*:profile` - All user profile keys - `rate_limit:*` - All rate limiter keys -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -80,7 +90,7 @@ The connector uses the `redis` library for connecting to DragonflyDB (Redis-comp redis ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/elastic_email/README.md b/elastic_email/README.md index 266408b..2374d53 100644 --- a/elastic_email/README.md +++ b/elastic_email/README.md @@ -16,6 +16,16 @@ This connector syncs email marketing data from Elastic Email to your destination Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template elastic_email +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Full sync of campaigns, contacts, lists, segments, and templates @@ -32,19 +42,19 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co The connector requires the following configuration parameters: -``` +```json { "api_key": "" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector uses only the standard libraries provided by the Fivetran Connector SDK runtime. No additional packages are required. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -92,7 +102,7 @@ Refer to the `make_request_with_retry` function in [connector.py](connector.py). The connector creates the following tables in the destination: -| Table Name | Primary Key | Description | +| Table name | Primary key | Description | |------------|-------------|-------------| | `CAMPAIGN` | `name` | Email marketing campaigns. | | `CONTACT` | `email` | Contact list with email addresses and metadata. | diff --git a/firebird_db/README.md b/firebird_db/README.md index 63fc155..b895608 100644 --- a/firebird_db/README.md +++ b/firebird_db/README.md @@ -10,25 +10,35 @@ The connector maintains tables as defined in the `schema.py` file: ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating System: - * Windows 10 or later - * macOS 13 (Ventura) or later - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows 10 or later + - macOS 13 (Ventura) or later + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template firebird_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -* Fetches data from multiple Firebird database tables -* Multi-threaded processing with configurable worker count -* Incremental sync based on configurable incremental columns -* Batch processing with configurable batch sizes -* Automatic datetime conversion to string format -* Thread-safe checkpointing for reliable data synchronization -* Concurrent table processing for improved performance +- Fetches data from multiple Firebird database tables +- Multi-threaded processing with configurable worker count +- Incremental sync based on configurable incremental columns +- Batch processing with configurable batch sizes +- Automatic datetime conversion to string format +- Thread-safe checkpointing for reliable data synchronization +- Concurrent table processing for improved performance ## Configuration file @@ -38,36 +48,12 @@ The connector requires configuration with your Firebird database credentials: { "host": "", "database": "", - "user": "", - "password": "" + "user": "", + "password": "" } ``` -> NOTE: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - -## Schema file - -The connector uses a separate `schema.py` file to define tables and their configuration: - -```python -table_list = [ - { - "table_name": "table_1", - "incremental_column": "ID", - "primary_key": ["ID"] - }, - { - "table_name": "table_2", - "incremental_column": "DATE_MODIFIED", - "primary_key": ["ID"] - }, - { - "table_name": "table_3", - "incremental_column": "DATE_MODIFIED", - "primary_key": ["ID"] - } -] -``` +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -77,7 +63,7 @@ The connector requires the `firebirdsql` package for database connectivity: firebirdsql==1.3.4 ``` -Note: The `fivetran_connector_sdk:latest` package is pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare it in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -110,7 +96,7 @@ The connector implements error handling for: All errors are logged using the Fivetran logging system for debugging and monitoring. -## Tables Created +## Tables created The connector creates tables as defined in the `schema.py` file. Each table includes: - Configurable table name @@ -131,7 +117,29 @@ Incremental column: `DATE_MODIFIED` Primary key: `ID` Incremental column: `DATE_MODIFIED` -## Performance Configuration +## Additional files + +The connector uses a separate `schema.py` file to define tables and their configuration: + +```python +table_list = [ + { + "table_name": "table_1", + "incremental_column": "ID", + "primary_key": ["ID"] + }, + { + "table_name": "table_2", + "incremental_column": "DATE_MODIFIED", + "primary_key": ["ID"] + }, + { + "table_name": "table_3", + "incremental_column": "DATE_MODIFIED", + "primary_key": ["ID"] + } +] +``` The connector includes several configurable performance parameters: - `batch_process_size`: Number of records processed per upsert (default: 1000) diff --git a/fleetio/README.md b/fleetio/README.md index 4df7e06..8319edd 100644 --- a/fleetio/README.md +++ b/fleetio/README.md @@ -13,18 +13,27 @@ With their data all in one place, a customer can create custom reporting for ins This example was contributed by [Fleetio](https://www.fleetio.com/). - ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template fleetio +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Configuration file @@ -32,14 +41,14 @@ To authenticate to your fleet data, update the configuration file with your deta The Fleetio API is authenticated by API Key. You can generate your Account Token and API Key [here](https://secure.fleetio.com/api_keys). -``` +```json { "Account-Token": "", "Authorization": "Token " } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -50,48 +59,38 @@ We currently only need a package for flattening the JSON information that's retu flatten_json==0.1.14 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Pagination -The connector paginates by checking whether `next_cursor` is returned by the endpoint and then calling the API with `start_cursor` set to that. +The connector paginates by checking whether `next_cursor` is returned by the endpoint and then calling the API with `start_cursor` set to that. The connector stops once `next_cursor` is null. See the function `continue_pagination` from lines 176-185 for more details on implementation. ## Data handling All data that comes in through the Fleetio API is flattened by one level before being upserted into the destination table. -The primary key for all tables for the upsert is the `id` field. +The primary key for all tables for the upsert is the `id` field. There are certain columns that are kept as JSON or lists that mainly hold custom fields and labels; if you're interested in finding out which columns are kept nested, i.e., which are set to the JSON data type, please check the base_schema defined from lines 17-160. ## Error handling -From lines 189-197 in `connector.py`, there's a try block when making the call to the Fleetio API. +From lines 189-197 in `connector.py`, there's a try block when making the call to the Fleetio API. If the API returns any errors, they are recorded in the Fivetran log and displayed in the Fivetran dashboard. -## Tables Created +## Tables created + The following tables are synced to the destination: -* `contacts` - * A Contact is a person employed by your organization or a person with whom your organization does business. -* `expense_entries` - * An Expense Entry is a record of a cost associated with a Vehicle. -* `fuel_entries` - * Fuel Entries represent a fuel transaction in Fleetio and are an important feature to maximize your data. - They contain all of the relevant information for the fuel-up, such as vendor and odometer information. -* `issues` - * Issues in Fleetio allows users to log and track unexpected, unplanned, or "one-time" problems and repairs. -* `parts` - * Parts management is key to running an efficient fleet. - Fleetio makes it easy to keep track of Parts data, including Part Manufacturer, Vendor, and Location. -* `purchase_orders` - * Purchase Orders in Fleetio are designed to help standardize Parts procurement through a purchasing workflow. -* `service_entries` - * Service Entries are a simple way to log completed Service Tasks and Issues-the routine Preventative Maintenance and one-time repairs that are performed on Vehicles in Fleetio. -* `submitted_inspection_forms` - * Submitted Inspection Forms are Inspection Forms that have been completed and submitted. -* `vehicles` - * A "vehicle" represents any asset or unit of equipment-moving or otherwise-managed in Fleetio. +- `contacts` – A Contact is a person employed by your organization or a person with whom your organization does business. +- `expense_entries` – An Expense Entry is a record of a cost associated with a Vehicle. +- `fuel_entries` – Fuel Entries represent a fuel transaction in Fleetio and are an important feature to maximize your data. They contain all of the relevant information for the fuel-up, such as vendor and odometer information. +- `issues` – Issues in Fleetio allows users to log and track unexpected, unplanned, or "one-time" problems and repairs. +- `parts` – Parts management is key to running an efficient fleet. Fleetio makes it easy to keep track of Parts data, including Part Manufacturer, Vendor, and Location. +- `purchase_orders` – Purchase Orders in Fleetio are designed to help standardize Parts procurement through a purchasing workflow. +- `service_entries` – Service Entries are a simple way to log completed Service Tasks and Issues-the routine Preventative Maintenance and one-time repairs that are performed on Vehicles in Fleetio. +- `submitted_inspection_forms` – Submitted Inspection Forms are Inspection Forms that have been completed and submitted. +- `vehicles` – A "vehicle" represents any asset or unit of equipment-moving or otherwise-managed in Fleetio. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/fred/README.md b/fred/README.md index af2c70c..68669f2 100644 --- a/fred/README.md +++ b/fred/README.md @@ -16,6 +16,16 @@ This connector syncs economic data from the Federal Reserve Economic Data (FRED) Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template fred +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Syncs series metadata including title, frequency, units, and seasonal adjustment information @@ -29,7 +39,6 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co ## Configuration file - ```json { "api_key": "", @@ -44,13 +53,13 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - `series_ids`: Comma-separated list of series IDs to sync (e.g., "GNPCA,UNRATE,CPIAUCSL") - `sync_start_date`: Optional start date for syncing observations in YYYY-MM-DD format (defaults to 2020-01-01) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require a `requirements.txt` file as it only uses standard library modules and the `requests` library, which is pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -96,7 +105,7 @@ The connector implements comprehensive error handling: ## Tables created -| Table Name | Primary Key | Description | +| Table name | Primary key | Description | |-----------------------|---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `series` | `id` | Contains metadata about economic data series including title, frequency, units, seasonal adjustment information, observation dates, popularity, and notes. | | `series_observations` | `series_id`, `date` | Contains time series data points for each series with values, series identifier, observation date, and realtime start/end dates indicating when the data was available. | diff --git a/gcp_pub_sub/README.md b/gcp_pub_sub/README.md index 884aec6..7b9cc3a 100644 --- a/gcp_pub_sub/README.md +++ b/gcp_pub_sub/README.md @@ -6,7 +6,6 @@ This connector demonstrates how to sync data from Google Cloud Pub/Sub to a dest > Note: This example includes topic creation and message publishing functionality for testing purposes only. In production deployments, your application would publish messages to existing topics, and the connector would only consume and sync those messages. - ## Requirements - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) @@ -15,11 +14,19 @@ This connector demonstrates how to sync data from Google Cloud Pub/Sub to a dest - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template gcp_pub_sub +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -28,7 +35,6 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - JSON message parsing – Decodes and parses JSON-formatted message payloads - State checkpointing – Tracks sync progress for reliable data delivery - ## Configuration file The connector requires the following configuration parameters: @@ -47,8 +53,7 @@ The connector requires the following configuration parameters: - `subscription_name` – Name of the Pub/Sub subscription to consume messages from - `topic_name` – Name of the Pub/Sub topic associated with the subscription -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -61,8 +66,7 @@ google-api-core protobuf==6.30.1 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -82,7 +86,6 @@ To obtain a service account key: 6. Copy the entire JSON content and paste it as a string in the `service_key` field of `configuration.json`. - ## Pagination The connector implements batch-based pagination using the `max_messages` configuration parameter. The `pull_and_upsert_messages()` function continuously pulls messages in batches until no more messages are available: @@ -94,7 +97,6 @@ The connector implements batch-based pagination using the `max_messages` configu This approach efficiently handles large message volumes while maintaining memory efficiency. - ## Data handling The connector processes Pub/Sub messages through the following workflow (see `pull_and_upsert_messages()`): @@ -106,7 +108,6 @@ The connector processes Pub/Sub messages through the following workflow (see `pu 5. Acknowledgment – Acknowledges processed messages to remove them from the subscription queue 6. Checkpointing – Saves sync state after processing all available messages - ## Error handling The connector implements error handling at multiple levels: @@ -116,7 +117,6 @@ The connector implements error handling at multiple levels: - Wraps message processing in try-except blocks to catch and report errors during: - All exceptions are re-raised as `RuntimeError` with contextual error messages for debugging. - ## Tables created The connector creates a single table to store Pub/Sub messages: diff --git a/github/README.md b/github/README.md index 9639f61..b7188e3 100644 --- a/github/README.md +++ b/github/README.md @@ -26,6 +26,16 @@ Example use cases include: Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template github +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - GitHub App authentication - Uses JWT-based authentication for secure, long-term access. @@ -58,7 +68,7 @@ Configuration parameters: - `organization` (required) - GitHub organization name to sync (can be comma-separated for multiple orgs). - `installation_id` (required) - Installation ID for the GitHub App on your organization. -Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -69,9 +79,9 @@ pyjwt==2.11.0 cryptography==36.0.1 ``` -Explanation: PyJWT is used for GitHub App JWT authentication. The cryptography package is required by PyJWT for RSA algorithm support (RS256), which is used to sign JWTs with the GitHub App's private key. +PyJWT is used for GitHub App JWT authentication. The cryptography package is required by PyJWT for RSA algorithm support (RS256), which is used to sign JWTs with the GitHub App's private key. -Note: The `fivetran_connector_sdk` and `requests` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/github_traffic/README.md b/github_traffic/README.md index acd715d..4593431 100644 --- a/github_traffic/README.md +++ b/github_traffic/README.md @@ -6,11 +6,41 @@ This connector syncs GitHub repository traffic data using the GitHub REST API. I - Top referral sources - Top popular content paths -## Prerequisites -1. [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) with the Read access to administration and metadata. -2. Repositories you want to monitor (in the format `owner/repo1, owner/repo2`) +## Connector overview + +This connector syncs GitHub repository traffic metrics (views, clones, referrers, and popular content paths) using the GitHub REST API. It demonstrates how to work with GitHub's traffic analytics endpoints, handle limited historical data (14 days), and use Personal Access Token authentication. + +## Requirements + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + +## Getting started + +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template github_traffic +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + +## Features + +- Syncs repository views, clones, referrers, and popular content paths +- Uses Personal Access Token authentication +- Handles GitHub's 14-day traffic data window +- Supports multiple repositories in a single sync + +## Configuration file -## Configuration Update the `configuration.json` file with your GitHub Personal Access Token and the repositories you want to monitor: ```json @@ -20,8 +50,30 @@ Update the `configuration.json` file with your GitHub Personal Access Token and } ``` -## Tables and Schemas +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. + +## Requirements file + +This connector uses only pre-installed packages in the Fivetran environment. + +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. + +## Authentication + +This connector uses [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) authentication. The token must have Read access to administration and metadata. + +The Personal Access Token used must have the `repo` scope to access traffic data for private repositories. For public repositories, only the `public_repo` scope is needed. + +The GitHub API has rate limits. For authenticated requests, the rate limit is 5,000 requests per hour. + +## Data handling + +The traffic data endpoints specifically (views, clones, referrers, paths) only return data for the past 14 days. + +## Tables created + ### repository_views + Contains daily view counts for each repository. | Column | Type | Description | @@ -34,6 +86,7 @@ Contains daily view counts for each repository. ### repository_clones Contains daily clone counts for each repository. + | Column | Type | Description | |-------------|--------------|---------------------------------------| | repository | STRING | Repository name in owner/repo format | @@ -42,6 +95,7 @@ Contains daily clone counts for each repository. | uniques | INT | Unique cloner count for the day | ### repository_referrers + Contains top referral sources for each repository. | Column | Type | Description | @@ -65,25 +119,6 @@ Contains top content paths for each repository. | uniques | INT | Unique visitors for this path | | fetch_date | NAIVE_DATE | Date when data was collected from the API | -## GitHub API Rate Limits - -The GitHub API has rate limits. For authenticated requests, the rate limit is 5,000 requests per hour. - -The traffic data endpoints specifically (views, clones, referrers, paths) only return data for the past 14 days. - -## Required GitHub Permissions - -The Personal Access Token used must have the `repo` scope to access traffic data for private repositories. For public repositories, only the `public_repo` scope is needed. - -## Running the Connector - -To run the connector locally for testing: - -```bash -fivetran debug --configuration configuration.json -``` - -## Resources +## Additional considerations -- [GitHub Traffic API Documentation](https://docs.github.com/en/rest/metrics/traffic) -- [Fivetran Connector SDK Documentation](https://fivetran.com/docs/connectors/connector-sdk) \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/gnews/README.md b/gnews/README.md index fdfafc8..84acaef 100644 --- a/gnews/README.md +++ b/gnews/README.md @@ -1,27 +1,42 @@ # GNews Search Connector Example ## Connector overview + This example demonstrates how to build a Fivetran Connector SDK integration for [GNews v4 Search API](https://gnews.io/docs/v4#tag/Search), a REST API service to search articles from 60,000+ worldwide sources. The API provides access to real-time news and historical data, as well as top headlines based on Google News rankings. The connector pulls Google search information from GNews's API, and delivers it to your Fivetran destination. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template gnews +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features -- Fetches news for a given `search_term` and `from_date` via `/api/v4/search`. -- Iterates over pages until the end of the result set or an optional `max_pages` cap. -- Retries transient failures (429/5xx and network timeouts) with exponential backoff and jitter. -- Flattens `articles[*]` and maps fields to stable column names (`image` → `urlToImage`). -- Defines one destination table — `news_stories` with primary key (`url`, `publishedAt`). + +- Fetches news for a given `search_term` and `from_date` via `/api/v4/search`. +- Iterates over pages until the end of the result set or an optional `max_pages` cap. +- Retries transient failures (429/5xx and network timeouts) with exponential backoff and jitter. +- Flattens `articles[*]` and maps fields to stable column names (`image` → `urlToImage`). +- Defines one destination table — `news_stories` with primary key (`url`, `publishedAt`). - Uses `fivetran_connector_sdk.Logging` for structured info and error logs, including plan-limit messages. ## Configuration file + The `configuration.json` file provides the GNews credentials and query parameters required for API requests. ```json @@ -31,18 +46,18 @@ The `configuration.json` file provides the GNews credentials and query parameter "from_date": "" } ``` + - `api_key` (required): Your GNews API key - `search_term` (required): The query you are looking to get data on - `from_date` (required): How far back the historical sync will go -Note: Ensure that `configuration.json` is not committed to version control. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require any additional packages beyond those provided by the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Data handling @@ -51,9 +66,9 @@ Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre - Persists only scalar fields (nested values are JSON-encoded strings) - Upserts: Each record is written via `op.upsert(...)` to `NEWS_STORIES`. - Stop conditions: `fetch_all_news` stops when: - - Fewer than `page_size` results are returned, or - - `page * page_size >= totalArticles`, or - - `max_pages` is reached, or + - Fewer than `page_size` results are returned, or + - `page * page_size >= totalArticles`, or + - `max_pages` is reached, or - A page returns zero results ## Error handling @@ -66,8 +81,10 @@ Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre The connector creates the `NEWS_STORIES` table. ### NEWS_STORIES + - Primary key: `url`, `publishedAt` - Sample columns: `url`, `publishedAt`, `title`, `description`, `content`, `urlToImage`, `source_id`, `source_name`, `author`, `query` ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. From 2468cdd04948f033a2ba2e772e9a94d2e34215b8 Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Tue, 7 Apr 2026 23:57:42 +0530 Subject: [PATCH 02/19] fix: correct fivetran init template paths for nested connector directories --- apache_hive/using_pyhive/README.md | 2 +- apache_hive/using_sqlalchemy/README.md | 2 +- aws_athena/using_boto3/README.md | 2 +- aws_athena/using_sqlalchemy/README.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/apache_hive/using_pyhive/README.md b/apache_hive/using_pyhive/README.md index 0f19111..9ea0c38 100644 --- a/apache_hive/using_pyhive/README.md +++ b/apache_hive/using_pyhive/README.md @@ -19,7 +19,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co To initialize a new Connector SDK project using this connector as a starting point, run: ``` -fivetran init --template apache_hive +fivetran init --template apache_hive/using_pyhive ``` `fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). diff --git a/apache_hive/using_sqlalchemy/README.md b/apache_hive/using_sqlalchemy/README.md index dca1308..4477851 100644 --- a/apache_hive/using_sqlalchemy/README.md +++ b/apache_hive/using_sqlalchemy/README.md @@ -19,7 +19,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co To initialize a new Connector SDK project using this connector as a starting point, run: ``` -fivetran init --template apache_hive +fivetran init --template apache_hive/using_sqlalchemy ``` `fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). diff --git a/aws_athena/using_boto3/README.md b/aws_athena/using_boto3/README.md index 4255962..b29cc9a 100644 --- a/aws_athena/using_boto3/README.md +++ b/aws_athena/using_boto3/README.md @@ -26,7 +26,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co To initialize a new Connector SDK project using this connector as a starting point, run: ``` -fivetran init --template aws_athena +fivetran init --template aws_athena/using_boto3 ``` `fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). diff --git a/aws_athena/using_sqlalchemy/README.md b/aws_athena/using_sqlalchemy/README.md index 53888f6..aa3b79d 100644 --- a/aws_athena/using_sqlalchemy/README.md +++ b/aws_athena/using_sqlalchemy/README.md @@ -26,7 +26,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co To initialize a new Connector SDK project using this connector as a starting point, run: ``` -fivetran init --template aws_athena +fivetran init --template aws_athena/using_sqlalchemy ``` `fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). From b51e11156936a3501e783cebf0f95eca0cba553b Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Wed, 8 Apr 2026 00:31:47 +0530 Subject: [PATCH 03/19] fix: rename aws_rds_oracle/readme.md to README.md for consistency --- aws_rds_oracle/{readme.md => README.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename aws_rds_oracle/{readme.md => README.md} (100%) diff --git a/aws_rds_oracle/readme.md b/aws_rds_oracle/README.md similarity index 100% rename from aws_rds_oracle/readme.md rename to aws_rds_oracle/README.md From 993fe508f7095ec1ab9671cf9ad47afc0303728e Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Wed, 8 Apr 2026 13:36:18 +0530 Subject: [PATCH 04/19] fix: update fivetran init doc link to correct URL --- _template_connector/README_template.md | 2 +- apache_hbase/README.md | 2 +- apache_hive/using_pyhive/README.md | 2 +- apache_hive/using_sqlalchemy/README.md | 2 +- apache_pulsar/README.md | 2 +- arango_db/README.md | 2 +- awardco/README.md | 2 +- aws_athena/using_boto3/README.md | 2 +- aws_athena/using_sqlalchemy/README.md | 2 +- aws_dynamo_db_authentication/README.md | 2 +- aws_rds_oracle/README.md | 2 +- betterstack/README.md | 2 +- cassandra/README.md | 2 +- checkly/README.md | 2 +- clerk/README.md | 2 +- clickhouse/README.md | 2 +- commonpaper/README.md | 2 +- couchbase_capella/README.md | 2 +- couchbase_magma/README.md | 2 +- courier/README.md | 2 +- customer_thermometer/README.md | 2 +- data_camp/README.md | 2 +- dgraph/README.md | 2 +- discord/README.md | 2 +- documentdb/README.md | 2 +- docusign/README.md | 2 +- dolphin_db/README.md | 2 +- dragonfly_db/README.md | 2 +- elastic_email/README.md | 2 +- firebird_db/README.md | 2 +- fleetio/README.md | 2 +- fred/README.md | 2 +- gcp_pub_sub/README.md | 2 +- github/README.md | 2 +- github_traffic/README.md | 2 +- gnews/README.md | 2 +- 36 files changed, 36 insertions(+), 36 deletions(-) diff --git a/_template_connector/README_template.md b/_template_connector/README_template.md index 9aefcf3..102954c 100644 --- a/_template_connector/README_template.md +++ b/_template_connector/README_template.md @@ -56,7 +56,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). *Include the following note only if the connector requires a `configuration.json` file to run.* diff --git a/apache_hbase/README.md b/apache_hbase/README.md index 54cb0b8..be17f13 100644 --- a/apache_hbase/README.md +++ b/apache_hbase/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template apache_hbase ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/apache_hive/using_pyhive/README.md b/apache_hive/using_pyhive/README.md index 9ea0c38..9d5e4cd 100644 --- a/apache_hive/using_pyhive/README.md +++ b/apache_hive/using_pyhive/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template apache_hive/using_pyhive ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/apache_hive/using_sqlalchemy/README.md b/apache_hive/using_sqlalchemy/README.md index 4477851..1897cdf 100644 --- a/apache_hive/using_sqlalchemy/README.md +++ b/apache_hive/using_sqlalchemy/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template apache_hive/using_sqlalchemy ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/apache_pulsar/README.md b/apache_pulsar/README.md index cdd665f..ac50686 100644 --- a/apache_pulsar/README.md +++ b/apache_pulsar/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template apache_pulsar ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/arango_db/README.md b/arango_db/README.md index 7945220..b094714 100644 --- a/arango_db/README.md +++ b/arango_db/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template arango_db ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/awardco/README.md b/awardco/README.md index 032bbda..63a7489 100644 --- a/awardco/README.md +++ b/awardco/README.md @@ -28,7 +28,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template awardco ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/aws_athena/using_boto3/README.md b/aws_athena/using_boto3/README.md index b29cc9a..5d7cf4a 100644 --- a/aws_athena/using_boto3/README.md +++ b/aws_athena/using_boto3/README.md @@ -29,7 +29,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template aws_athena/using_boto3 ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/aws_athena/using_sqlalchemy/README.md b/aws_athena/using_sqlalchemy/README.md index aa3b79d..2abfe52 100644 --- a/aws_athena/using_sqlalchemy/README.md +++ b/aws_athena/using_sqlalchemy/README.md @@ -29,7 +29,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template aws_athena/using_sqlalchemy ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/aws_dynamo_db_authentication/README.md b/aws_dynamo_db_authentication/README.md index 9a32ba7..cd31be1 100644 --- a/aws_dynamo_db_authentication/README.md +++ b/aws_dynamo_db_authentication/README.md @@ -30,7 +30,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template aws_dynamo_db_authentication ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/aws_rds_oracle/README.md b/aws_rds_oracle/README.md index 7e34c3f..6f91d60 100644 --- a/aws_rds_oracle/README.md +++ b/aws_rds_oracle/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template aws_rds_oracle ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/betterstack/README.md b/betterstack/README.md index d32171d..383f084 100644 --- a/betterstack/README.md +++ b/betterstack/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template betterstack ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/cassandra/README.md b/cassandra/README.md index 76b95b3..fd7f83d 100644 --- a/cassandra/README.md +++ b/cassandra/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template cassandra ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/checkly/README.md b/checkly/README.md index 4c995bb..adeeda2 100644 --- a/checkly/README.md +++ b/checkly/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template checkly ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/clerk/README.md b/clerk/README.md index d995d0f..a573ffd 100644 --- a/clerk/README.md +++ b/clerk/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template clerk ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/clickhouse/README.md b/clickhouse/README.md index 146335a..5f8c0c4 100644 --- a/clickhouse/README.md +++ b/clickhouse/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template clickhouse ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/commonpaper/README.md b/commonpaper/README.md index e8b1dd2..f4ec393 100644 --- a/commonpaper/README.md +++ b/commonpaper/README.md @@ -26,7 +26,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template commonpaper ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 6bccdcc..65c86bb 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -25,7 +25,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template couchbase_capella ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index 981a238..4006211 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -23,7 +23,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template couchbase_magma ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/courier/README.md b/courier/README.md index b446cd5..e6d443d 100644 --- a/courier/README.md +++ b/courier/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template courier ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/customer_thermometer/README.md b/customer_thermometer/README.md index 385b24e..a3fab28 100644 --- a/customer_thermometer/README.md +++ b/customer_thermometer/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template customer_thermometer ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/data_camp/README.md b/data_camp/README.md index 28523f4..e0a5cad 100644 --- a/data_camp/README.md +++ b/data_camp/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template data_camp ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/dgraph/README.md b/dgraph/README.md index 336fa48..061d84f 100644 --- a/dgraph/README.md +++ b/dgraph/README.md @@ -28,7 +28,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template dgraph ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/discord/README.md b/discord/README.md index edd04f4..410df03 100644 --- a/discord/README.md +++ b/discord/README.md @@ -31,7 +31,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template discord ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/documentdb/README.md b/documentdb/README.md index ab465e8..493b58c 100644 --- a/documentdb/README.md +++ b/documentdb/README.md @@ -28,7 +28,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template documentdb ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/docusign/README.md b/docusign/README.md index 17088d4..718469c 100644 --- a/docusign/README.md +++ b/docusign/README.md @@ -26,7 +26,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template docusign ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/dolphin_db/README.md b/dolphin_db/README.md index 0b73d4f..113475d 100644 --- a/dolphin_db/README.md +++ b/dolphin_db/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template dolphin_db ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/dragonfly_db/README.md b/dragonfly_db/README.md index 49bd95b..4ff3465 100644 --- a/dragonfly_db/README.md +++ b/dragonfly_db/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template dragonfly_db ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/elastic_email/README.md b/elastic_email/README.md index 2374d53..641d518 100644 --- a/elastic_email/README.md +++ b/elastic_email/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template elastic_email ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/firebird_db/README.md b/firebird_db/README.md index b895608..c976a74 100644 --- a/firebird_db/README.md +++ b/firebird_db/README.md @@ -26,7 +26,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template firebird_db ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/fleetio/README.md b/fleetio/README.md index 8319edd..9b8fb74 100644 --- a/fleetio/README.md +++ b/fleetio/README.md @@ -31,7 +31,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template fleetio ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/fred/README.md b/fred/README.md index 68669f2..df63d99 100644 --- a/fred/README.md +++ b/fred/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template fred ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/gcp_pub_sub/README.md b/gcp_pub_sub/README.md index 7b9cc3a..741be4b 100644 --- a/gcp_pub_sub/README.md +++ b/gcp_pub_sub/README.md @@ -24,7 +24,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template gcp_pub_sub ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/github/README.md b/github/README.md index b7188e3..2117cd2 100644 --- a/github/README.md +++ b/github/README.md @@ -32,7 +32,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template github ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/github_traffic/README.md b/github_traffic/README.md index 4593431..55499ce 100644 --- a/github_traffic/README.md +++ b/github_traffic/README.md @@ -28,7 +28,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template github_traffic ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. diff --git a/gnews/README.md b/gnews/README.md index 84acaef..b61c109 100644 --- a/gnews/README.md +++ b/gnews/README.md @@ -22,7 +22,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template gnews ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). > Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. From 53eee6de81da088459f6d4f1fecc0226073e27ab Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Wed, 8 Apr 2026 13:48:03 +0530 Subject: [PATCH 05/19] revert: remove template changes from batch 01 --- _template_connector/README_template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_template_connector/README_template.md b/_template_connector/README_template.md index 102954c..9aefcf3 100644 --- a/_template_connector/README_template.md +++ b/_template_connector/README_template.md @@ -56,7 +56,7 @@ To initialize a new Connector SDK project using this connector as a starting poi fivetran init --template ``` -`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK init documentation](https://fivetran.com/docs/connectors/connector-sdk/technical-reference/init). *Include the following note only if the connector requires a `configuration.json` file to run.* From d1872d5509dea12959129a2a15eeeadc22a70948 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 16:13:18 +0530 Subject: [PATCH 06/19] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- checkly/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/checkly/README.md b/checkly/README.md index adeeda2..7c04cb2 100644 --- a/checkly/README.md +++ b/checkly/README.md @@ -75,8 +75,8 @@ The connector uses Bearer Token authentication with the Checkly API. Authenticat You'll need to obtain the following credentials: -1. API key: A Checkly API key with read permissions for checks and analytics endpoints -2. Account ID: Your Checkly account identifier for proper API access +1. API key: A Checkly API key with read permissions for checks and analytics endpoints. +2. Account ID: Your Checkly account identifier for proper API access. ### Steps to obtain credentials From de204c05ca9d1a339fa1f308bc5af9d95736634a Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 16:14:14 +0530 Subject: [PATCH 07/19] Update aws_dynamo_db_authentication/README.md --- aws_dynamo_db_authentication/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aws_dynamo_db_authentication/README.md b/aws_dynamo_db_authentication/README.md index cd31be1..98e054e 100644 --- a/aws_dynamo_db_authentication/README.md +++ b/aws_dynamo_db_authentication/README.md @@ -79,7 +79,7 @@ This connector authenticates by: ## Pagination Pagination is handled via: -- `aws_dynamodb_parallel_scan.get_paginator(` +- `aws_dynamodb_parallel_scan.get_paginator()` - Parallel scanning with `TotalSegments=4` and `Limit=10` per page. ## Data handling From 4d7ef9a2204a63b8d19c09c1d99e676e0282e56a Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 17:26:55 +0530 Subject: [PATCH 08/19] Added the correct couchbase magma connector link Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- couchbase_capella/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 65c86bb..7a28f10 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -5,7 +5,7 @@ This connector example demonstrates how to sync data from Couchbase Capella using the Connector SDK. It connects to a Couchbase Capella instance, executes SQL++ (N1QL) queries to fetch data from a specific bucket, scope, and collection, and efficiently streams the data to destination table while implementing best practices for handling large datasets. This connector is built for Couchbase Capella. It supports buckets using either the Magma or Couchstore storage engines. -> For syncing data from a Magma bucket on self-managed Couchbase Server, please refer to the [Couchbase Magma connector example](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/examples/source_examples/couchbase_magma) +> For syncing data from a Magma bucket on self-managed Couchbase Server, please refer to the [Couchbase Magma connector example](https://github.com/fivetran/fivetran-csdk-connectors/tree/main/couchbase_magma) ## Requirements From 2c097eca1a22cef814ddc4e4823c4c0d6afb2678 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 17:46:37 +0530 Subject: [PATCH 09/19] Added placeholders --- couchbase_magma/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index 4006211..e6ddd0f 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -40,11 +40,11 @@ The connector requires the following configuration parameters to connect to your ```json { - "username": "YOUR_COUCHBASE_USERNAME", - "password": "YOUR_COUCHBASE_PASSWORD", - "endpoint": "YOUR_COUCHBASE_ENDPOINT", - "bucket_name": "YOUR_COUCHBASE_BUCKET_NAME", - "scope": "YOUR_COUCHBASE_SCOPE_NAME", + "username": "", + "password": "", + "endpoint": "", + "bucket_name": "", + "scope": "", "collection": "YOUR_COUCHBASE_COLLECTION_NAME", "use_tls": "", "cert_path": "PATH_TO_YOUR_TLS_CERTIFICATE" From 26e1be0ef1cb8a533756d2e067735f480b22fa76 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 17:48:33 +0530 Subject: [PATCH 10/19] Apply suggestions from code review --- couchbase_magma/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index e6ddd0f..dd50a85 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -45,7 +45,7 @@ The connector requires the following configuration parameters to connect to your "endpoint": "", "bucket_name": "", "scope": "", - "collection": "YOUR_COUCHBASE_COLLECTION_NAME", + "collection": "", "use_tls": "", "cert_path": "PATH_TO_YOUR_TLS_CERTIFICATE" } From f90c4bffcd39a1e58e6d7fb76ed5d413014507a7 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 17:49:31 +0530 Subject: [PATCH 11/19] Update couchbase_capella/README.md --- couchbase_capella/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 7a28f10..5bd49f2 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -43,8 +43,8 @@ The connector requires the following configuration parameters to connect to your ```json { - "username": "YOUR_COUCHBASE_USERNAME", - "password": "YOUR_COUCHBASE_PASSWORD", + "username": "", + "password": "", "endpoint": "YOUR_COUCHBASE_ENDPOINT", "bucket_name": "YOUR_COUCHBASE_BUCKET_NAME", "scope": "YOUR_COUCHBASE_SCOPE_NAME", From 7fd03f9de8fa767d3130548597f649e0811c30b1 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 17:50:51 +0530 Subject: [PATCH 12/19] Update couchbase_capella/README.md --- couchbase_capella/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 5bd49f2..788dd07 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -45,10 +45,10 @@ The connector requires the following configuration parameters to connect to your { "username": "", "password": "", - "endpoint": "YOUR_COUCHBASE_ENDPOINT", - "bucket_name": "YOUR_COUCHBASE_BUCKET_NAME", - "scope": "YOUR_COUCHBASE_SCOPE_NAME", - "collection": "YOUR_COUCHBASE_COLLECTION_NAME" + "endpoint": "", + "bucket_name": "", + "scope": "", + "collection": "" } ``` From a601be9d14dd783a8efdf15b9e173ce6884f03db Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 18:14:43 +0530 Subject: [PATCH 13/19] Update couchbase_magma/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- couchbase_magma/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index dd50a85..de9db99 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -47,7 +47,7 @@ The connector requires the following configuration parameters to connect to your "scope": "", "collection": "", "use_tls": "", - "cert_path": "PATH_TO_YOUR_TLS_CERTIFICATE" + "cert_path": "" } ``` From 82d869aa9fdb82e29b9f424556ce1dfb60a14c37 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 18:17:51 +0530 Subject: [PATCH 14/19] Update documentdb/README.md --- documentdb/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/documentdb/README.md b/documentdb/README.md index 493b58c..412ccc3 100644 --- a/documentdb/README.md +++ b/documentdb/README.md @@ -156,9 +156,9 @@ This connector requires that the following prerequisites exist in your DocumentD Example document structure: ```json { - "_id": "ObjectId(...)", - "created_at": "ISODate(2023-01-01T00:00:00Z)", - "updated_at": "ISODate(2023-01-01T00:00:00Z)", + "_id": ObjectId("..."), + "created_at": ISODate("2023-01-01T00:00:00Z"), + "updated_at": ISODate("2023-01-01T00:00:00Z"), "metadata": {"source": "web"} } ``` From d1298e3ace624b4cc0e68eccf8c3ba4cb1eb7984 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Wed, 8 Apr 2026 18:26:43 +0530 Subject: [PATCH 15/19] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- firebird_db/README.md | 6 +++--- gnews/README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/firebird_db/README.md b/firebird_db/README.md index c976a74..82e046a 100644 --- a/firebird_db/README.md +++ b/firebird_db/README.md @@ -10,10 +10,10 @@ The connector maintains tables as defined in the `schema.py` file: ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows 10 or later - - macOS 13 (Ventura) or later + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started diff --git a/gnews/README.md b/gnews/README.md index b61c109..8fee6ef 100644 --- a/gnews/README.md +++ b/gnews/README.md @@ -64,7 +64,7 @@ This connector does not require any additional packages beyond those provided by - Data normalization: `normalize_articles` flattens each `articles[*]` object and maps: - `image` → `urlToImage` (to keep parity with existing schema/column names - Persists only scalar fields (nested values are JSON-encoded strings) -- Upserts: Each record is written via `op.upsert(...)` to `NEWS_STORIES`. +- Upserts: Each record is written via `op.upsert(...)` to `news_stories`. - Stop conditions: `fetch_all_news` stops when: - Fewer than `page_size` results are returned, or - `page * page_size >= totalArticles`, or @@ -78,7 +78,7 @@ This connector does not require any additional packages beyond those provided by ## Tables created -The connector creates the `NEWS_STORIES` table. +The connector creates the `news_stories` table. ### NEWS_STORIES From 2499884507d1140497a33c6822c38e2edc1d14ec Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Sat, 11 Apr 2026 21:11:51 +0530 Subject: [PATCH 16/19] docs: update pre-installed packages note to link --- _template_connector/README_template.md | 2 +- apache_hbase/README.md | 2 +- apache_hive/using_pyhive/README.md | 2 +- apache_hive/using_sqlalchemy/README.md | 2 +- apache_pulsar/README.md | 2 +- arango_db/README.md | 2 +- awardco/README.md | 2 +- aws_athena/using_boto3/README.md | 2 +- aws_athena/using_sqlalchemy/README.md | 2 +- aws_dynamo_db_authentication/README.md | 2 +- aws_rds_oracle/README.md | 2 +- betterstack/README.md | 2 +- cassandra/README.md | 6 +++--- checkly/README.md | 2 +- clerk/README.md | 2 +- clickhouse/README.md | 6 +++--- commonpaper/README.md | 4 ++-- couchbase_capella/README.md | 2 +- couchbase_magma/README.md | 4 +++- courier/README.md | 2 +- customer_thermometer/README.md | 2 +- data_camp/README.md | 2 +- dgraph/README.md | 2 +- discord/README.md | 2 +- documentdb/README.md | 2 +- docusign/README.md | 2 +- dolphin_db/README.md | 2 +- dragonfly_db/README.md | 2 +- elastic_email/README.md | 2 +- firebird_db/README.md | 4 ++-- fleetio/README.md | 11 ++++------- fred/README.md | 2 +- gcp_pub_sub/README.md | 2 +- github/README.md | 2 +- github_traffic/README.md | 12 +++--------- gnews/README.md | 2 +- 36 files changed, 49 insertions(+), 56 deletions(-) diff --git a/_template_connector/README_template.md b/_template_connector/README_template.md index 9aefcf3..58456c1 100644 --- a/_template_connector/README_template.md +++ b/_template_connector/README_template.md @@ -90,7 +90,7 @@ fivetran init --template pandas ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/apache_hbase/README.md b/apache_hbase/README.md index be17f13..d213d13 100644 --- a/apache_hbase/README.md +++ b/apache_hbase/README.md @@ -57,7 +57,7 @@ This connector requires the happybase library to communicate with Apache HBase: happybase==1.2.0 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/apache_hive/using_pyhive/README.md b/apache_hive/using_pyhive/README.md index 9d5e4cd..8adb234 100644 --- a/apache_hive/using_pyhive/README.md +++ b/apache_hive/using_pyhive/README.md @@ -58,7 +58,7 @@ thrift_sasl sasl ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/apache_hive/using_sqlalchemy/README.md b/apache_hive/using_sqlalchemy/README.md index 1897cdf..304b011 100644 --- a/apache_hive/using_sqlalchemy/README.md +++ b/apache_hive/using_sqlalchemy/README.md @@ -60,7 +60,7 @@ sasl `PyHive` is required for actual dialect implementation for Hive along with the `SQLAlchemy` ORM. The `thrift_sasl` and `sasl` packages are necessary for SASL authentication, which is commonly used with Hive. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/apache_pulsar/README.md b/apache_pulsar/README.md index ac50686..7185e9c 100644 --- a/apache_pulsar/README.md +++ b/apache_pulsar/README.md @@ -68,7 +68,7 @@ The `requirements.txt` file specifies the Python libraries required by the conne pulsar-client>=3.4.0 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/arango_db/README.md b/arango_db/README.md index b094714..13b8727 100644 --- a/arango_db/README.md +++ b/arango_db/README.md @@ -66,7 +66,7 @@ The connector requires the `python-arango` package to connect to ArangoDB databa python-arango ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/awardco/README.md b/awardco/README.md index 63a7489..7c6c195 100644 --- a/awardco/README.md +++ b/awardco/README.md @@ -68,7 +68,7 @@ Configuration parameters: This connector does not require any additional packages beyond those provided by the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/aws_athena/using_boto3/README.md b/aws_athena/using_boto3/README.md index 5d7cf4a..716c0fb 100644 --- a/aws_athena/using_boto3/README.md +++ b/aws_athena/using_boto3/README.md @@ -66,7 +66,7 @@ This connector requires the following Python packages: boto3 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/aws_athena/using_sqlalchemy/README.md b/aws_athena/using_sqlalchemy/README.md index 2abfe52..51518ee 100644 --- a/aws_athena/using_sqlalchemy/README.md +++ b/aws_athena/using_sqlalchemy/README.md @@ -65,7 +65,7 @@ PyAthena sqlalchemy ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/aws_dynamo_db_authentication/README.md b/aws_dynamo_db_authentication/README.md index 98e054e..b16871e 100644 --- a/aws_dynamo_db_authentication/README.md +++ b/aws_dynamo_db_authentication/README.md @@ -67,7 +67,7 @@ aws_dynamodb_parallel_scan==1.1.0 boto3==1.37.4 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/aws_rds_oracle/README.md b/aws_rds_oracle/README.md index 6f91d60..f45f388 100644 --- a/aws_rds_oracle/README.md +++ b/aws_rds_oracle/README.md @@ -57,7 +57,7 @@ The connector requires the `oracledb` package to connect to Oracle databases. oracledb==3.3.0 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/betterstack/README.md b/betterstack/README.md index 383f084..0b991a8 100644 --- a/betterstack/README.md +++ b/betterstack/README.md @@ -51,7 +51,7 @@ The connector requires the following configuration parameter: This connector uses only the standard library and SDK-provided packages. No additional dependencies are required in `requirements.txt`. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/cassandra/README.md b/cassandra/README.md index fd7f83d..344459f 100644 --- a/cassandra/README.md +++ b/cassandra/README.md @@ -1,4 +1,4 @@ -# Cassandra Database Example +# Cassandra Connector Example ## Connector overview @@ -68,7 +68,7 @@ cassandra-driver python-dateutil ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -128,7 +128,7 @@ The `SAMPLE_TABLE` contains sample data from the Cassandra database. The schema ## Additional files -- `adding_dummy_data_to_cassandra.py`: This python file contains functions to add dummy data to the Cassandra database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Cassandra table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Cassandra database. +- **`adding_dummy_data_to_cassandra.py`**: This python file contains functions to add dummy data to the Cassandra database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Cassandra table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Cassandra database. ## Additional considerations diff --git a/checkly/README.md b/checkly/README.md index 7c04cb2..2987703 100644 --- a/checkly/README.md +++ b/checkly/README.md @@ -67,7 +67,7 @@ Optional parameters: This connector example uses standard libraries provided by Python and does not require any additional packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/clerk/README.md b/clerk/README.md index a573ffd..2e17b24 100644 --- a/clerk/README.md +++ b/clerk/README.md @@ -58,7 +58,7 @@ Configuration parameters: This connector example uses standard libraries provided by Python and does not require any additional packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/clickhouse/README.md b/clickhouse/README.md index 5f8c0c4..42e999a 100644 --- a/clickhouse/README.md +++ b/clickhouse/README.md @@ -1,4 +1,4 @@ -# ClickHouse Database Example +# ClickHouse Connector Example ## Connector overview @@ -61,7 +61,7 @@ The connector requires the clickhouse_connect Python library for connecting to C clickhouse_connect==0.8.17 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -110,7 +110,7 @@ The `TEST_TABLE` table contains dummy data with the following schema: ## Additional files -`clickhouse_dummy_data_generator.py`: This python file contains functions to add dummy data to the Clickhouse database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Clickhouse table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Clickhouse database. +**`clickhouse_dummy_data_generator.py`**: This python file contains functions to add dummy data to the Clickhouse database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Clickhouse table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Clickhouse database. ## Additional considerations diff --git a/commonpaper/README.md b/commonpaper/README.md index f4ec393..b2b854e 100644 --- a/commonpaper/README.md +++ b/commonpaper/README.md @@ -1,4 +1,4 @@ -# Common Paper Connector +# Common Paper Connector Example ## Connector overview @@ -59,7 +59,7 @@ Configuration parameters: The connector does not use any additional Python packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/couchbase_capella/README.md b/couchbase_capella/README.md index 788dd07..2071be8 100644 --- a/couchbase_capella/README.md +++ b/couchbase_capella/README.md @@ -62,7 +62,7 @@ The connector requires the Couchbase Python SDK: couchbase==4.3.6 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/couchbase_magma/README.md b/couchbase_magma/README.md index de9db99..0f202a9 100644 --- a/couchbase_magma/README.md +++ b/couchbase_magma/README.md @@ -1,5 +1,7 @@ # Couchbase Magma Connector Example +## Connector overview + This connector example demonstrates how to sync data from a self-managed Couchbase Server (local, on-premises, or self-hosted in the cloud) using the Magma storage engine with the Connector SDK. It connects to a Couchbase instance, runs SQL++ (N1QL) queries to retrieve data from a specified Magma bucket, scope, and collection, and streams the results efficiently to a destination table—following best practices for handling large datasets. @@ -61,7 +63,7 @@ The connector requires the Couchbase Python SDK: couchbase==4.3.6 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/courier/README.md b/courier/README.md index e6d443d..57d196c 100644 --- a/courier/README.md +++ b/courier/README.md @@ -54,7 +54,7 @@ The connector requires the following configuration parameters: This connector does not require any additional packages beyond the pre-installed libraries in the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/customer_thermometer/README.md b/customer_thermometer/README.md index a3fab28..7b5b56c 100644 --- a/customer_thermometer/README.md +++ b/customer_thermometer/README.md @@ -60,7 +60,7 @@ The connector requires API key authentication for the Customer Thermometer API. This connector example uses standard libraries provided by Python and does not require any additional packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/data_camp/README.md b/data_camp/README.md index e0a5cad..8e5e0f7 100644 --- a/data_camp/README.md +++ b/data_camp/README.md @@ -66,7 +66,7 @@ The connector requires bearer token authentication for the DataCamp LMS Catalog This connector example uses the standard libraries provided by Python and does not require any additional packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/dgraph/README.md b/dgraph/README.md index 061d84f..396f52f 100644 --- a/dgraph/README.md +++ b/dgraph/README.md @@ -65,7 +65,7 @@ Configuration parameters: This connector uses only the Python standard library and SDK-provided packages. No additional dependencies are required beyond what is pre-installed in the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/discord/README.md b/discord/README.md index 410df03..43dbc95 100644 --- a/discord/README.md +++ b/discord/README.md @@ -73,7 +73,7 @@ The connector requires the following configuration parameters in `configuration. The `requirements.txt` file specifies the Python libraries required by the connector. This connector uses only the standard library and pre-installed packages. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/documentdb/README.md b/documentdb/README.md index 412ccc3..3a1aeff 100644 --- a/documentdb/README.md +++ b/documentdb/README.md @@ -73,7 +73,7 @@ pymongo python-dateutil ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/docusign/README.md b/docusign/README.md index 718469c..fe4806c 100644 --- a/docusign/README.md +++ b/docusign/README.md @@ -60,7 +60,7 @@ Configuration parameters: This connector uses the `requests` library, which is pre-installed in the Fivetran environment. No additional libraries are required, so the `requirements.txt` file can be empty. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/dolphin_db/README.md b/dolphin_db/README.md index 113475d..4d9e6c0 100644 --- a/dolphin_db/README.md +++ b/dolphin_db/README.md @@ -64,7 +64,7 @@ sqlalchemy==1.4.54 The `pydolphindb` package is used to connect to the DolphinDB database, while `pandas` and `numpy` are used for data manipulation and processing. `SQLAlchemy` is included for database interaction. `dolphindb` is a pre-requisite for `pydolphindb`, which is a Python client for DolphinDB. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/dragonfly_db/README.md b/dragonfly_db/README.md index 4ff3465..d55f6bb 100644 --- a/dragonfly_db/README.md +++ b/dragonfly_db/README.md @@ -90,7 +90,7 @@ The connector uses the `redis` library for connecting to DragonflyDB (Redis-comp redis ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/elastic_email/README.md b/elastic_email/README.md index 641d518..432f440 100644 --- a/elastic_email/README.md +++ b/elastic_email/README.md @@ -54,7 +54,7 @@ The connector requires the following configuration parameters: This connector uses only the standard libraries provided by the Fivetran Connector SDK runtime. No additional packages are required. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/firebird_db/README.md b/firebird_db/README.md index 82e046a..d39928c 100644 --- a/firebird_db/README.md +++ b/firebird_db/README.md @@ -63,7 +63,7 @@ The connector requires the `firebirdsql` package for database connectivity: firebirdsql==1.3.4 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -119,7 +119,7 @@ Incremental column: `DATE_MODIFIED` ## Additional files -The connector uses a separate `schema.py` file to define tables and their configuration: +The connector uses a separate **`schema.py`** file to define tables and their configuration: ```python table_list = [ diff --git a/fleetio/README.md b/fleetio/README.md index 9b8fb74..2342115 100644 --- a/fleetio/README.md +++ b/fleetio/README.md @@ -1,13 +1,10 @@ # Fleetio Connector Example -The Fleetio Connector allows Fleetio customers to easily pull their data into their destination databases with Fivetran. -For more information about Fleetio, check out [Fleetio site](https://www.fleetio.com) or [contact Fleetio](https://www.fleetio.com/contact)! -If you need more details about the endpoints pulled by the connector, visit Fleetio's [API docs](https://developer.fleetio.com/docs/category/api). - ## Connector overview -This connector allows Fleetio customers on a [Professional Plan](https://www.fleetio.com/pricing) or above to bring in their most crucial data into any destination systems. -With their data all in one place, a customer can create custom reporting for insight into the operation of their fleets. +The Fleetio Connector allows Fleetio customers to easily pull their data into their destination databases with Fivetran. For more information about Fleetio, check out [Fleetio site](https://www.fleetio.com) or [contact Fleetio](https://www.fleetio.com/contact). If you need more details about the endpoints pulled by the connector, visit Fleetio's [API docs](https://developer.fleetio.com/docs/category/api). + +This connector allows Fleetio customers on a [Professional Plan](https://www.fleetio.com/pricing) or above to bring in their most crucial data into any destination systems. With their data all in one place, a customer can create custom reporting for insight into the operation of their fleets. ## Accreditation @@ -59,7 +56,7 @@ We currently only need a package for flattening the JSON information that's retu flatten_json==0.1.14 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Pagination diff --git a/fred/README.md b/fred/README.md index df63d99..0a42059 100644 --- a/fred/README.md +++ b/fred/README.md @@ -59,7 +59,7 @@ fivetran init --template fred This connector does not require a `requirements.txt` file as it only uses standard library modules and the `requests` library, which is pre-installed in the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/gcp_pub_sub/README.md b/gcp_pub_sub/README.md index 741be4b..b35f578 100644 --- a/gcp_pub_sub/README.md +++ b/gcp_pub_sub/README.md @@ -66,7 +66,7 @@ google-api-core protobuf==6.30.1 ``` -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/github/README.md b/github/README.md index 2117cd2..8ec5ef8 100644 --- a/github/README.md +++ b/github/README.md @@ -81,7 +81,7 @@ cryptography==36.0.1 PyJWT is used for GitHub App JWT authentication. The cryptography package is required by PyJWT for RSA algorithm support (RS256), which is used to sign JWTs with the GitHub App's private key. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/github_traffic/README.md b/github_traffic/README.md index 55499ce..e23a7b2 100644 --- a/github_traffic/README.md +++ b/github_traffic/README.md @@ -1,14 +1,8 @@ -# GitHub Repository Traffic Connector - -This connector syncs GitHub repository traffic data using the GitHub REST API. It collects the following information: -- Repository views (count and unique visitors) -- Repository clones (count and unique cloners) -- Top referral sources -- Top popular content paths +# GitHub Repository Traffic Connector Example ## Connector overview -This connector syncs GitHub repository traffic metrics (views, clones, referrers, and popular content paths) using the GitHub REST API. It demonstrates how to work with GitHub's traffic analytics endpoints, handle limited historical data (14 days), and use Personal Access Token authentication. +This connector syncs GitHub repository traffic data using the GitHub REST API. It collects repository views (count and unique visitors), repository clones (count and unique cloners), top referral sources, and top popular content paths. It demonstrates how to work with GitHub's traffic analytics endpoints, handle limited historical data (14 days), and use Personal Access Token authentication. ## Requirements @@ -56,7 +50,7 @@ Update the `configuration.json` file with your GitHub Personal Access Token and This connector uses only pre-installed packages in the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/gnews/README.md b/gnews/README.md index 8fee6ef..4b54ad8 100644 --- a/gnews/README.md +++ b/gnews/README.md @@ -57,7 +57,7 @@ The `configuration.json` file provides the GNews credentials and query parameter This connector does not require any additional packages beyond those provided by the Fivetran environment. -> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Data handling From c4804638176fc32cf0a631a2bc329a145d72dc69 Mon Sep 17 00:00:00 2001 From: fivetran-JenasVimal Date: Mon, 13 Apr 2026 11:44:19 +0530 Subject: [PATCH 17/19] docs: revert template README change from batch commit --- _template_connector/README_template.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_template_connector/README_template.md b/_template_connector/README_template.md index 58456c1..9aefcf3 100644 --- a/_template_connector/README_template.md +++ b/_template_connector/README_template.md @@ -90,7 +90,7 @@ fivetran init --template pandas ``` -> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: The `fivetran_connector_sdk:latest`, `requests:2.33.0`, `grpcio:1.78.0`, and `grpcio-tools:1.78.0` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication From 04370293b5cde501cb37edc4c857f91356c17536 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Mon, 13 Apr 2026 12:30:23 +0530 Subject: [PATCH 18/19] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- aws_rds_oracle/README.md | 6 +++--- clickhouse/README.md | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/aws_rds_oracle/README.md b/aws_rds_oracle/README.md index f45f388..13dacbf 100644 --- a/aws_rds_oracle/README.md +++ b/aws_rds_oracle/README.md @@ -114,9 +114,9 @@ You can add more tables to the `TABLES` list in `connector.py` as needed. This connector doesn't require any additional files, apart from the standard ones listed below: -- `connector.py` contains the `update()` and `schema()` implementations for this example. -- `configuration.json` provides a placeholder configuration you can copy and fill with your Oracle credentials. -- `requirements.txt` declares the Oracle driver dependency used at runtime. +- **connector.py** - Contains the `update()` and `schema()` implementations for this example. +- **configuration.json** - Provides a placeholder configuration you can copy and fill with your Oracle credentials. +- **requirements.txt** - Declares the Oracle driver dependency used at runtime. ## Additional considerations diff --git a/clickhouse/README.md b/clickhouse/README.md index 42e999a..2709fbc 100644 --- a/clickhouse/README.md +++ b/clickhouse/README.md @@ -67,10 +67,10 @@ clickhouse_connect==0.8.17 The connector uses basic username and password authentication to connect to ClickHouse. These credentials are specified in the configuration file. The connector uses TLS/SSL for secure connections (indicated by the secure=True parameter in the client configuration). To obtain credentials: -1. Create an account in the ClickHouse Cloud console -2. Click on the "Connect" button for your cluster -3. Choose the python Language client to get the connection details -4. Use the connection details to fill in the configuration file +1. Create an account in the ClickHouse Cloud console. +2. Click on the "Connect" button for your cluster. +3. Choose the python Language client to get the connection details. +4. Use the connection details to fill in the configuration file. ## Pagination From 7449da117ca25d53135e250561a9c263568d5612 Mon Sep 17 00:00:00 2001 From: Jenas Anton Vimal Date: Tue, 14 Apr 2026 00:21:18 +0530 Subject: [PATCH 19/19] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- cassandra/README.md | 4 ---- clickhouse/README.md | 4 ---- 2 files changed, 8 deletions(-) diff --git a/cassandra/README.md b/cassandra/README.md index 344459f..d267220 100644 --- a/cassandra/README.md +++ b/cassandra/README.md @@ -126,10 +126,6 @@ The `SAMPLE_TABLE` contains sample data from the Cassandra database. The schema } ``` -## Additional files - -- **`adding_dummy_data_to_cassandra.py`**: This python file contains functions to add dummy data to the Cassandra database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Cassandra table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Cassandra database. - ## Additional considerations The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/clickhouse/README.md b/clickhouse/README.md index 2709fbc..4558b9d 100644 --- a/clickhouse/README.md +++ b/clickhouse/README.md @@ -108,10 +108,6 @@ The `TEST_TABLE` table contains dummy data with the following schema: } ``` -## Additional files - -**`clickhouse_dummy_data_generator.py`**: This python file contains functions to add dummy data to the Clickhouse database. It creates dummy database and table and generates random records with unique IDs and timestamps. This dummy data is inserted into the Clickhouse table for testing purposes. In production, you will not need to insert dummy data, as the connector will work with your existing Clickhouse database. - ## Additional considerations The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team.