diff --git a/quest_db/README.md b/quest_db/README.md index d5e4fe9..23cf7f9 100644 --- a/quest_db/README.md +++ b/quest_db/README.md @@ -16,6 +16,16 @@ This connector syncs high-performance time series data from QuestDB to your Five Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template quest_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Syncs time series data from multiple QuestDB tables @@ -45,8 +55,7 @@ The connector requires the following configuration parameters: Configuration parameters: - -- `tables` (required) - Comma-separated list of table names to sync (e.g., `SENSOR_DATA`, `MARKET_TICKS`) +- `tables` (required) - Comma-separated list of table names to sync (e.g., `SENSOR_DATA`, `MARKET_TICKS`) - `host` (optional) - QuestDB server hostname or IP address (defaults to `localhost`) - `port` (optional) - QuestDB REST API port (defaults to `9000`) - `username` (optional) - QuestDB username for HTTP basic authentication @@ -54,13 +63,13 @@ Configuration parameters: - `batch_size` (optional) - Number of records to fetch per API request (defaults to `1000`) - `timestamp_column` (optional) - Column name for timestamp-based incremental sync (defaults to `timestamp`) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require a `requirements.txt` file as it only uses standard library modules and the `requests` library, which is pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication diff --git a/rabbitmq/README.md b/rabbitmq/README.md index 6272a6c..554a25c 100644 --- a/rabbitmq/README.md +++ b/rabbitmq/README.md @@ -1,9 +1,11 @@ -# RabbitMQ Connector SDK Example +# RabbitMQ Connector Example ## Connector overview + This connector syncs messages from RabbitMQ queues to Fivetran. The connector retrieves messages from specified queues using RabbitMQ's `basic_get` method. It consumes the messages only after a successful sync to prevent duplicate reads. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -11,9 +13,21 @@ This connector syncs messages from RabbitMQ queues to Fivetran. The connector re - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template rabbitmq +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Synchronizes messages from multiple RabbitMQ queues - Captures complete message metadata including delivery tags, routing keys, headers, and timestamps - Implements incremental sync with per-queue state tracking using delivery tags @@ -24,7 +38,6 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co ## Configuration file - ```json { "connection_url": "", @@ -39,7 +52,7 @@ The configuration keys required for your connector are as follows: - `queues` (required) - Comma-separated list of queue names to sync (for example: "orders,payments,notifications") - `batch_size` (optional) - Number of messages to process per batch (defaults to 100) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -51,9 +64,10 @@ pika==1.3.2 The connector uses the `pika` library (version 1.3.2) for connecting to and consuming messages from RabbitMQ. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses AMQP connection URLs for authentication. The connection URL includes credentials embedded in the URL format. Example configurations: @@ -83,7 +97,7 @@ The connector automatically handles AMQPS/TLS connections when using the `amqps: ## Data handling -**Important**: Messages are permanently removed from RabbitMQ after a successful sync. This connector consumes messages using `basic_ack`, which deletes them from the queue. Use this connector only with dedicated analytical queues (for example, dead-letter or audit queues); never use it on production operational queues. +> Note: Messages are permanently removed from RabbitMQ after a successful sync. This connector consumes messages using `basic_ack`, which deletes them from the queue. Use this connector only with dedicated analytical queues (for example, dead-letter or audit queues); never use it on production operational queues. The connector performs message consumption and transformation. Refer to the `fetch_and_upsert_messages_batch` function for message handling: diff --git a/raven_db/README.md b/raven_db/README.md index 1272116..3d2b873 100644 --- a/raven_db/README.md +++ b/raven_db/README.md @@ -10,14 +10,24 @@ The connector is designed to handle large datasets efficiently through streaming - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template raven_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Connect to RavenDB clusters with certificate-based authentication @@ -40,13 +50,14 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co ``` ### Configuration parameters + - `ravendb_urls` (required): Your RavenDB Cloud URLs. For clusters, provide comma-separated URLs - `database_name` (required): Name of the RavenDB database to sync from - `certificate_base64` (required): Base64-encoded client certificate (PEM format) -Important: You must provide the certificate as a base64-encoded PEM certificate string. +> Note: You must provide the certificate as a base64-encoded PEM certificate string. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -56,7 +67,7 @@ The connector requires the RavenDB Python client: ravendb ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -68,11 +79,11 @@ To obtain your certificate: 3. Download the client certificate package. 4. Convert the certificate to base64 format: - - **macOS/Linux:** + - macOS/Linux: ```sh base64 -i certificate.pem ``` - - **Windows:** + - Windows: ```cmd certutil -encode certificate.pem certificate.b64 ``` @@ -114,20 +125,21 @@ The connector implements the following error-handling strategies: This connector creates tables in your destination based on the collections you sync. The table structure follows this pattern: ### Example: `ORDERS` table (if syncing an Orders collection) -- Primary Key: `Id` (STRING - RavenDB document ID) + +- Primary key: `Id` (STRING - RavenDB document ID) - Columns: - - `Id` (STRING): Document unique identifier - - `OrderNumber` (STRING): Order number - - `CustomerName` (STRING): Customer name - - `TotalAmount` (FLOAT): Order total - - `Status` (STRING): Order status - - `Items` (JSON): Order items as JSON array - - `ShippingAddress_Street` (STRING): Flattened nested field - - `ShippingAddress_City` (STRING): Flattened nested field - - `LastModified` (UTC_DATETIME): Document last update timestamp from metadata - -Note: The actual schema depends on your document structure. Nested objects are flattened with underscore separators, and arrays are stored as JSON strings. + - `Id` (STRING): Document unique identifier + - `OrderNumber` (STRING): Order number + - `CustomerName` (STRING): Customer name + - `TotalAmount` (FLOAT): Order total + - `Status` (STRING): Order status + - `Items` (JSON): Order items as JSON array + - `ShippingAddress_Street` (STRING): Flattened nested field + - `ShippingAddress_City` (STRING): Flattened nested field + - `LastModified` (UTC_DATETIME): Document last update timestamp from metadata + +> Note: The actual schema depends on your document structure. Nested objects are flattened with underscore separators, and arrays are stored as JSON strings. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/redis/README.md b/redis/README.md index 082f432..ff77df9 100644 --- a/redis/README.md +++ b/redis/README.md @@ -1,6 +1,7 @@ # Redis Connector Example ## Connector overview + This connector syncs data from Redis to Fivetran for historical analytics. It supports both traditional key-value data and RedisTimeSeries time series data with incremental synchronization. The connector is designed for applications that use Redis with persistence (AOF/RDB) as their primary database. It handles leaderboards, player profiles, real-time game state, and time series metrics. The connector retrieves all key-value pairs including sorted sets (leaderboards), hashes (player profiles), counters, and TimeSeries keys. It uses the Redis SCAN command for efficient, resumable synchronization with cursor-based pagination and state management. @@ -13,13 +14,14 @@ The connector supports the following use cases: - Session analytics - User journey analysis when Redis stores session data with persistence enabled. - Time series metrics - Incremental sync of RedisTimeSeries data (IoT sensors, application metrics, financial data) using timestamp-based queries. -Note: This connector is designed for scenarios where Redis is used as a persistent database (not cache). +> Note: This connector is designed for scenarios where Redis is used as a persistent database (not cache). This connector is NOT suitable for: - Redis used purely as cache layer (use source database connector instead) - Ephemeral session stores with no persistence ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -27,9 +29,21 @@ This connector is NOT suitable for: - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template redis +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Synchronizes gaming leaderboards (sorted sets), player profiles (hashes), and statistics from Redis - Supports all Redis data types: strings (counters), hashes (player data), lists (match history), sets (achievements), and sorted sets (leaderboards) - RedisTimeSeries support: Automatic detection and incremental sync of TimeSeries keys using timestamp-based queries @@ -43,11 +57,11 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - SSL/TLS connection support for secure communication - Password authentication for protected Redis instances - ## Configuration file + The configuration keys required for your connector are as follows: -``` +```json { "host": "", "port": "", @@ -81,9 +95,10 @@ The configuration keys required for your connector are as follows: - `"game:*:stats"` - All game statistics - `"achievement:*"` - All achievement data -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies the Python libraries required by the connector: ``` @@ -92,14 +107,15 @@ redis The connector uses the `redis` library for connecting to and querying Redis databases. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector supports both local and cloud Redis deployments with flexible authentication. ### Basic configuration (no password) -``` +```json { "host": "", "port": "", @@ -109,7 +125,7 @@ The connector supports both local and cloud Redis deployments with flexible auth ### Password-protected Redis -``` +```json { "host": "", "port": "", @@ -118,9 +134,10 @@ The connector supports both local and cloud Redis deployments with flexible auth } ``` -Note: Redis Cloud and other managed services typically require both username (usually "default") and password. The connector automatically handles SSL/TLS when connecting to cloud providers. +> Note: Redis Cloud and other managed services typically require both username (usually "default") and password. The connector automatically handles SSL/TLS when connecting to cloud providers. ## Pagination + The connector implements Redis' `SCAN` command for efficient, non-blocking key iteration. Refer to the `scan_redis_keys` function in `connector.py` which handles pagination with the following approach: - Uses cursor-based iteration starting from cursor position 0 @@ -133,6 +150,7 @@ The connector implements Redis' `SCAN` command for efficient, non-blocking key i The `SCAN` operation is memory-efficient, non-blocking, and allows other Redis operations to continue during data extraction. ## Data handling + The connector performs comprehensive data extraction and transformation. - Key discovery - Uses `SCAN` command with optional pattern filtering to discover matching keys (`scan_redis_keys`) @@ -145,6 +163,7 @@ The connector performs comprehensive data extraction and transformation. The `update` function orchestrates the complete sync process with state management and error handling, coordinated by `sync_redis_data`. ## Error handling + The connector implements comprehensive error handling strategies. Refer to the following functions in `connector.py`: - Connection validation (`create_redis_client`): Tests Redis connectivity during client creation with ping operation, handles SSL configuration errors @@ -155,6 +174,7 @@ The connector implements comprehensive error handling strategies. Refer to the f - TimeSeries errors (`get_timeseries_range`): Catches and logs failures when querying TimeSeries data, returns empty list instead of crashing ## Tables created + The connector creates a table in the destination based on your configuration (configurable via `table_name` parameter, defaults to `REDIS_DATA`): | Column | Type | Description | @@ -169,4 +189,5 @@ The connector creates a table in the destination based on your configuration (co The table uses `key` as the primary key, enabling upserts for handling updated Redis values across syncs. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/redshift/large_data_volume/README.md b/redshift/large_data_volume/README.md index 236fce4..66630f3 100644 --- a/redshift/large_data_volume/README.md +++ b/redshift/large_data_volume/README.md @@ -1,21 +1,33 @@ # Redshift Large Data Volume Example Connector ## Connector overview -This example connector demonstrates how to sync large tables from Amazon Redshift efficiently by using the Connector SDK. The connector follows best practices for high-volume ingestion scenarios using Connector SDK. It implements optimized data extraction techniques, including parallel processing and incremental loading, to handle large datasets effectively. +This example connector demonstrates how to sync large tables from Amazon Redshift efficiently by using the Connector SDK. The connector follows best practices for high-volume ingestion scenarios using Connector SDK. It implements optimized data extraction techniques, including parallel processing and incremental loading, to handle large datasets effectively. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template redshift/large_data_volume +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Incremental sync via `replication_key` with ordered SQL queries. - Automatic schema detection from the source schema. - Automatic replication key inference based on column semantic types. @@ -26,8 +38,8 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Connection pooling to reduce overhead during parallel query execution - Graceful fallback to complete resync when no suitable replication key is found - ## Configuration file + The configuration file (`configuration.json`) contains the necessary parameters to connect to Amazon Redshift. The content of this file is as follows: ```json @@ -44,6 +56,7 @@ The configuration file (`configuration.json`) contains the necessary parameters "max_parallel_workers": "" } ``` + The parameters include: - `redshift_host`: The hostname of the Redshift cluster. - `redshift_port`: The port number for the Redshift cluster. @@ -56,28 +69,28 @@ The parameters include: - `enable_complete_resync`: A boolean flag that defines whether each sync is a [full re-sync](https://fivetran.com/docs/using-fivetran/features#fullresync). - `max_parallel_workers`: The maximum number of parallel workers to use for data extraction. We recommend setting this value between 2 and 4. Setting it too high may lead to potential performance degradation. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector requires the following packages, which should be listed in the `requirements.txt` file: ``` redshift_connector ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector uses username and password authentication to connect to the Redshift database. The credentials are provided in the `configuration.json` file. Ensure that the Redshift user has the necessary permissions to read data from the specified schema and tables. +The connector uses username and password authentication to connect to the Redshift database. The credentials are provided in the `configuration.json` file. Ensure that the Redshift user has the necessary permissions to read data from the specified schema and tables. ## Pagination -The connector handles large datasets by implementing batch fetching. The `batch_size` parameter in the `configuration.json` file determines the number of rows fetched in each batch. This approach helps manage memory usage and improves performance when dealing with large tables. +The connector handles large datasets by implementing batch fetching. The `batch_size` parameter in the `configuration.json` file determines the number of rows fetched in each batch. This approach helps manage memory usage and improves performance when dealing with large tables. ## Data handling + The connector uses the `redshift_connector` library to connect to the Redshift database and execute SQL queries. It retrieves data in batches, processes each batch, and sends it for ingestion. The connector also supports incremental loading by using a `replication_key` to track changes in the source data. The steps involved in data handling include: @@ -91,22 +104,22 @@ The steps involved in data handling include: The connector also implements parallel processing to speed up data extraction. The `max_parallel_workers` parameter controls the number of concurrent workers used for fetching data from multiple tables simultaneously. - ## Error handling -The connector includes robust error handling mechanisms to manage potential issues during data extraction and processing. +The connector includes robust error handling mechanisms to manage potential issues during data extraction and processing. ## Tables created + The connector creates tables in the destination based on the source schema. The table names and structures are derived from the Redshift schema specified in the `configuration.json` file. The connector creates a table for each table found in the specified Redshift schema with the name format `_`. The connector automatically detects the schema of each table and creates corresponding tables in the destination with appropriate data types. If automatic schema detection is disabled, the connector uses the schema defined in the `table_spec.py` file. - ## Additional files -The connector includes the following additional files: -- `table_spec.py`: This file defines the schema for each table in the Redshift database. It is used when automatic schema detection is disabled. You can customize this file to specify the exact schema for each table, including column names and data types. -- `redshift_client.py`: This file contains the logic for connecting to the Redshift database and executing SQL queries. It encapsulates the connection handling, query execution, and data fetching logic. +The connector includes the following additional files: +- **`table_spec.py`**: This file defines the schema for each table in the Redshift database. It is used when automatic schema detection is disabled. You can customize this file to specify the exact schema for each table, including column names and data types. +- **`redshift_client.py`**: This file contains the logic for connecting to the Redshift database and executing SQL queries. It encapsulates the connection handling, query execution, and data fetching logic. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/redshift/simple_redshift_connector/README.md b/redshift/simple_redshift_connector/README.md index 55b72bb..8c4d25f 100644 --- a/redshift/simple_redshift_connector/README.md +++ b/redshift/simple_redshift_connector/README.md @@ -6,15 +6,25 @@ This connector demonstrates how to sync records from a Redshift database using t ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template redshift/simple_redshift_connector +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -26,7 +36,7 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se The connector requires the following configuration parameters: -``` +```json { "host": "", "database": "", @@ -36,7 +46,7 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -46,7 +56,7 @@ The connector requires the `redshift_connector` package to connect to Redshift d redshift_connector ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -81,12 +91,12 @@ The connector handles data as follows (refer to the `update()` function): The `dt2str()` function handles timestamp formatting, converting database datetime objects to the required string format. -## Tables Created +## Tables created The connector creates a single table named `customers` with the following schema (refer to the `schema()` function): ```json - { +{ "table": "customers", "primary_key": ["customer_id"], "columns": { diff --git a/redshift/using_unload/README.md b/redshift/using_unload/README.md index 1b25974..301a7b2 100644 --- a/redshift/using_unload/README.md +++ b/redshift/using_unload/README.md @@ -4,18 +4,27 @@ This example connector demonstrates how to efficiently sync large tables from Amazon Redshift by leveraging the native `UNLOAD` command to export data to S3 in Parquet format, then reading from S3 for ingestion into Fivetran. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template redshift/using_unload +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -29,7 +38,6 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Automatic S3 cleanup: Removes all temporary files (including manifests) after sync - Graceful fallback: Falls back to FULL sync when no suitable replication key is found - ## Configuration file The configuration file (`configuration.json`) contains the necessary parameters to connect to Amazon Redshift and S3: @@ -54,11 +62,12 @@ The configuration file (`configuration.json`) contains the necessary parameters } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ### Configuration parameters #### Redshift connection + | Parameter | Description | Required | |-----------|-------------|----------| | `redshift_host` | The hostname of the Redshift cluster | Yes | @@ -69,6 +78,7 @@ Note: Ensure that the `configuration.json` file is not checked into version cont | `redshift_schema` | The default schema to extract data from | Yes | #### S3 configuration + | Parameter | Description | Required | |-----------|-------------|----------| | `s3_bucket` | S3 bucket name for UNLOAD output | Yes | @@ -81,13 +91,13 @@ Note: Ensure that the `configuration.json` file is not checked into version cont The IAM role specified in `iam_role` must have `s3:PutObject` permission on the target S3 bucket. The AWS credentials provided in `aws_access_key_id` and `aws_secret_access_key` must have `s3:GetObject`, `s3:ListBucket`, and `s3:DeleteObject` permissions. #### Sync behavior + | Parameter | Description | Required | |-----------|-------------|----------| | `auto_schema_detection` | Enable automatic table and schema discovery (`true`/`false`) | Yes | | `enable_complete_resync` | Force FULL sync for all tables, ignoring incremental settings (`true`/`false`) | Yes | | `max_parallel_workers` | Maximum concurrent table syncs | Yes | - ## Requirements file The connector requires the following packages, which should be listed in the `requirements.txt` file: @@ -98,51 +108,50 @@ boto3==1.35.0 pyarrow==17.0.0 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. Do not declare them in your `requirements.txt` to avoid dependency conflicts. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses username and password authentication to connect to the Redshift database. The credentials are provided in the `configuration.json` file. Ensure that the Redshift user has the necessary permissions to read data from the specified schema and tables. The connector also requires an IAM role for Redshift to write `UNLOAD` output to S3, as well as AWS credentials for the connector to read and delete files from S3. - ## Pagination -The connector handles large datasets by leveraging Redshift's `UNLOAD` command to export data in Parquet format to `S3`. This approach allows efficient handling of large tables without the need for traditional pagination. The connector reads the Parquet files from `S3` in a memory-efficient manner using PyArrow's `S3FileSystem`, streaming data row-group by row-group. +The connector handles large datasets by leveraging Redshift's `UNLOAD` command to export data in Parquet format to `S3`. This approach allows efficient handling of large tables without the need for traditional pagination. The connector reads the Parquet files from `S3` in a memory-efficient manner using PyArrow's `S3FileSystem`, streaming data row-group by row-group. ## Data handling The connector uses the following workflow: -1. Schema discovery: Connects to Redshift and discovers tables and columns (or uses predefined `TABLE_SPECS`) -2. Table plan building: Builds sync plans for each table including primary keys, replication strategy, and column selection -3. `UNLOAD` execution: For each table, executes UNLOAD command to export data to S3 as Parquet files -4. S3 reading: Reads Parquet files from S3 using PyArrow S3FileSystem for memory-efficient streaming -5. Data sync: Upserts records using the connector SDK -6. Checkpointing: Periodically saves sync progress (every `CHECKPOINT_EVERY_ROWS` rows) -7. Cleanup: Deletes all temporary S3 files after successful sync - +1. Schema discovery: Connects to Redshift and discovers tables and columns (or uses predefined `TABLE_SPECS`). +2. Table plan building: Builds sync plans for each table including primary keys, replication strategy, and column selection. +3. `UNLOAD` execution: For each table, executes UNLOAD command to export data to S3 as Parquet files. +4. S3 reading: Reads Parquet files from S3 using PyArrow S3FileSystem for memory-efficient streaming. +5. Data sync: Upserts records using the connector SDK. +6. Checkpointing: Periodically saves sync progress (every `CHECKPOINT_EVERY_ROWS` rows). +7. Cleanup: Deletes all temporary S3 files after successful sync. ## Error handling + The connector includes robust error handling mechanisms to manage potential issues during data extraction and processing. Key strategies include: - Connection errors: Retries Redshift and S3 connections with exponential backoff - UNLOAD failures: Catches and logs `UNLOAD` command errors, aborting syncs if necessary - S3 read errors: Handles S3 read failures and retries as needed - ## Tables created + The connector creates tables in the destination based on the source schema. Table names and structures are derived from the Redshift schema specified in the `configuration.json` file. The connector creates a table for each table found in the specified Redshift schema with the name format `.`. The connector automatically detects the schema of each table and creates corresponding tables in the destination with appropriate data types. If automatic schema detection is disabled, the connector uses the schema defined in the `table_spec.py` file. - ## Additional files -The connector includes the following additional files: -- `table_spec.py` - This file defines the schema for each table in the Redshift database. The connector uses it when automatic schema detection is disabled. You can customize this file to specify the exact schema for each table, including column names and data types. -- `redshift_client.py` - This file contains the logic for connecting to the Redshift database and executing SQL queries. It encapsulates the connection handling, query execution, and data fetching logic. -- `s3_client.py` - This file contains the logic for interacting with S3, including reading Parquet files and deleting temporary files after sync. +The connector includes the following additional files: +- **`table_spec.py`** - This file defines the schema for each table in the Redshift database. The connector uses it when automatic schema detection is disabled. You can customize this file to specify the exact schema for each table, including column names and data types. +- **`redshift_client.py`** - This file contains the logic for connecting to the Redshift database and executing SQL queries. It encapsulates the connection handling, query execution, and data fetching logic. +- **`s3_client.py`** - This file contains the logic for interacting with S3, including reading Parquet files and deleting temporary files after sync. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/resend/README.md b/resend/README.md index f26a439..7a1509d 100644 --- a/resend/README.md +++ b/resend/README.md @@ -1,9 +1,11 @@ # Resend Connector Example ## Connector overview + This connector demonstrates how to fetch email data from Resend and upsert it into your destination using the Fivetran Connector SDK. The custom connector synchronizes email records from your Resend account and implements pagination handling to efficiently process large datasets with incremental synchronization based on email IDs. The connector tracks the newest email ID from each sync and stops fetching when it encounters previously synced data, ensuring efficient incremental updates without fetching the entire dataset on each run. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -11,9 +13,21 @@ This connector demonstrates how to fetch email data from Resend and upsert it in - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template resend +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Synchronizes emails from Resend's API `/emails` endpoint - Supports incremental syncing by tracking the newest email ID and stopping at previously synced emails - Implements cursor-based pagination handling for large datasets @@ -23,6 +37,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Flattens nested JSON structures for optimal table schemas ## Configuration file + The configuration key required for your connector is as follows: ```json @@ -35,14 +50,16 @@ The configuration key required for your connector is as follows: - `api_token` (required) - Your Resend API token for authentication. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector uses the `requests` library for HTTP communication, which is pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses Bearer token authentication with Resend API keys. To obtain your API key: 1. Log in to your Resend account. @@ -52,9 +69,10 @@ The connector uses Bearer token authentication with Resend API keys. To obtain y 5. Select the appropriate permissions (ensure the key has read access to emails). 6. Make a note of the generated API key. You will use it as the `api_token` in your connector's `configuration.json` file. -Note: Resend API keys are shown only once upon creation. Store them securely. +> Note: Resend API keys are shown only once upon creation. Store them securely. ## Pagination + The connector implements Resend's cursor-based pagination system using the `after` parameter. It processes emails in pages and uses the `has_more` flag to determine if additional pages exist. The connector uses the last email ID from each page to fetch subsequent pages. The Resend API returns emails sorted by creation date in descending order (newest first). This ordering is leveraged for efficient incremental syncing as follows: @@ -63,13 +81,15 @@ For the first sync, the connector fetches all emails using pagination and saves Since the Resend API returns emails sorted newest-first without date filtering, the connector reads pages from newest to oldest, stops pagination when encountering a previously synced email, and only upserts new emails to prevent duplicates. This approach ensures efficient incremental syncing without fetching the entire dataset on each run. -Note: Refer to the `sync_emails` function for the implementation. +> Note: Refer to the `sync_emails` function for the implementation. ## Data handling + The connector processes email data from the `/emails` endpoint which contains email metadata including sender, recipients, subject, timestamps, and delivery status. All nested JSON structures are flattened using the `flatten_dict` function to create optimal table schemas. Array fields (such as `to`, `cc`, `bcc`, and `reply_to`) are serialized to JSON strings for storage. This approach ensures compatibility with flat table schemas but requires downstream parsing if you need to access individual array elements. ## Error handling + The connector implements comprehensive error handling strategies. Refer to the `fetch_emails_from_api` function: - HTTP timeout handling with 30-second timeouts @@ -81,6 +101,7 @@ The connector implements comprehensive error handling strategies. Refer to the ` - Graceful error logging without exposing sensitive information ## Tables created + The connector creates a single table named `EMAIL` with the following columns: | Column Name | Type | Description | @@ -98,14 +119,16 @@ The connector creates a single table named `EMAIL` with the following columns: The table uses `id` as the primary key. -Note: Detailed event tracking (such as opens, clicks, individual event timestamps) is not available through the List Emails API. Resend provides this data through webhooks, which is outside the scope of this connector. +> Note: Detailed event tracking (such as opens, clicks, individual event timestamps) is not available through the List Emails API. Resend provides this data through webhooks, which is outside the scope of this connector. ## Additional files + The example includes additional utility files for testing and data population: -- `populate_data.py` - Sends multiple test emails to Resend for testing the connector with sample data. -- `explore_data.py` - Explores the Resend API by fetching emails and displaying response structure. -- `test_connection.py` - Tests the API connection and validates the API token. +- **`populate_data.py`** - Sends multiple test emails to Resend for testing the connector with sample data. +- **`explore_data.py`** - Explores the Resend API by fetching emails and displaying response structure. +- **`test_connection.py`** - Tests the API connection and validates the API token. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/rethink_db/README.md b/rethink_db/README.md index fd75d50..fec0c40 100644 --- a/rethink_db/README.md +++ b/rethink_db/README.md @@ -1,9 +1,11 @@ # RethinkDB Connector Example ## Connector overview + This example demonstrates how to use the Fivetran Connector SDK to sync data from RethinkDB, an open-source database designed for real-time applications. RethinkDB is known for its changefeeds feature that pushes data to applications in real-time, making it ideal for collaborative apps, multiplayer games, streaming analytics, and IoT systems. This connector enables organizations to replicate their RethinkDB data to a data warehouse for analysis, reporting, and business intelligence. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -11,9 +13,21 @@ This example demonstrates how to use the Fivetran Connector SDK to sync data fro - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template rethink_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Automatic schema discovery that detects all tables and their primary keys in the RethinkDB database - Intelligent incremental sync using timestamp fields (updated_at, modified_at, timestamp, created_at) - Automatic fallback to full table sync for tables without timestamp fields @@ -23,6 +37,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Connection pooling and proper resource cleanup ## Configuration file + The connector requires the following configuration parameters: ```json @@ -45,18 +60,20 @@ Configuration parameters: - `password`: Password for authentication (optional, leave empty for unauthenticated connections) - `use_ssl`: Enable SSL/TLS encryption for secure connections (true or false, defaults to false) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector requires the `rethinkdb` package to connect to RethinkDB databases. ``` rethinkdb ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector supports both authenticated and unauthenticated connections to RethinkDB. Authentication is configured in the `connect_to_rethinkdb()` function. For authenticated connections: @@ -73,9 +90,11 @@ SSL/TLS connections: 2. The connector uses Python's default SSL context for secure connections. ## Pagination + This connector does not use pagination. RethinkDB cursors automatically stream data from tables without requiring pagination logic. The connector processes records sequentially using cursor iteration (refer to the `sync_table_data()` function at line 281), with checkpointing every 100 records to handle large datasets efficiently. ## Data handling + The connector processes RethinkDB data as follows: Schema discovery: The `schema()` function dynamically discovers all tables in the database and retrieves their primary key definitions using `table.info()`. This ensures the connector adapts to schema changes automatically. @@ -89,6 +108,7 @@ Streaming: The connector uses RethinkDB cursors to stream data from tables, avoi Checkpointing: Progress is checkpointed every 100 records (configured via `__CHECKPOINT_INTERVAL`) to enable resumable syncs in case of interruptions. The state also tracks the maximum timestamp for each table to support incremental syncs. Refer to the `sync_table_data()` function. ## Error handling + Error handling is implemented throughout the connector: Connection errors: The `connect_to_rethinkdb()` function catches connection failures and raises descriptive errors to help diagnose network or authentication issues. @@ -111,7 +131,8 @@ The connector dynamically creates tables based on the schema discovered in your | `COMMENTS` | `id` | Comments on tasks with reactions stored as JSON objects | | `ACTIVITY_LOG` | `id` | Real-time activity tracking with action details stored as JSON | -Note: The connector automatically detects the primary key for each table. Complex data types like arrays and nested objects are converted to JSON strings for storage in the destination warehouse. +> Note: The connector automatically detects the primary key for each table. Complex data types like arrays and nested objects are converted to JSON strings for storage in the destination warehouse. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/s3_csv_validation/README.md b/s3_csv_validation/README.md index f7aa51a..34649ca 100644 --- a/s3_csv_validation/README.md +++ b/s3_csv_validation/README.md @@ -1,6 +1,7 @@ # S3 CSV File Reader with Data Validation Connector Example ## Connector overview + This connector demonstrates how to retrieve a CSV file from an Amazon S3 bucket, validate its contents, and sync the records into a destination table using the Fivetran Connector SDK. Key features include: @@ -14,20 +15,30 @@ This example is ideal for: - Handling semi-structured input with field-type enforcement. - Ingesting flat-file data from cloud object storage. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template s3_csv_validation +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Connects to Amazon S3 using `boto3`. - Reads a CSV file and loads it into a DataFrame. - Validates: @@ -41,8 +52,8 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se - Skips and logs invalid rows. - Upserts valid rows into a table called `DATA`. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -55,10 +66,10 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` @@ -66,32 +77,32 @@ boto3==1.35.97 pandas==2.2.3 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -Authentication is handled via standard AWS credentials passed into the `boto3.client()` constructor. +Authentication is handled via standard AWS credentials passed into the `boto3.client()` constructor. ## Pagination -Not Applicable for this connector, as it processes a single CSV file. +Not applicable for this connector, as it processes a single CSV file. ## Data handling + - CSV is read into a `pandas.DataFrame`. - Each row is validated field-by-field. - If any validation fails, the row is skipped. - Valid rows are upserted using `op.upsert()`. - ## Error handling + - Rows with invalid fields are skipped and logged using `log.warning()`. - Errors include specific reasons (e.g., invalid format, missing value). - The `print_error_message()` function outputs detailed validation failures. - Sync state is checkpointed after processing all rows. - ## Tables created + The connector creates a `DATA` table: ```json @@ -110,6 +121,6 @@ The connector creates a `DATA` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/sam_gov/README.md b/sam_gov/README.md index ad01c99..8c8c558 100644 --- a/sam_gov/README.md +++ b/sam_gov/README.md @@ -1,9 +1,11 @@ # SAM.gov Opportunities Connector Example ## Connector overview + This connector fetches government contracting opportunities from the SAM.gov (System for Award Management) API. It replicates opportunity data, including solicitations, awards, contact information, and related metadata. The connector supports incremental synchronization and handles large datasets through pagination. Data is organized into multiple tables with proper foreign key relationships to maintain referential integrity while providing a normalized structure for analysis. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -11,9 +13,21 @@ This connector fetches government contracting opportunities from the SAM.gov (Sy - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sam_gov +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Configurable date range sync of government contracting opportunities from SAM.gov - Pagination support for large datasets (up to 1000 records per API call) - Upsert-based sync captures updates to existing opportunities within the configured date window @@ -24,6 +38,7 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Support for all SAM.gov opportunity types (solicitations, awards, notices, etc.) ## Configuration file + The configuration requires your SAM.gov API key and date range for opportunity posting dates. The API key can be obtained from your SAM.gov account. ```json @@ -35,17 +50,20 @@ The configuration requires your SAM.gov API key and date range for opportunity p ``` ### Configuration parameters + - `api_key`: Your SAM.gov public API key (required) - `posted_from`: Start date for initial sync in MM/dd/yyyy format (required for first sync) - `posted_to`: End date for initial sync in MM/dd/yyyy format (required for first sync) ### Important date range limitation + - The date range between `posted_from` and `posted_to` must be less than 1 year (maximum 364 days) - This is a SAM.gov API limitation, not a connector limitation - The connector will validate this requirement and throw an error if exceeded during initial sync - For incremental syncs exceeding this limit, the connector automatically chunks the date range into 364-day windows ### Sync strategy + The connector supports hybrid sliding window incremental sync: Initial sync (first run): @@ -58,12 +76,14 @@ Incremental sync (subsequent runs): - If the date range exceeds one year, the connector automatically chunks the range into successive 364-day windows and processes them all in a single sync run, ensuring no data is missed even after extended downtime ### Overlap window strategy + - Default 30-day overlap window ensures recent opportunity updates are captured - The overlap window is configured in the code as a constant named __INCREMENTAL_WINDOW_DAYS with a value of 30 - A smaller overlap makes the sync more API-efficient but reduces the time window in which updates are detected - A larger overlap captures more late or out-of-order updates but increases the number of API calls ### Example progression + ``` Initial Sync: [01/01/2024 - 12/30/2024] (364 days) → State saves: last_posted_to = 12/30/2024 @@ -79,20 +99,28 @@ Sync 3 (long downtime - ~2 years): [02/14/2025 - 12/17/2026] (30-day overlap, 67 → State saves: last_posted_to = 12/17/2026 ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. + +## Requirements file + +The connector uses only the pre-installed packages in the Fivetran environment and does not require a `requirements.txt` file. + +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The SAM.gov API uses API key authentication. To obtain an API key: -1. Create an account on [SAM.gov](https://sam.gov) -2. Navigate to your Account Details page -3. Enter your account password to access API key information -4. Generate a new public API key -5. Copy the API key (it's visible until you navigate away from the page) +1. Create an account on [SAM.gov](https://sam.gov). +2. Navigate to your Account Details page. +3. Enter your account password to access API key information. +4. Generate a new public API key. +5. Copy the API key (it's visible until you navigate away from the page). The API key should be included in every request as the `api_key` parameter. Different user roles (federal vs non-federal) have different rate limits. ## Pagination + The connector implements pagination using the SAM.gov API's `limit` and `offset` parameters (refer to the `fetch_opportunities_page` function and the pagination loop in the `update` function): - Page size: Maximum 1000 records per request - Pagination logic: Processes data page by page in a loop, tracking offset position @@ -100,9 +128,10 @@ The connector implements pagination using the SAM.gov API's `limit` and `offset` - Checkpointing: Saves progress every 100 records to handle interruptions gracefully ## Data handling + The connector processes SAM.gov opportunity data through several transformation steps (refer to the `process_main_opportunity_record` and `process_breakout_tables` functions): -1. Main opportunity table - Stores core opportunity information with flattened nested objects +1. Main opportunity table - Stores core opportunity information with flattened nested objects. 2. Breakout tables - Separate tables for array data with foreign key relationships: - `point_of_contact` - Contact information for each opportunity - `naics_code` - NAICS classification codes @@ -120,6 +149,7 @@ Data transformation (refer to the `process_main_opportunity_record` function): - Foreign key relationships maintained through `notice_id` ## Error handling + The connector implements comprehensive error handling with retry logic and detailed error messages (refer to the `make_api_request`, `make_api_request_with_retry`, and `validate_configuration` functions): Configuration validation errors: @@ -142,37 +172,44 @@ Additional features: - Detailed logging - Comprehensive error tracking for troubleshooting ## Tables created + The connector creates the following tables (refer to the `schema` function): ### opportunity + - Primary key: `notice_id` - Description: Core opportunity information with flattened nested objects - Key columns: title, solicitation_number, posted_date, type, naics_code, response_dead_line, organization_type, office_address fields, place_of_performance fields, award fields - Note: `award_amount` field is defined as DECIMAL(15,2) for financial precision ### point_of_contact + - Primary key: `notice_id`, `contact_index` - Description: Contact information for each opportunity - Foreign key: `notice_id` references opportunity table - Key columns: type, full_name, email, phone, fax, title ### naics_code + - Primary key: `notice_id`, `code_index` - Description: NAICS classification codes for each opportunity - Foreign key: `notice_id` references opportunity table - Key columns: naics_code ### link + - Primary key: `notice_id`, `link_index` - Description: API reference links - Foreign key: `notice_id` references opportunity table - Key columns: rel, href ### resource_link + - Primary key: `notice_id`, `link_index` - Description: Document and resource URLs - Foreign key: `notice_id` references opportunity table - Key columns: resource_link ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/sap_ariba/README.md b/sap_ariba/README.md index 1789241..895a005 100644 --- a/sap_ariba/README.md +++ b/sap_ariba/README.md @@ -1,21 +1,35 @@ # SAP Ariba Purchase Orders Connector Example ## Connector overview + This example demonstrates how to use the Fivetran Connector SDK to extract purchase order and line-item data from the SAP Ariba API. The connector retrieves purchase order headers and details, processes them into row-oriented records, and loads them into a Fivetran destination. It also supports paging, timestamp normalization, and retry logic. This connector uses the SAP Ariba Sandbox API for illustration, but it can be adapted for production tenants by updating the configuration values. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sap_ariba +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Fetches purchase order and item data from the SAP Ariba API. - Handles pagination using `$top` and `$skip` query parameters. - Tracks sync progress using state and checkpoints. @@ -24,38 +38,47 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Performs idempotent data loading using `op.upsert`. ## Configuration file + The connector reads configuration values from `configuration.json`, which defines the authentication values required to access SAP Ariba. -Example: +```json { "api_key": "" } +``` Key descriptions: - `api_key` (required) – SAP Ariba API key used for authentication. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file lists additional Python dependencies used by the connector. This example does not require any external libraries beyond those preinstalled in the Fivetran execution environment. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. + ## Authentication + The connector authenticates using a single API key passed through HTTP headers. The API key is supplied by the `api_key` field in `configuration.json` and is applied to all API requests. Refer to `make_api_request()` in `connector.py`. ## Pagination + The SAP Ariba API uses the `$top` and `$skip` query parameters to return paginated data. The connector increments the `$skip` value after every page and continues making requests until the API returns no additional records. Refer to `sync_rows()` in `connector.py`. ## Data handling + The connector retrieves, parses, normalizes, and loads SAP Ariba data into destination tables. Every record is delivered using `op.upsert`, and large sync operations periodically call `op.checkpoint(state)` to save progress. Timestamp fields returned in Ariba date format are converted to ISO 8601 using the `convert_to_iso` helper. Refer to `update()` and `filter_columns()` in `connector.py`. ## Error handling -The connector includes automatic retry logic for transient errors and rate-limited responses. Error handling is performed in `make_api_request()`, which uses backoff logic, raises authentication errors, and logs unexpected responses. + +The connector includes automatic retry logic for transient errors and rate-limited responses. Error handling is performed in `make_api_request()`, which uses backoff logic, raises authentication errors, and logs unexpected responses. ## Tables created @@ -112,4 +135,5 @@ The connector creates the `ORDER` and `ITEM` tables in the destination from SAP - `last_updated_at` (UTC_DATETIME): Timestamp of last sync ## Additional considerations -This example demonstrates how to work with timestamp conversion, pagination, and API retry patterns. It may require modifications before being used in production environments. For assistance, contact Fivetran Support. + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/sap_hana_sql/README.md b/sap_hana_sql/README.md index 5cb2f6a..ca7b000 100644 --- a/sap_hana_sql/README.md +++ b/sap_hana_sql/README.md @@ -6,7 +6,7 @@ This connector extracts data from a SAP HANA database using the `hdbcli`. The co ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) @@ -14,7 +14,17 @@ This connector extracts data from a SAP HANA database using the `hdbcli`. The co ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sap_hana_sql +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -38,7 +48,7 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -48,7 +58,7 @@ This connector requires the `hdbcli` library to connect to SAP HANA databases. hdbcli==2.25.29 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -68,8 +78,8 @@ The connector uses a batch-based approach for data retrieval rather than traditi This approach accomplishes the following: 1. Prevents memory overflow when handling large datasets. -2. Enables incremental processing of data without loading the entire result set -3. Allows for checkpointing progress after each batch +2. Enables incremental processing of data without loading the entire result set. +3. Allows for checkpointing progress after each batch. Refer to the `fetch_and_upsert()` function, specifically the `cursor.fetchmany(batch_size)` implementation. @@ -101,4 +111,4 @@ This connector replicates the `sales` table which contains sales information wit ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/sendcloud/README.md b/sendcloud/README.md index 3496686..a1dd85c 100644 --- a/sendcloud/README.md +++ b/sendcloud/README.md @@ -16,6 +16,16 @@ This connector integrates with the Sendcloud API to extract shipment data includ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sendcloud +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Incremental data synchronization using cursor-based pagination @@ -45,35 +55,35 @@ Configuration parameters: - `start_date`: ISO date to begin synchronization in YYYY-MM-DD format (required) - `use_mock_server`: Set to "true" to use Stoplight mock server for testing, "false" for production API (optional, defaults to "false") -Note: Ensure that [`configuration.json`](configuration.json) file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require any additional Python packages beyond the pre-installed packages in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -This connector uses HTTP Basic Authentication to connect to the Sendcloud API. The credentials are specified in the configuration file and encoded as Base64 in the Authorization header (refer to the `get_auth_header()` function in [`connector.py`](connector.py)). +This connector uses HTTP Basic Authentication to connect to the Sendcloud API. The credentials are specified in the configuration file and encoded as Base64 in the Authorization header (refer to the `get_auth_header()` function in `connector.py`). To set up authentication: 1. Log in to your Sendcloud account at https://account.sendcloud.com. 2. Navigate to **Settings > Integrations > API**. 3. Generate or retrieve your API public key (`username`) and secret key (`password`). -4. Add the `username` and `password` to the [`configuration.json`](configuration.json) file. +4. Add the `username` and `password` to the `configuration.json` file. 5. Set the `start_date` to begin synchronization from a specific date. ## Pagination -The connector implements cursor-based pagination to handle large datasets efficiently (refer to the `update()` and `determine_next_cursor()` functions in [`connector.py`](connector.py)). +The connector implements cursor-based pagination to handle large datasets efficiently (refer to the `update()` and `determine_next_cursor()` functions in `connector.py`). The Sendcloud API returns pagination metadata in the response with a next_cursor field. The connector extracts this cursor and uses it in subsequent requests to fetch the next page. If the API does not provide an explicit cursor and the page is full, the connector generates a cursor using the encode_cursor function as a fallback mechanism. Pagination continues until no more records are returned or the API indicates there are no more pages. ## Data handling -The connector processes shipment data through multiple stages (refer to the `flatten_shipment_data()` and `process_shipment_arrays()` functions in [`connector.py`](connector.py)): +The connector processes shipment data through multiple stages (refer to the `flatten_shipment_data()` and `process_shipment_arrays()` functions in `connector.py`): - Main shipment fields are flattened into the primary shipments table - Nested single objects (addresses, prices, delivery dates) are flattened into the same table with prefixed column names @@ -83,7 +93,7 @@ The connector processes shipment data through multiple stages (refer to the `fla ## Error handling -The connector implements comprehensive error handling with automatic retry logic (refer to the `fetch_shipments_page()` function in [`connector.py`](connector.py)): +The connector implements comprehensive error handling with automatic retry logic (refer to the `fetch_shipments_page()` function in `connector.py`): - HTTP errors with status codes 400, 401, 403, 404 are treated as permanent failures and fail immediately without retry - Transient errors (500, 502, 503, 504, timeouts, connection errors) trigger exponential backoff retry logic up to 3 attempts @@ -96,7 +106,7 @@ The connector implements comprehensive error handling with automatic retry logic The connector creates the following tables in the destination: -| Table Name | Primary Key | Description | Key Columns | +| Table name | Primary key | Description | Key columns | |---------------------|-----------------------------------------------------------|---------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------| | shipment | `id` | Main table with flattened shipment information | order_number, from_name, from_address_line_1, to_name, to_address_line_1, ship_with_type, total_order_price_value, total_order_price_currency | | shipment_parcel | `shipment_id`, `parcel_id` | Parcels with tracking and dimensional information | tracking_number, tracking_url, status_message, status_code, dimensions_length, dimensions_width, dimensions_height, weight_value, weight_unit | diff --git a/sensor_tower/README.md b/sensor_tower/README.md index c639b85..368f632 100644 --- a/sensor_tower/README.md +++ b/sensor_tower/README.md @@ -1,50 +1,63 @@ -# Sensor Tower Connector SDK Example +# Sensor Tower Connector Example + +## Connector overview This example demonstrates how to build a Fivetran Connector SDK integration for [Sensor Tower](https://sensortower.com), a market intelligence and analytics platform that provides insights into mobile apps, app store trends, and digital advertising. The connector pulls data from the Sensor Tower API for three tables: `SALES_REPORT_ESTIMATES`, `ACTIVE_USERS`, and `RETENTION`. You can configure the connector to track specific iOS and Android app IDs to gather the necessary information for your use case. ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows 10 or later - * macOS 13 (Ventura) or later - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sensor_tower +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -* Supports three Sensor Tower endpoints: `SALES_REPORT_ESTIMATES`, `ACTIVE_USERS`, and `RETENTION` (see `ENDPOINTS` in `connector.py`) -* Handles both iOS and Android app IDs (see `IOS_APP_IDS` and `ANDROID_APP_IDS`) -* Incremental syncs with configurable lookback window (see `update` function) -* Schema mapping and column renaming for clarity (see `key_mapping` and `process_sales_report`) -* Modular data processing for each endpoint (see `process_active_users`, `process_sales_report`, and `process_retention`) +- Supports three Sensor Tower endpoints: `SALES_REPORT_ESTIMATES`, `ACTIVE_USERS`, and `RETENTION` (see `ENDPOINTS` in `connector.py`) +- Handles both iOS and Android app IDs (see `IOS_APP_IDS` and `ANDROID_APP_IDS`) +- Incremental syncs with configurable lookback window (see `update` function) +- Schema mapping and column renaming for clarity (see `key_mapping` and `process_sales_report`) +- Modular data processing for each endpoint (see `process_active_users`, `process_sales_report`, and `process_retention`) ## Configuration file The connector expects a `configuration.json` file with the following structure: -``` +```json { "auth_token": "YOUR_SENSOR_TOWER_API_TOKEN" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file The `requirements.txt` file specifies any additional Python libraries required by the connector. For this example, no extra dependencies are required beyond the Fivetran environment defaults. -*Example content of `requirements.txt`:* +Example content of `requirements.txt`: ``` -requests -dateutil +python_dateutil==2.9.0.post0 +pytz==2024.2 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -56,14 +69,14 @@ This example does not implement explicit pagination because the Sensor Tower API ## Data handling -* Data is retrieved from the Sensor Tower API using the `get_data` function. -* Each endpoint is processed by a dedicated function: - * `process_active_users` (see lines ~90-102) - * `process_sales_report` (see lines ~104-115) - * `process_retention` (see lines ~117-129) -* Data is mapped and transformed as needed (e.g., column renaming for `sales_report_estimates` via `key_mapping`). -* All records are delivered to Fivetran using the `op.upsert` operation. -* The schema is defined in the `schema` function (see lines ~44-66). +- Data is retrieved from the Sensor Tower API using the `get_data` function. +- Each endpoint is processed by a dedicated function: + - `process_active_users` (see lines ~90-102) + - `process_sales_report` (see lines ~104-115) + - `process_retention` (see lines ~117-129) +- Data is mapped and transformed as needed (e.g., column renaming for `sales_report_estimates` via `key_mapping`). +- All records are delivered to Fivetran using the `op.upsert` operation. +- The schema is defined in the `schema` function (see lines ~44-66). ## Error handling @@ -71,19 +84,15 @@ This example does not implement custom error handling. Any HTTP or API errors wi ## Tables created -* `sales_report_estimates` – Contains app sales and revenue estimates by date and country. -* `active_users` – Tracks active user counts by app, date, time period, and country. -* `retention` – Provides retention metrics by app, date, and country. - -*Sample data structure:* - -| App ID | Date | Country Code | iPhone Downloads | iPhone Revenue | ... | -|--------|------------|--------------|------------------|---------------|-----| -| 123456 | 2024-05-01 | US | 1000 | 5000 | ... | +- `sales_report_estimates` – Contains app sales and revenue estimates by date and country. +- `active_users` – Tracks active user counts by app, date, time period, and country. +- `retention` – Provides retention metrics by app, date, and country. -## Additional files +Sample data structure: -This example does not include additional files beyond the main connector script and configuration. +| App ID | Date | Country code | iPhone downloads | iPhone revenue | ... | +|--------|------------|--------------|------------------|----------------|-----| +| 123456 | 2024-05-01 | US | 1000 | 5000 | ... | ## Additional considerations diff --git a/sensource/README.md b/sensource/README.md index 953cd25..29d1e36 100644 --- a/sensource/README.md +++ b/sensource/README.md @@ -14,26 +14,40 @@ The connector supports: The connector processes data in 30-day chunks to handle large historical datasets efficiently and uses checkpointing to ensure reliable resumption after interruptions. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sensource +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features -- OAuth2 Authentication: Secure API access using client credentials flow -- Static Data Sync: Reference data for locations, sites, zones, spaces, and sensors -- Traffic Metrics: Entry and exit counts with hourly granularity -- Occupancy Metrics: Maximum, minimum, and average occupancy with hourly granularity -- Incremental Syncing: Date-based cursor tracking for efficient data replication -- Retry Logic: Automatic retry with exponential backoff for 5xx errors -- Configurable Metrics: Customizable traffic and occupancy metrics via configuration -- Configurable Start Date: Support for specific start dates in YYYY-MM-DD format + +- OAuth2 authentication: Secure API access using client credentials flow +- Static data sync: Reference data for locations, sites, zones, spaces, and sensors +- Traffic metrics: Entry and exit counts with hourly granularity +- Occupancy metrics: Maximum, minimum, and average occupancy with hourly granularity +- Incremental syncing: Date-based cursor tracking for efficient data replication +- Retry logic: Automatic retry with exponential backoff for 5xx errors +- Configurable metrics: Customizable traffic and occupancy metrics via configuration +- Configurable start date: Support for specific start dates in YYYY-MM-DD format ## Configuration file + The configuration file contains the following keys for SenSource API access: ```json @@ -47,23 +61,25 @@ The configuration file contains the following keys for SenSource API access: } ``` -Configuration paramaters: +Configuration parameters: - `client_id` (required): OAuth2 client ID from SenSource. - `client_secret` (required): OAuth2 client secret from SenSource. - `traffic_metrics` (optional): Comma-separated list of traffic metrics (default: "ins,outs"). - `occupancy_metrics` (optional): Comma-separated list of occupancy metrics (default: "occupancy(max),occupancy(min),occupancy(avg)"). - `static_endpoints` (optional): Comma-separated list of static endpoints to sync (default: "location,site,zone,space"). -- `start_date` (optional): Start date for historical data collection in YYYY-MM-DD format (default: "2022-01-01"). +- `start_date` (optional): Start date for historical data collection in YYYY-MM-DD format (default: "2022-01-01"). -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies the Python libraries required by the connector. For the SenSource connector, no additional dependencies are needed as it only uses the pre-installed `fivetran_connector_sdk` and `requests` packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses an OAuth2 client credentials flow for authentication with the SenSource API. To obtain credentials: 1. Contact SenSource to get your OAuth2 client ID and client secret. @@ -72,39 +88,42 @@ The connector uses an OAuth2 client credentials flow for authentication with the For detailed authentication examples and API reference, see [SenSource's API documentation](https://vea.sensourceinc.com/api-docs/). -**Authentication Flow** - Refer to `get_access_token()` function +Authentication flow - Refer to `get_access_token()` function ## Pagination + The SenSource API returns all data for a given date range in a single response, so no traditional pagination is required. However, the connector implements date-based chunking to handle large historical datasets efficiently. -**Date Range Processing** - Refer to `generate_date_ranges()` function and the main update loop in `update()` function +Date range processing - Refer to `generate_date_ranges()` function and the main update loop in `update()` function ## Data handling + The connector processes data in the following way: -1. **Static Data**: Fetches reference data (locations, sites, zones, spaces, sensors) without state tracking -2. **Traffic Data**: Fetches hourly traffic metrics by zone with configurable metrics -3. **Occupancy Data**: Fetches hourly occupancy metrics by space with configurable metrics -4. **State Management**: Tracks the last synced date to enable incremental updates -5. **Data Transformation**: Converts API responses to standardized format for Fivetran +1. Static data: Fetches reference data (locations, sites, zones, spaces, sensors) without state tracking. +2. Traffic data: Fetches hourly traffic metrics by zone with configurable metrics. +3. Occupancy data: Fetches hourly occupancy metrics by space with configurable metrics. +4. State management: Tracks the last synced date to enable incremental updates. +5. Data transformation: Converts API responses to standardized format for Fivetran. -**Data Processing** - Refer to `fetch_data()` function and `fetch_static_data()` function +Data processing - Refer to `fetch_data()` function and `fetch_static_data()` function ## Error handling + The connector implements comprehensive error handling: -- **Authentication Errors**: Validates OAuth2 token acquisition and provides clear error messages -- **API Request Errors**: Implements retry logic with exponential backoff for 5xx errors -- **Configuration Validation**: Ensures all required configuration parameters are present -- **Data Processing Errors**: Graceful handling of malformed API responses +- Authentication errors: Validates OAuth2 token acquisition and provides clear error messages +- API request errors: Implements retry logic with exponential backoff for 5xx errors +- Configuration validation: Ensures all required configuration parameters are present +- Data processing errors: Graceful handling of malformed API responses -**Error Handling** - Refer to `make_request_with_retry()` function and configuration validation in `update()` function +Error handling - Refer to `make_request_with_retry()` function and configuration validation in `update()` function ## Tables created The connector creates the following tables: -| Table | Primary Key | Description | +| Table | Primary key | Description | |-------|-------------|-------------| | `TRAFFIC` | `zone_id`, `record_date_hour_1` | Hourly traffic metrics by zone | | `OCCUPANCY` | `space_id`, `record_date_hour_1` | Hourly occupancy metrics by space | @@ -114,10 +133,12 @@ The connector creates the following tables: | `SPACE` | `space_id` | Space reference data | | `SENSOR` | `sensor_id` | Sensor reference data | -**Schema Definition** - Refer to `schema()` function +Schema definition - Refer to `schema()` function ## Additional files + This connector uses the standard file structure with no additional modular files. All functionality is contained within the main `connector.py` file. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/similarweb/README.md b/similarweb/README.md index 8380bc0..fd20536 100644 --- a/similarweb/README.md +++ b/similarweb/README.md @@ -1,38 +1,50 @@ -# Similarweb Connector SDK Example +# Similarweb Connector Example + +## Connector overview This example demonstrates how to build a Fivetran Connector SDK integration for [Similarweb](https://www.similarweb.com/), a digital intelligence platform that provides data, insights, and analytics about websites and apps. The connector pulls web traffic data from Similarweb's API for a configurable set of domains and countries, and delivers it to your Fivetran destination. ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template similarweb +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -* Supports batch report generation and download from Similarweb's API (see `request_report`, `check_report_status`, `download_report`) -* Configurable list of domains, countries, and metrics (see `DOMAIN_LIST`, `COUNTRY_LIST`, `__METRIC_LIST`) -* Handles both historical and incremental syncs (see `update` function) -* Delivers data to a single table: `ALL_TRAFFIC_VISITS` -* Uses Fivetran Connector SDK logging for status and error reporting (see `log` usage) +- Supports batch report generation and download from Similarweb's API (see `request_report`, `check_report_status`, `download_report`) +- Configurable list of domains, countries, and metrics (see `DOMAIN_LIST`, `COUNTRY_LIST`, `__METRIC_LIST`) +- Handles both historical and incremental syncs (see `update` function) +- Delivers data to a single table: `ALL_TRAFFIC_VISITS` +- Uses Fivetran Connector SDK logging for status and error reporting (see `log` usage) ## Configuration file The connector expects a `configuration.json` file with the following structure: -``` +```json { "api_key": "YOUR_SIMILARWEB_API_KEY" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -45,7 +57,7 @@ pytz dateutil ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -57,20 +69,20 @@ This connector does not implement explicit pagination. Data is retrieved in batc ## Data handling -* Data is requested from Similarweb using the `request_report` function -* The status of the report is checked using `check_report_status` -* Once ready, the report is downloaded and parsed using `download_report` -* Data is delivered to Fivetran using the `op.upsert` operation in the `update` function -* The schema is defined in the `schema` function +- Data is requested from Similarweb using the `request_report` function +- The status of the report is checked using `check_report_status` +- Once ready, the report is downloaded and parsed using `download_report` +- Data is delivered to Fivetran using the `op.upsert` operation in the `update` function +- The schema is defined in the `schema` function ## Error handling -* Uses Fivetran Connector SDK logging for info, warning, and severe error messages (see `log` usage throughout) -* Raises `RuntimeError` for failed report creation, failed status, or download errors (see `request_report`, `check_report_status`, `download_report`) +- Uses Fivetran Connector SDK logging for info, warning, and severe error messages (see `log` usage throughout) +- Raises `RuntimeError` for failed report creation, failed status, or download errors (see `request_report`, `check_report_status`, `download_report`) ## Tables created -* `all_traffic_visits` – Contains daily web traffic metrics for each domain and country combination. +- `all_traffic_visits` – Contains daily web traffic metrics for each domain and country combination. Sample data structure: @@ -78,10 +90,6 @@ Sample data structure: |---------------------|---------|------------|----------|-----| | example-website.com | US | 2024-05-01 | 1000000 | ... | -## Additional files - -This example does not include additional files beyond the main connector script and configuration. - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/smartsheets/README.md b/smartsheets/README.md index 6988185..6536413 100644 --- a/smartsheets/README.md +++ b/smartsheets/README.md @@ -1,22 +1,33 @@ # Smartsheet API Connector Example ## Connector overview -This connector demonstrates how to sync row-level data from Smartsheet using the Fivetran Connector SDK, and the Smartsheet [Sheets API](https://smartsheet.redoc.ly/tag/sheets) and [Reports API](https://smartsheet.redoc.ly/tag/reports). It retrieves rows from sheets and reports, dynamically maps columns using column IDs, and emits those rows to destination tables. +This connector demonstrates how to sync row-level data from Smartsheet using the Fivetran Connector SDK, and the Smartsheet [Sheets API](https://smartsheet.redoc.ly/tag/sheets) and [Reports API](https://smartsheet.redoc.ly/tag/reports). It retrieves rows from sheets and reports, dynamically maps columns using column IDs, and emits those rows to destination tables. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template smartsheets +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Incremental syncs using `rowsModifiedSince` to sync only updated rows - Dynamic schema based on column headers, mapping `columnId` to column names - Optional row-level metadata fields like `rowNumber`, `createdAt`, and `modifiedAt` @@ -24,8 +35,8 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se - Multiple sheets and reports in a single connector instance - Uses `op.upsert()` for each row and checkpoints with the current sync timestamp - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -37,40 +48,40 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` requests ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -Authentication is handled via a Bearer token using the Authorization header. +Authentication is handled via a Bearer token using the Authorization header. ## Pagination -Pagination is not currently implemented in this example. +Pagination is not currently implemented in this example. ## Data handling + - Each row in the Smartsheet is returned as a dictionary of cell values, mapped to the correct column titles. - Column names are dynamically derived from the `columns` array in the API response. - Null or empty cells are ignored. - ## Error handling + - API errors are surfaced using `requests.raise_for_status()`. - Any failure in authentication, network, or sheet access will raise an exception. - You can extend this with `try/except` blocks and log warnings for partial errors. - ## Tables created + The connector creates a `SMARTSHEET_TABLE_NAME` table: ```json @@ -83,6 +94,6 @@ The connector creates a `SMARTSHEET_TABLE_NAME` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/snipeitapp/README.md b/snipeitapp/README.md index f629299..6fec964 100644 --- a/snipeitapp/README.md +++ b/snipeitapp/README.md @@ -16,6 +16,16 @@ This connector demonstrates how to integrate Snipe-IT Asset Management data with Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template snipeitapp +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features - Incremental syncing based on `updated_at` timestamps for efficient data updates @@ -42,13 +52,13 @@ The connector requires the following configuration parameters: - `api_token` (required): Your Snipe-IT Bearer token for API authentication with read permissions for all resources you want to sync - `base_url` (required): The base URL of your Snipe-IT instance (e.g., `https://your-snipeit-instance.com` for cloud or `http://localhost:8000` for self-hosted) -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file This connector does not require any additional Python packages beyond the standard library and the pre-installed SDK packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -93,7 +103,7 @@ The connector implements comprehensive error-handling strategies (refer to the ` The connector creates the following tables in your destination (refer to the `schema()` function): -| Table | Description | Primary Key | Key Columns | +| Table | Description | Primary key | Key columns | |-------|-------------|-------------|-------------| | hardware | Primary asset/hardware items tracked in Snipe-IT | `id` | `id`, `asset_tag`, `serial`, `name`, `model_id`, `model_name`, `status_label_id`, `status_label_name`, `category_id`, `category_name`, `manufacturer_id`, `manufacturer_name`, `supplier_id`, `location_id`, `location_name`, `created_at_datetime`, `updated_at_datetime`, `purchase_date_date`, `purchase_cost` | | users | Users who can be assigned assets or manage the system | `id` | `id`, `username`, `email`, `first_name`, `last_name`, `employee_num`, `jobtitle`, `phone`, `department_id`, `department_name`, `company_id`, `company_name`, `activated`, `created_at_datetime`, `updated_at_datetime` | diff --git a/solace/README.md b/solace/README.md index 1695641..eb6ed4b 100644 --- a/solace/README.md +++ b/solace/README.md @@ -2,33 +2,41 @@ ## Connector overview -This connector demonstrates how to sync data from a **Solace** queue using the [Fivetran Connector SDK](https://fivetran.com/docs/connectors/connector-sdk). It fetches messages from a durable Solace queue using the **Solace Messaging API**, processes the events, and upserts them into a destination table. The connector supports **incremental syncs** using message timestamps and checkpointing to ensure continuity. +This connector demonstrates how to sync data from a Solace queue using the [Fivetran Connector SDK](https://fivetran.com/docs/connectors/connector-sdk). It fetches messages from a durable Solace queue using the Solace Messaging API, processes the events, and upserts them into a destination table. The connector supports incremental syncs using message timestamps and checkpointing to ensure continuity. ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template solace +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features -- Connects to **Solace PubSub+** using the [Solace PubSub+ Python API](https://solace.dev). -- Pulls events from a **durable exclusive queue**. -- Supports **incremental data syncing** using timestamp-based filtering. +- Connects to Solace PubSub+ using the [Solace PubSub+ Python API](https://solace.dev). +- Pulls events from a durable exclusive queue. +- Supports incremental data syncing using timestamp-based filtering. - Deduplicates messages using an internally generated `event_id`. - Graceful error handling and logging. - Tracks sync state for resumable operations. - Optionally supports publishing test messages for development. - -## Configuration +## Configuration file The connector expects the following configuration in the `configuration.json` file: @@ -42,7 +50,7 @@ The connector expects the following configuration in the `configuration.json` fi } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -53,7 +61,7 @@ pandas==2.3.0 solace-pubsubplus==1.10.0 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -82,7 +90,7 @@ This allows the connector to stream messages efficiently with timeout-based pagi The connector processes events from Solace as follows: - Establishes a connection to the Solace broker using durable queue subscriptions. -- Receives messages one at a time using Solace’s persistent message receiver. +- Receives messages one at a time using Solace's persistent message receiver. - Parses each message payload (expects JSON structure). - Extracts relevant metadata including timestamp, topic, and message ID. - Skips and removes messages older than the last sync timestamp from queue. @@ -101,41 +109,30 @@ Each event includes: - `details`: Optional field from payload. - `processed_at`: Timestamp the message was processed. ---- - ## Error handling This connector includes robust error handling at various stages: -- **Connection errors**: Fail fast with meaningful error messages if connection to the broker fails. -- **Message processing errors**: Malformed or unexpected payloads are logged and skipped without halting the sync. -- **Upsert failures**: Logged per record, allowing the connector to continue processing other events. -- **Timeouts and retries**: Configurable timeout ensures the sync completes even if the queue is empty or slow. - ---- +- Connection errors: Fail fast with meaningful error messages if connection to the broker fails. +- Message processing errors: Malformed or unexpected payloads are logged and skipped without halting the sync. +- Upsert failures: Logged per record, allowing the connector to continue processing other events. +- Timeouts and retries: Configurable timeout ensures the sync completes even if the queue is empty or slow. -## Testing & local debugging +## Tables created -You can run the connector locally with a `configuration.json` file. The connector includes optional logic to publish test messages to a Solace topic: +The connector creates a single table named `solace_events` (refer to the `schema()` function): -```python -def publish_messages_for_testing(config: dict, count: int): - publisher = SolacePublisher( - host=config["solace_host"], - username=config["solace_username"], - password=config["solace_password"], - topic_name="demo/topic", - vpn=config.get("solace_vpn", "default") - ) - - publisher.connect() - publisher.publish_messages(count) -``` - -This is helpful for development and testing. You can remove or comment this logic for production use. +| Column | Type | Description | +|--------|------|-------------| +| `event_id` | STRING | Primary key – hash-based unique identifier derived from topic, timestamp, and payload | +| `timestamp` | STRING | Primary key – event timestamp | +| `message_id` | STRING | Optional message ID from the payload | +| `topic` | STRING | Source Solace topic or queue name | +| `message_payload` | STRING | Raw message body | +| `message_type` | STRING | Message type extracted from payload (default: `"raw"`) | +| `details` | STRING | Optional details field from payload | +| `processed_at` | STRING | Timestamp when the message was processed by the connector | ## Additional considerations The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. - - diff --git a/sql_server/README.md b/sql_server/README.md index 93e9bf9..7021de4 100644 --- a/sql_server/README.md +++ b/sql_server/README.md @@ -1,6 +1,7 @@ # SQL Server Connector Example ## Connector overview + This connector demonstrates how to connect to a Microsoft SQL Server database using `pyodbc` and the Fivetran Connector SDK. It retrieves records from the `EMPLOYEE_DETAILS` table and syncs only those that have changed since the last run, using the `updated_time` column for incremental replication. This example covers: @@ -10,28 +11,38 @@ This example covers: - Batched reads using `fetchmany()`. - State checkpointing after each run. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template sql_server +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Connects to a SQL Server database using `pyodbc`. - Defines a static schema using the `schema()` method. - Uses `updated_time` to identify records modified since the last sync. - Supports batched record processing using `fetchmany()`. - Emits each row with `op.upsert()` and saves progress with `op.checkpoint()`. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -44,41 +55,41 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + This connector requires the following Python packages: ``` pyodbc ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -Authentication is handled via the SQL Server username and password specified in the configuration. +Authentication is handled via the SQL Server username and password specified in the configuration. ## Pagination -The connector uses cursor-based pagination via the `updated_time` column. Batching is implemented using `cursor.fetchmany(batch_size)`. This allows large datasets to be processed incrementally and avoids loading the full result set into memory. +The connector uses cursor-based pagination via the `updated_time` column. Batching is implemented using `cursor.fetchmany(batch_size)`. This allows large datasets to be processed incrementally and avoids loading the full result set into memory. ## Data handling + - The connector only syncs records where `updated_time > last_checkpoint_time`. - All fields are explicitly mapped in the schema. - Sync state is updated to the latest `updated_time` seen in the batch. - ## Error handling + - Database connection errors are caught and logged with descriptive messages. - Queries are validated before execution. - Connections and cursors are closed safely with `try/finally` blocks. - State is checkpointed even if only partial sync is completed. - ## Tables created + The connector creates an `EMPLOYEE_DETAILS` table: ```json @@ -94,6 +105,6 @@ The connector creates an `EMPLOYEE_DETAILS` table: } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/status_cake/README.md b/status_cake/README.md index 3878854..b71b6d0 100644 --- a/status_cake/README.md +++ b/status_cake/README.md @@ -1,32 +1,43 @@ # StatusCake Uptime Monitoring Connector Example - ## Connector overview -This connector fetches uptime monitoring data from [StatusCake](https://app.statuscake.com/), a website monitoring service. It extracts uptime test configurations, historical performance data, downtime periods, and alert information to provide comprehensive monitoring insights. The connector integrates with StatusCake's API to pull data for all configured uptime tests and their associated monitoring metrics, enabling users to analyze website performance and reliability trends in their data warehouse. +This connector fetches uptime monitoring data from [StatusCake](https://app.statuscake.com/), a website monitoring service. It extracts uptime test configurations, historical performance data, downtime periods, and alert information to provide comprehensive monitoring insights. The connector integrates with StatusCake's API to pull data for all configured uptime tests and their associated monitoring metrics, enabling users to analyze website performance and reliability trends in their data warehouse. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template status_cake +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Fetches all uptime test configurations and metadata - Retrieves historical performance data with response times and status codes - Collects downtime period information with start/end times and duration - Extracts alert history with trigger timestamps and status details - Handles API rate limiting with automatic retry logic - Flattens nested JSON structures for optimal data warehouse storage -- Supports full data refresh on each sync - +- Supports full data refresh on each sync ## Configuration file + The configuration requires your StatusCake API key for authentication. You can obtain this from your StatusCake account dashboard under API settings. ```json @@ -35,15 +46,16 @@ The configuration requires your StatusCake API key for authentication. You can o } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies additional Python libraries required by the connector. Following Fivetran best practices, this connector doesn't require additional dependencies. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses Bearer token authentication with StatusCake's API. You need to provide your API key in the configuration file. To obtain an API key: 1. Log into your StatusCake account. @@ -51,23 +63,23 @@ The connector uses Bearer token authentication with StatusCake's API. You need t 3. Create a new API key or copy an existing one. 4. Add the key to your `configuration.json` file as shown above. - ## Data handling -The connector processes data from four main StatusCake API endpoints and flattens complex nested structures into relational table format. Each API response is parsed and nested objects are flattened using underscore notation. Arrays are converted to comma-separated strings for easier analysis. All data is upserted to ensure the latest information is always available. +The connector processes data from four main StatusCake API endpoints and flattens complex nested structures into relational table format. Each API response is parsed and nested objects are flattened using underscore notation. Arrays are converted to comma-separated strings for easier analysis. All data is upserted to ensure the latest information is always available. ## Error handling -The connector implements comprehensive error handling with exponential backoff retry logic for transient failures. API rate limiting (HTTP 429) is handled with automatic delays, and connection errors are retried up to 3 times with increasing delay intervals. +The connector implements comprehensive error handling with exponential backoff retry logic for transient failures. API rate limiting (HTTP 429) is handled with automatic delays, and connection errors are retried up to 3 times with increasing delay intervals. ## Tables created -The connector creates four tables to store different aspects of StatusCake monitoring data: -- **UPTIME_TEST** - Core uptime test configurations and current status information -- **UPTIME_TEST_HISTORY** - Historical performance data with response times and status codes -- **UPTIME_TEST_PERIOD** - Downtime and uptime period records with durations -- **UPTIME_TEST_ALERT** - Alert notifications with trigger times and status changes +The connector creates four tables to store different aspects of StatusCake monitoring data: +- `UPTIME_TEST` - Core uptime test configurations and current status information +- `UPTIME_TEST_HISTORY` - Historical performance data with response times and status codes +- `UPTIME_TEST_PERIOD` - Downtime and uptime period records with durations +- `UPTIME_TEST_ALERT` - Alert notifications with trigger times and status changes ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/suitedash/README.md b/suitedash/README.md index 3f1c387..cd87b27 100644 --- a/suitedash/README.md +++ b/suitedash/README.md @@ -1,31 +1,43 @@ # SuiteDash CRM Connector Example ## Connector overview -This example shows how to use Connector SDK to extracts companies, contacts, and their relationships data from the [SuiteDash CRM](https://app.suitedash.com/secure-api), handling proper data flattening and pagination. It's designed for organizations using SuiteDash CRM who want to analyze their customer and prospect data in their data warehouse. +This example shows how to use Connector SDK to extract companies, contacts, and their relationships data from the [SuiteDash CRM](https://app.suitedash.com/secure-api), handling proper data flattening and pagination. It's designed for organizations using SuiteDash CRM who want to analyze their customer and prospect data in their data warehouse. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template suitedash +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -- **Companies sync**: Extracts all company records from the `/companies` endpoint. -- **Contacts sync**: Extracts all contact records from the `/contacts` endpoint. -- **Relationship mapping**: Creates a separate table for contact-company relationships data. -- **Data flattening**: Flattens nested JSON objects (primaryContact, category, address) into flat table columns. -- **Array handling**: Converts tags and circles arrays to comma-separated strings. -- **Pagination support**: Handles SuiteDash's next-page-URL pagination automatically. -- **Error handling**: Robust error handling for API failures and network issues. +- Companies sync: Extracts all company records from the `/companies` endpoint. +- Contacts sync: Extracts all contact records from the `/contacts` endpoint. +- Relationship mapping: Creates a separate table for contact-company relationships data. +- Data flattening: Flattens nested JSON objects (primaryContact, category, address) into flat table columns. +- Array handling: Converts tags and circles arrays to comma-separated strings. +- Pagination support: Handles SuiteDash's next-page-URL pagination automatically. +- Error handling: Robust error handling for API failures and network issues. ## Configuration file + The configuration file defines the authentication credentials required to connect to the SuiteDash API. ```json @@ -35,16 +47,17 @@ The configuration file defines the authentication credentials required to connec } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file -The requirements.txt file specifies additional Python libraries required by the connector. Following Fivetran best practices, this connector doesn't require additional dependencies. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +This connector uses only the pre-installed packages in the Fivetran environment and does not require any additional dependencies. + +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses SuiteDash's secure API authentication mechanism which requires: - `X-Public-ID`: Your SuiteDash account public ID (UUID format) - `X-Secret-Key`: Your SuiteDash secret key @@ -55,8 +68,8 @@ To obtain these credentials: 3. Generate or copy your Public ID and Secret Key. 4. Configure these values in your `configuration.json` file. - ## Pagination + The connector handles SuiteDash's pagination using the next-page-URL mechanism. The API returns pagination metadata in the response: ```json @@ -76,29 +89,28 @@ The connector handles SuiteDash's pagination using the next-page-URL mechanism. The pagination logic is implemented in the `sync_endpoint_with_pagination()` function. - ## Data handling + The connector processes and transforms SuiteDash data as follows: -**Data flattening**: Nested objects are flattened using the `flatten_nested_object()` function: +Data flattening: Nested objects are flattened using the `flatten_nested_object()` function: - `primaryContact` → `primaryContact_uid`, `primaryContact_first_name`, etc. - `category` → `category_name`, `category_color`, `category_isDefault` - `address` → `address_address_line_1`, `address_city`, `address_state`, etc. -**Array processing**: +Array processing: - `tags` arrays are converted to comma-separated strings - `circles` arrays are converted to comma-separated strings - `companies` arrays in contacts create separate relationship records -**Data types**: The connector lets Fivetran infer data types automatically, only defining primary keys in the schema. - +Data types: The connector lets Fivetran infer data types automatically, only defining primary keys in the schema. ## Error handling -Error handling is implemented throughout the connector: -- **Configuration validation**: The `validate_configuration()` function ensures required credentials are present -- **API request handling**: The `make_api_request()` function catches HTTP errors and network issues -- **Exception propagation**: All functions raise `RuntimeError` with descriptive messages for upstream handling +Error handling is implemented throughout the connector: +- Configuration validation: The `validate_configuration()` function ensures required credentials are present +- API request handling: The `make_api_request()` function catches HTTP errors and network issues +- Exception propagation: All functions raise `RuntimeError` with descriptive messages for upstream handling ## Tables created @@ -110,6 +122,6 @@ The connector creates the following tables in your destination: | `CONTACT` | `uid` | Contains flattened contact data from the `/contacts` endpoint | | `CONTACT_COMPANY_RELATIONSHIP` | `contact_uid`, `company_uid` | Junction table mapping contacts to their associated companies | - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/supabase/README.md b/supabase/README.md index 78fc410..85f0cfb 100644 --- a/supabase/README.md +++ b/supabase/README.md @@ -1,21 +1,33 @@ -# Supabase Connector Example +# Supabase Connector Example ## Connector overview -This connector demonstrates how to fetch employee data from a Supabase database and sync it to Fivetran using the Fivetran Connector SDK with advanced batch processing capabilities. The connector retrieves employee records from a Supabase table using efficient batch processing with `.range(start, end)` method and performs incremental synchronization based on the hire_date field. It handles Supabase's 1000 record limit by using configurable batch sizes and processes each batch completely before requesting the next one. The connector connects to Supabase using the Python Supabase client library and handles data extraction, transformation, and loading into the destination with proper checkpointing strategies. +This connector demonstrates how to fetch employee data from a Supabase database and sync it to Fivetran using the Fivetran Connector SDK with advanced batch processing capabilities. The connector retrieves employee records from a Supabase table using efficient batch processing with `.range(start, end)` method and performs incremental synchronization based on the hire_date field. It handles Supabase's 1000 record limit by using configurable batch sizes and processes each batch completely before requesting the next one. The connector connects to Supabase using the Python Supabase client library and handles data extraction, transformation, and loading into the destination with proper checkpointing strategies. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template supabase +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Batch processing using Supabase's `.range(start, end)` method for efficient data retrieval - Handles Supabase's record limit by using configurable batch sizes (default: 1000 records per batch) - Processes each batch completely before requesting the next batch for optimal performance @@ -28,8 +40,8 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Error handling and logging for debugging - Connection to Supabase using official Python SDK with ClientOptions - ## Configuration file + The configuration requires your Supabase project URL and API key to establish a connection to your Supabase database. Optionally, you can also specify the schema name and table name. Below is a breakdown of the available configuration parameters: - `supabase_url` (required) - Your Supabase project URL. @@ -38,30 +50,30 @@ The configuration requires your Supabase project URL and API key to establish a - `table_name` (optional) - Table name to sync (defaults to `employee` if not specified otherwise). - `batch_size` (optional): Number of records to fetch per batch (defaults to `1000`) -``` +```json { -"supabase_url": "", -"supabase_key": "", -"schema_name": "", -"table_name": "", -"batch_size": "" + "supabase_url": "", + "supabase_key": "", + "schema_name": "", + "table_name": "", + "batch_size": "" } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies the Python libraries required by the connector. The main dependency is the Supabase Python client library. ``` supabase ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector uses API key authentication to access Supabase. To authenticate, you need to provide the following: - Your Supabase project URL - Found in your Supabase project dashboard - Supabase API key - Use the anon/public key for read-only access to your database @@ -72,14 +84,14 @@ To obtain these credentials: 3. Go to **Settings > API**. 4. Make a note of the Project URL and anon public key. You will need it later to set up your Connector SDK connector. - ## Pagination -The connector handles data retrieval using Supabase's built-in query capabilities with explicit ordering by `hire_date`. The connector fetches data in chronological order (ascending by `hire_date`) to ensure proper incremental sync behavior. -For large datasets, the connector implements checkpointing every 1000 rows to ensure the sync can resume from the correct position if interrupted. The explicit sorting ensures that checkpoints always capture the correct state for resuming syncs. +The connector handles data retrieval using Supabase's built-in query capabilities with explicit ordering by `hire_date`. The connector fetches data in chronological order (ascending by `hire_date`) to ensure proper incremental sync behavior. +For large datasets, the connector implements checkpointing every 1000 rows to ensure the sync can resume from the correct position if interrupted. The explicit sorting ensures that checkpoints always capture the correct state for resuming syncs. ## Data handling + The connector fetches data from the specified Supabase table and performs the following operations: - Queries records where `hire_date` is greater than the last synced `hire_date` - Explicitly orders results by `hire_date` ascending to ensure consistent incremental syncs @@ -88,24 +100,24 @@ The connector fetches data from the specified Supabase table and performs the fo - Upserts each record to the destination table (using configurable table name) - Updates state with the latest `hire_date` for subsequent syncs - ## Error handling + The connector implements comprehensive error handling strategies: -- Connection validation for Supabase client creation -- Configuration validation to ensure required parameters are present -- Data retrieval error handling with detailed logging +- Connection validation for Supabase client creation +- Configuration validation to ensure required parameters are present +- Data retrieval error handling with detailed logging - Runtime exception handling in the main update loop with proper error propagation - ## Tables created + The connector creates a table in the destination based on your configuration: - Table name - Configurable via `table_name` parameter - Primary key - `id` (INT - maps to Supabase int8) - Fields - All fields from the source table (e.g., `id`, `first_name`, `last_name`, `email`, `department`, `hire_date`) -Note: Data types are automatically inferred by Fivetran except for the primary key. - +> Note: Data types are automatically inferred by Fivetran except for the primary key. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/talon_one/README.md b/talon_one/README.md index 9cc747d..151f73a 100644 --- a/talon_one/README.md +++ b/talon_one/README.md @@ -6,18 +6,27 @@ This connector fetches event data from Talon.one using the Management API. The c The connector maintains one table, `event`. This table contains application event data from Talon.one. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -- Operating System: -- Windows 10 or later -- macOS 13 (Ventura) or later -- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template talon_one +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -34,35 +43,35 @@ The connector requires configuration with your Talon.one credentials and sync se ```json { - "base_url": "", + "base_url": "", "api_key": "", "application_id": "" } ``` -> NOTE: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file The connector uses minimal external dependencies. The `requirements.txt` file should be empty because all required packages are pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication This connector uses Talon.one Management API authentication: -- API Key authentication using `ManagementKey-v1` format +- API key authentication using `ManagementKey-v1` format - Requires a valid Talon.one Management API key - API key must be included in the Authorization header ## Data handling The connector processes data in the following way: -1. Connects to Talon.one Management API using provided credentials -2. Fetches application events for the configured application ID (currently set to 5) -3. Implements pagination to handle large datasets -4. Converts list-type fields to JSON strings for database compatibility -5. Uses incremental sync based on `createdAfter` parameter +1. Connects to Talon.one Management API using provided credentials. +2. Fetches application events for the configured application ID (currently set to 5). +3. Implements pagination to handle large datasets. +4. Converts list-type fields to JSON strings for database compatibility. +5. Uses incremental sync based on `createdAfter` parameter. The connector uses incremental sync based on the `created` field to avoid duplicate data and ensure efficient synchronization. @@ -76,11 +85,12 @@ The connector implements error handling for: All errors are logged using the Fivetran logging system for debugging and monitoring. -## Tables Created +## Tables created The connector creates one table: ### event + Primary key: `id` Contains application event data including: @@ -92,4 +102,4 @@ Contains application event data including: ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/teradata/README.md b/teradata/README.md index 70a88ec..d8cedbe 100644 --- a/teradata/README.md +++ b/teradata/README.md @@ -1,25 +1,39 @@ # Teradata Vantage Connector Example ## Connector overview + This connector demonstrates how to use the Fivetran Connector SDK to extract data from a Teradata Vantage database using the `teradatasql` Python library. It fetches employee records from a specified table, incrementally syncs new records based on the `JoiningDate`, and upserts them into a destination table. ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template teradata +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Connects to Teradata Vantage using `teradatasql` - Incremental sync based on `JoiningDate` - Batch data fetching to optimize memory usage ## Configuration file + The connector uses a configuration file to specify connection details and table information. ```json @@ -32,42 +46,44 @@ The connector uses a configuration file to specify connection details and table } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies the Python libraries required by the connector. This connector uses the following library: ``` teradatasql ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. - +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector authenticates to Teradata Vantage using username and password credentials provided in the configuration file. Obtain these credentials from your Teradata administrator. +The connector authenticates to Teradata Vantage using username and password credentials provided in the configuration file. Obtain these credentials from your Teradata administrator. ## Pagination + Data is fetched in batches using the fetchmany method with a configurable batch size (`__BATCH_SIZE`). Refer to method `fetch_and_upsert_data(cursor, configuration, state)`. - ## Data handling + - Data is selected from the source table where `JoiningDate` is greater than the last synced value. - Each row is converted to a dictionary and upserted into the destination table. - The state is updated with the latest `JoiningDate` after each batch. -Refer to method `fetch_and_upsert_data(cursor, configuration, state)`. +Refer to method `fetch_and_upsert_data(cursor, configuration, state)`. ## Error handling + - Configuration validation raises a `ValueError` if required keys are missing. - Connection errors raise a `ConnectionError`. - All exceptions during update are caught and re-raised as `RuntimeError`. - Database connections and cursors are closed in a `finally` block. ## Tables created + - `EMPLOYEE`: Contains columns `EmployeeID` (INT, primary key) and `JoiningDate` (NAIVE_DATE). The destination table schema is as follows: @@ -86,4 +102,5 @@ The destination table schema is as follows: ``` ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/tidb/README.md b/tidb/README.md index 094a7ae..335808b 100644 --- a/tidb/README.md +++ b/tidb/README.md @@ -1,36 +1,52 @@ # TiDB Connector Example ## Connector overview + This example demonstrates a source connector that reads rows from a TiDB database and upserts them into a Fivetran destination using the Connector SDK. It supports incremental replication based on a `created_at` timestamp, vector column parsing (optional), and stores per-table progress in the connector `state`. -The connector is well suited for the following use cases: +The connector is well suited for the following use cases: - Incremental sync of application tables - Vector/embedding export for ML workflows - Incremental change capture for analytics -## Contributor +## Accreditation + This example was contributed by [Nikhil Mankani](https://www.linkedin.com/in/nikhilmankani/). ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template tidb +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Incremental replication driven by `created_at` timestamp. - Configurable table list and primary keys via `TABLES_PRIMARY_KEY_COLUMNS`. - Optional support for vector columns: Parse serialized embeddings into proper JSON lists via `VECTOR_TABLES_DATA`. - Robust error handling with per-table error markers stored in the connector `state`. ## Configuration file + The connector expects a `configuration.json` file when running locally. -``` +```json { "TIDB_HOST": "", "TIDB_USER": "", @@ -42,7 +58,7 @@ The connector expects a `configuration.json` file when running locally. } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. Configuration parameters: @@ -56,7 +72,6 @@ Configuration parameters: - `primary_key_column`: The primary key column name. - `vector_column`: The column containing vector data. - ## Requirements file Include the following dependencies in your `requirements.txt` file: @@ -66,7 +81,7 @@ pytidb>=0.0.11 certifi>=2025.8.3 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -80,17 +95,20 @@ To set up authentication: For details on creating users and granting privileges, see [TiDB documentation](https://docs.pingcap.com/tidb/stable/manage-users-and-privileges). -Note: For production usage, use secure secret storage and avoid checking credentials into source control. +> Note: For production usage, use secure secret storage and avoid checking credentials into source control. ## Pagination + The connector implements offset-based pagination using the limit and offset parameters in the `fetch_and_upsert_data` function, processing data in batches of 50 rows at a time. This approach helps manage memory usage and allows incremental reads of large tables without loading all matching rows into memory at once. If you need to adjust the batch size or use a different pagination strategy (such as keyset pagination), you can modify the relevant logic in `fetch_and_upsert_data`. For tables that require a different incremental column or cursor-based pagination, update the function accordingly. ## Data handling + - Rows are fetched and passed to `process_row` for normalization. - `process_row` ensures naive datetimes are made timezone-aware (UTC) and attempts to parse vector columns into Python lists when `VECTOR_TABLES_DATA` is configured. - The declared schema (the `schema` function) depends on the `TABLES_PRIMARY_KEY_COLUMNS` and optional `VECTOR_TABLES_DATA` configuration for typed JSON columns. ## Error handling + - Query-level failures (for example, missing `created_at` column) are logged, added to `state` under `{table_name}_last_error`, and the connector checkpoints state so operators can inspect errors without losing progress on other tables. See `fetch_and_upsert_data`. - Row-level failures are logged, and a sample of the row is stored in state under `{table_name}_last_row_error_sample` for debugging. See `fetch_and_upsert_data` and `process_row`. - Connection-level failures are recorded in the state under `last_connection_error`, and the exception is raised to allow the runtime to retry according to its backoff policy. See `create_tidb_connection` and `update`. @@ -121,4 +139,5 @@ For vector tables (if configured), the schema includes typed JSON columns: ``` ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/timescale_db/README.md b/timescale_db/README.md index a674ce4..08a1892 100644 --- a/timescale_db/README.md +++ b/timescale_db/README.md @@ -6,19 +6,29 @@ The TimescaleDB connector allows you to extract time-series data and vector data ## Requirements -* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) -* Operating system: - * Windows: 10 or later (64-bit only) - * macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - * Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template timescale_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features -- Connects to TimescaleDb database using `psycopg2` +- Connects to TimescaleDB database using `psycopg2` - Supports incremental data extraction using timestamps with checkpointing - Handles both standard time-series data and vector data - Uses server-side cursors for memory-efficient data streaming @@ -38,7 +48,7 @@ The connector requires the following configuration parameters in your `configura } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file @@ -48,11 +58,11 @@ The connector requires the `psycopg2` library to connect to TimescaleDB. Include psycopg2-binary==2.9.10 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -The connector uses standard `PostgreSQL` authentication with a `username` and `password`. These credentials must be provided in the configuration file. Ensure the database user has appropriate read permissions on the tables you want to sync. +The connector uses standard PostgreSQL authentication with a `username` and `password`. These credentials must be provided in the configuration file. Ensure the database user has appropriate read permissions on the tables you want to sync. ## Pagination @@ -69,7 +79,7 @@ The connector: The connector processes and transforms data as follows (refer to `timescaleclient.py` lines 148-158): -1. Queries data from TimescaleDb filtered by a timestamp column. +1. Queries data from TimescaleDB filtered by a timestamp column. 2. Converts row results to dictionaries using psycopg2's `RealDictCursor`. 3. Converts vector data (iterables) to lists for proper JSON serialization. 4. Tracks the latest timestamp for each table to support incremental syncs. @@ -85,14 +95,14 @@ Error handling is implemented at several levels: ## Tables created -The connector creates two tables in the destination. `sensor_data`contains time-series sensor data and `sensor_embeddings` contains vector embeddings related to sensors. +The connector creates two tables in the destination. `sensor_data` contains time-series sensor data and `sensor_embeddings` contains vector embeddings related to sensors. The schemas for these tables are as follows: ```json { "table": "sensor_data", - "primary_key": ["sensor_id"], + "primary_key": ["sensor_id"], "columns":{ "sensor_id": "INT", "time": "UTC_DATETIME" @@ -115,8 +125,7 @@ The schemas for these tables are as follows: ## Additional files -- `timescaleclient.py` – This file handles the connection and data extraction from `TimescaleDb`, implementing server-side cursors, batch processing, and data serialization. - +- **`timescaleclient.py`** – This file handles the connection and data extraction from TimescaleDB, implementing server-side cursors, batch processing, and data serialization. ## Additional considerations diff --git a/toast/README.md b/toast/README.md index c7b7e2c..b0dd312 100644 --- a/toast/README.md +++ b/toast/README.md @@ -1,10 +1,32 @@ -# Toast Fivetran Connector +# Toast Connector Example + +## Connector overview This is a custom [Fivetran connector](https://fivetran.com/docs/connectors/connector-sdk) implementation to extract and sync data from the [Toast POS API](https://doc.toasttab.com/) into a destination warehouse. Toast is a restaurant management platform providing point-of-sale, labor, menu, and operational data. -For full implementation details, see the [Toast connector example code](https://github.com/fivetran/fivetran-csdk-connectors/edit/main/examples/source_examples/toast/). +For full implementation details, see the [Toast connector example code](https://github.com/fivetran/fivetran-csdk-connectors/tree/main/toast/). + +## Requirements + +- Toast API credentials: `clientId`, `clientSecret`, `userAccessType` +- Domain to connect to (e.g., `api.toasttab.com`) +- A Fernet key (`key`) for encrypting access tokens +- `initialSyncStart` ISO timestamp to define the start of sync window +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +## Getting started + +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template toast +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). ---- +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features @@ -15,14 +37,7 @@ For full implementation details, see the [Toast connector example code](https:// - Supports voids, deletions, and nested child entities - Uses Fernet encryption for token security in state ---- - -## Requirements - -- Toast API credentials: `clientId`, `clientSecret`, `userAccessType` -- Domain to connect to (e.g., `api.toasttab.com`) -- A Fernet key (`key`) for encrypting access tokens -- `initialSyncStart` ISO timestamp to define the start of sync window +## Configuration file Example `configuration.json`: @@ -37,61 +52,66 @@ Example `configuration.json`: } ``` ---- +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. + +## Requirements file + +The connector requires the `cryptography` library for Fernet-based token encryption. Add it to your `requirements.txt`: + +``` +cryptography +``` -## How it works +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +## Data handling The connector performs the following actions for each key aspect: - Authentication: Generates and caches a Toast token using the provided credentials -- Sync Loop: Runs in 30-day time chunks, paginating through endpoints -- Data Normalization: Flattens nested objects and lists using `flatten_dict`, `extract_fields`, and `stringify_lists` -- Upserts and Deletes: Emits operations using `op.upsert()` and `op.delete()` +- Sync loop: Runs in 30-day time chunks, paginating through endpoints +- Data normalization: Flattens nested objects and lists using `flatten_dict`, `extract_fields`, and `stringify_lists` +- Upserts and deletes: Emits operations using `op.upsert()` and `op.delete()` - Checkpointing: Updates state after each window to resume seamlessly ---- +## Error handling -## Data coverage +- Retries on 401 Unauthorized (max 3) +- Skips 403 Forbidden +- Backs off on 429 Too Many Requests +- Skips on 400 and 409 errors with logging + +Uses Fivetran SDK logging levels (`info`, `fine`, `warning`, `severe`) for detailed sync visibility. + +## Tables created The entity-relationship diagram (ERD) below shows how tables are linked in the Toast schema. - Fivetran Toast Connector ERD +Fivetran Toast Connector ERD ### Core tables + - `restaurant` - `job`, `employee`, `shift`, `break`, `time_entry` - `orders`, `orders_check`, `payment` -### Configuration +### Configuration tables + - `menu`, `menu_item`, `menu_group`, `discounts`, `tables`, etc. ### Nested children + - `orders_check_payment`, `orders_check_selection`, `orders_check_selection_modifier`, etc. ### Cash management -- `cash_entry`, `cash_deposit` ---- - - -## Error handling - -- Retries on 401 Unauthorized (max 3) -- Skips 403 Forbidden -- Backs off on 429 Too Many Requests -- Skips on 400 and 409 errors with logging - ---- - -## Logging +- `cash_entry`, `cash_deposit` -Uses Fivetran SDK logging levels (`info`, `fine`, `warning`, `severe`) for detailed sync visibility. +## Additional considerations ---- +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. ## Resources - [Fivetran Connector SDK Docs](https://fivetran.com/docs/connectors/connector-sdk) - [Toast API Reference](https://doc.toasttab.com/) - [Fernet Encryption](https://cryptography.io/en/latest/fernet/) - diff --git a/tulip_interfaces/README.md b/tulip_interfaces/README.md index cb6467a..0913ae3 100644 --- a/tulip_interfaces/README.md +++ b/tulip_interfaces/README.md @@ -6,10 +6,28 @@ This connector syncs data from Tulip to Fivetran destinations. Tulip is a frontl To use this connector, you need access to a Tulip instance and a Tulip API key with tables read scope. +## Requirements + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) +- Operating system: + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + ## Getting started Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template tulip_interfaces +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features This connector provides incremental data replication from Tulip Tables using the Tulip Table API with a two-phase synchronization strategy. It supports: @@ -50,9 +68,14 @@ Configuration parameters: - `sync_from_date` (optional) - ISO 8601 timestamp to start initial sync (defaults to beginning of time if omitted). - `custom_filter_json` (optional) - JSON array of Tulip API filter objects, refer to [Tulip API docs for JSON definition](https://support.tulip.co/apidocs/list-records-of-a-tulip-table#:~:text=false-,filters,-An%20optional%20array) (defaults to empty array if omitted). -Note: The connector automatically excludes Linked Table Fields (`tableLink` type) to reduce database load on the Tulip API. System fields (`id`, `_createdAt`, `_updatedAt`, `_sequenceNumber`) and all non-`tableLink` custom fields are always included. +> Note: The connector automatically excludes Linked Table Fields (`tableLink` type) to reduce database load on the Tulip API. System fields (`id`, `_createdAt`, `_updatedAt`, `_sequenceNumber`) and all non-`tableLink` custom fields are always included. + +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. + +## Requirements file -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +This example does not require a `requirements.txt` file because it uses only packages that are already available in the Fivetran environment. If you add a `requirements.txt` file for customization, leave it empty unless you introduce additional external dependencies. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication @@ -66,7 +89,7 @@ This connector uses HTTP Basic Authentication with Tulip API credentials: The connector passes the API key and secret as HTTP Basic Auth credentials to all Tulip API endpoints. -Refer to the `def schema()` and `def update()` funtions for implementation details. +Refer to the `def schema()` and `def update()` functions for implementation details. ## Pagination @@ -75,6 +98,7 @@ The connector implements a two-phase cursor-based synchronization strategy to ha Refer to `def update()` function for implementation details. ### Phase 1: BOOTSTRAP mode (historical data load) + - Uses `_sequenceNumber` as primary cursor to avoid offset pagination overhead - Filters: `_sequenceNumber > last_sequence` (and optionally `_updatedAt > sync_from_date`) - Sorts by `_sequenceNumber` in ascending order @@ -82,6 +106,7 @@ Refer to `def update()` function for implementation details. - Transitions to INCREMENTAL mode when bootstrap completes ### Phase 2: INCREMENTAL mode (ongoing updates) + - Primary cursor: `_sequenceNumber > last_sequence` - Secondary filter: `_updatedAt > (last_updated_at - 60s)` (60-second lookback) - Sorts by `_sequenceNumber` in ascending order @@ -89,6 +114,7 @@ Refer to `def update()` function for implementation details. - Prevents infinite loops when multiple records share the same `_updatedAt` timestamp ### Checkpointing + - State is checkpointed every 500 records with: `cursor_mode`, `last_sequence`, and `last_updated_at` - Enables resumable syncs if connector fails mid-batch diff --git a/veeva_vault/basic_auth/README.md b/veeva_vault/basic_auth/README.md index c62a8d4..7f27279 100644 --- a/veeva_vault/basic_auth/README.md +++ b/veeva_vault/basic_auth/README.md @@ -1,6 +1,7 @@ # Veeva Vault Connector - Using Basic Authentication Example ## Connector overview + This connector demonstrates how to sync all object types and their records from Veeva Vault using the Fivetran Connector SDK. It leverages the Vault Configuration API to dynamically retrieve schema information and uses VQL (Vault Query Language) to query and paginate through object records. This example supports: @@ -9,33 +10,43 @@ This example supports: - Executing paginated queries with `VQL`. - Upserting object data into a destination table per object type. - Incremental syncing via `modified_date__v` filtering. -- Handling Veeva’s burst rate limits. +- Handling Veeva's burst rate limits. See [Veeva Vault API documentation](https://developer.veevavault.com/api/19.3/) for endpoint details. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started -Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. + +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template veeva_vault/basic_auth +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Dynamically defines schema from `Objecttype` metadata. - Queries records using `VQL` with optional `modified_date__v` filters. -- Supports paginated record sync via Vault’s `next_page` mechanism. +- Supports paginated record sync via Vault's `next_page` mechanism. - Automatically flattens and serializes list fields. - Handles burst limit throttling with cooldown periods. - Uses `op.checkpoint()` to persist per-object sync cursors. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -46,42 +57,42 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file -This connector does not require any additional Python packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +This connector does not require any additional Python packages. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication -This connector uses Basic Auth with `Base64` encoding of the `username:password` string. +This connector uses Basic Auth with `Base64` encoding of the `username:password` string. ## Pagination + Pagination is handled via: - `next_page` in the responseDetails object -- URL rewriting logic based on Vault’s `/api/v19.3/` URL structure +- URL rewriting logic based on Vault's `/api/v19.3/` URL structure Pagination continues until `next_page` is no longer returned. - ## Data handling + - Each object (e.g., `document__v`, `country__c`) results in a separate destination table. - Fields are flattened and list values are serialized as JSON strings. - Syncs are incremental based on `modified_date__v` (tracked per object). - ## Error handling + - Failed API responses raise errors with descriptive logs. - Burst limit violations result in a 5-minute delay before retrying. - Field flattening ensures no nested structures are lost during sync. - Sync state is saved per object in the form: `{"document__v_cursor": "timestamp"}`. - ## Tables created -The connector creates tables dynamically based on the object types defined in Veeva Vault. Each table corresponds to an object type and includes all fields defined in the `Objecttype` metadata.: + +The connector creates tables dynamically based on the object types defined in Veeva Vault. Each table corresponds to an object type and includes all fields defined in the `Objecttype` metadata. Here is an example of the `DOCUMENT__V` table structure for the `document__v` object type: @@ -93,6 +104,6 @@ Here is an example of the `DOCUMENT__V` table structure for the `document__v` ob } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/veeva_vault/session_id_auth/README.md b/veeva_vault/session_id_auth/README.md index f9e20f1..6ff7ca9 100644 --- a/veeva_vault/session_id_auth/README.md +++ b/veeva_vault/session_id_auth/README.md @@ -1,6 +1,7 @@ # Veeva Vault Connector - Using Session Authentication Example ## Connector overview + This connector demonstrates how to sync all object types and their records from Veeva Vault using the Fivetran Connector SDK. It leverages the Vault Configuration API to dynamically retrieve schema information and uses VQL (Vault Query Language) to query and paginate through object records. This example supports: @@ -9,33 +10,43 @@ This example supports: - Executing paginated queries with `VQL`. - Upserting object data into a destination table per object type. - Incremental syncing via `modified_date__v` filtering. -- Handling Veeva’s burst rate limits. +- Handling Veeva's burst rate limits. See [Veeva Vault API documentation](https://developer.veevavault.com/api/19.3/) for endpoint details. - ## Requirements -- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) + +- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) - ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template veeva_vault/session_id_auth +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. ## Features + - Dynamically defines schema from `Objecttype` metadata. - Queries records using `VQL` with optional `modified_date__v` filters. -- Supports paginated record sync via Vault’s `next_page` mechanism. +- Supports paginated record sync via Vault's `next_page` mechanism. - Automatically flattens and serializes list fields. - Handles burst limit throttling with cooldown periods. - Uses `op.checkpoint()` to persist per-object sync cursors. - ## Configuration file + The connector requires the following configuration parameters: ```json @@ -46,46 +57,46 @@ The connector requires the following configuration parameters: } ``` -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. - +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file -This connector does not require any additional Python packages. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +This connector does not require any additional Python packages. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector uses Session ID authentication. The process involves: - Obtaining a session ID using username and password via the `/api/v24.2/auth` endpoint - Including the session ID in the Authorization header for subsequent API calls - Properly ending the session with a `DELETE` request to `/api/v24.2/session` when the sync is complete - ## Pagination + Pagination is handled via: - `next_page` in the responseDetails object -- URL rewriting logic based on Vault’s `/api/v24.2/` URL structure +- URL rewriting logic based on Vault's `/api/v24.2/` URL structure Pagination continues until `next_page` is no longer returned. - ## Data handling + - Each object (e.g., `document__v`, `country__c`) results in a separate destination table. - Fields are flattened and list values are serialized as JSON strings. - Syncs are incremental based on `modified_date__v` (tracked per object). - ## Error handling + - Failed API responses raise errors with descriptive logs. - Burst limit violations result in a 5-minute delay before retrying. - Field flattening ensures no nested structures are lost during sync. - Sync state is saved per object in the form: `{"document__v_cursor": "timestamp"}`. - ## Tables created -The connector creates tables dynamically based on the object types defined in Veeva Vault. Each table corresponds to an object type and includes all fields defined in the `Objecttype` metadata.: + +The connector creates tables dynamically based on the object types defined in Veeva Vault. Each table corresponds to an object type and includes all fields defined in the `Objecttype` metadata. Here is an example of the `DOCUMENT__V` table structure for the `document__v` object type: @@ -97,6 +108,6 @@ Here is an example of the `DOCUMENT__V` table structure for the `document__v` ob } ``` - ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/vercel/README.md b/vercel/README.md index 2d2d407..2882898 100644 --- a/vercel/README.md +++ b/vercel/README.md @@ -1,26 +1,41 @@ # Vercel Connector Example ## Connector overview + This connector demonstrates how to fetch deployment data from [Vercel](https://vercel.com/) and upsert it into your destination using the Fivetran Connector SDK. The custom connector synchronizes deployment records from your personal Vercel account and implements pagination handling to efficiently process large datasets with incremental synchronization using timestamps. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - - Windows: 10 or later (64-bit only) - - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) - - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) + - Windows: 10 or later (64-bit only) + - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) + - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template vercel +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Synchronizes deployments from Vercel's API `/v6/deployments` endpoint - Supports both personal account and team resource access -- Implements pagination handling for large datasets with configurable limit -- Incremental synchronization using timestamp-based state management +- Implements pagination handling for large datasets with configurable limit +- Incremental synchronization using timestamp-based state management - Comprehensive error handling with retry logic for API requests ## Configuration file + The configuration keys required for your connector are as follows: ```json @@ -35,14 +50,16 @@ The configuration keys required for your connector are as follows: - `api_token` (required): Your Vercel access token for API authentication - `team_id` (optional): Team identifier to access team resources instead of a personal account. If not provided, the connector accesses your personal account resources. -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector uses the `requests` library for HTTP communication, which is pre-installed in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses Vercel access tokens for authentication. To obtain your API token: 1. Go to your [Vercel account settings](https://vercel.com/account/tokens). @@ -52,7 +69,8 @@ The connector uses Vercel access tokens for authentication. To obtain your API t 5. Select an expiration date. 6. Make a note of the generated token. You will use it as the `api_token` in your connector's configuration. -## Team access +### Team access + By default, the connector accesses resources from your personal Vercel account. To access deployments owned by a team: 1. Find your Team ID by navigating to your team settings in the Vercel dashboard or use the Vercel CLI. @@ -62,12 +80,15 @@ By default, the connector accesses resources from your personal Vercel account. When `team_id` is provided, the connector appends `?teamId=[teamID]` to the API endpoint URL to access team resources instead of personal account resources. ## Pagination + The connector implements Vercel's timestamp-based pagination system as described in the API documentation. It processes data in configurable batches and uses the `next` timestamp from the pagination response to fetch subsequent pages. ## Data handling -The connector processes data from the `/v6/deployments` endpoint which contains deployment records with status, timing, and build information. All nested JSON structures are flattened to create optimal table schemas for your destination. + +The connector processes data from the `/v6/deployments` endpoint which contains deployment records with status, timing, and build information. All nested JSON structures are flattened to create optimal table schemas for your destination. ## Error handling + The connector implements comprehensive error handling strategies: - 30-second HTTP timeout handling - Rate limiting detection with exponential backoff @@ -76,7 +97,9 @@ The connector implements comprehensive error handling strategies: - Retry logic for transient errors ## Tables created + The connector creates the `DEPLOYMENT` table, which uses `uid` as the primary key. The table contains deployment records with status, timing, and build information flattened from the `/v6/deployments` endpoint. It includes flattened projectSettings data if available. ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/yugabyte_db/README.md b/yugabyte_db/README.md index 7b09019..3dda78c 100644 --- a/yugabyte_db/README.md +++ b/yugabyte_db/README.md @@ -1,11 +1,13 @@ # YugabyteDB Connector Example ## Connector overview + This connector fetches data from YugabyteDB database and syncs it to Fivetran destinations. YugabyteDB is a distributed SQL database that is PostgreSQL-compatible, making it possible to use standard PostgreSQL drivers for connectivity. The connector automatically discovers all tables in the specified schema, extracts their primary keys, and performs incremental syncs based on the `updated_at` column when available. YugabyteDB is particularly popular for IoT sensor data, real-time analytics, time-series monitoring, and distributed OLTP applications. This example demonstrates syncing IoT sensor data from an industrial equipment monitoring system, which is a common use case for YugabyteDB deployments. ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -13,9 +15,21 @@ YugabyteDB is particularly popular for IoT sensor data, real-time analytics, tim - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template yugabyte_db +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Automatic table discovery from specified schema - Primary key detection for each table using PostgreSQL system catalogs - Incremental sync support using `updated_at` column when available @@ -30,19 +44,21 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - SQL injection prevention using psycopg2.sql identifier quoting ## Configuration file + The connector requires database connection credentials to access YugabyteDB. ```json { - "host": "", - "port": "", + "host": "", + "port": "", "database": "", "user": "", - "password": "", - "schema": "", - "sslmode": "" + "password": "", + "schema": "", + "sslmode": "" } ``` + Configuration parameters: - `host` (required) - Hostname or IP address of your YugabyteDB server. @@ -53,18 +69,20 @@ Configuration parameters: - `schema` (optional) - Database schema to sync (default: `public`). - `sslmode` (optional) - SSL mode (possible values: `require`, `prefer`, `allow`, `disable`). -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The connector requires the `psycopg2_binary` package for PostgreSQL-compatible database connectivity with YugabyteDB. ``` psycopg2_binary==2.9.11 ``` -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + This connector uses username and password authentication to connect to YugabyteDB. The credentials are specified in the configuration file and passed to the `psycopg2.connect()` function (refer to the `create_connection()` function). To set up authentication: @@ -74,6 +92,7 @@ To set up authentication: 3. Ensure the user has `SELECT` permissions on the tables in the schema that need to be synced. ## Data handling + The connector processes data using an optimized streaming approach to manage memory efficiently: - Tables are discovered dynamically from the database schema using `information_schema` - Primary keys are automatically detected for each table using PostgreSQL system catalogs @@ -90,6 +109,7 @@ The connector processes data using an optimized streaming approach to manage mem Refer to the `sync_table()` function for detailed batch processing and checkpointing logic, `build_sync_query()` for SQL query construction, `normalize_record()` for type conversion, and `update()` function for overall state management. ## Error handling + The connector implements comprehensive error handling with specific exception types: - Connection errors - `psycopg2.OperationalError` caught with guidance to check host, port, and credentials - Query errors - `psycopg2.ProgrammingError` caught with guidance about table existence and permissions @@ -101,6 +121,7 @@ The connector implements comprehensive error handling with specific exception ty Refer to the `create_connection()` function for connection error handling and the `update()` function for comprehensive sync error handling. ## Tables created + The connector dynamically replicates all tables found in the specified schema. Each table retains its original name and structure from the source database. Table schemas are automatically inferred by Fivetran, with primary keys explicitly defined during schema discovery. For tables with an `updated_at` column, incremental syncing is enabled. Tables without this column perform full syncs on each run. @@ -115,4 +136,5 @@ The connector creates destination tables matching all tables found in your Yugab These will be automatically discovered and replicated to your destination with the same table names and their detected primary keys. ## Additional considerations + The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. diff --git a/zigpoll/README.md b/zigpoll/README.md index 11ae2bc..651610f 100644 --- a/zigpoll/README.md +++ b/zigpoll/README.md @@ -1,8 +1,11 @@ # Zigpoll Connector Example ## Connector overview + This connector retrieves survey response data from [Zigpoll](https://apidocs.zigpoll.com/reference) API and syncs it using Fivetran Connector SDK. The connector fetches data from all accounts accessible via the provided API token. It also handles pagination and incremental syncing to efficiently process large datasets. + ## Requirements + - [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements) - Operating system: - Windows: 10 or later (64-bit only) @@ -10,9 +13,21 @@ This connector retrieves survey response data from [Zigpoll](https://apidocs.zig - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) ## Getting started + Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. +To initialize a new Connector SDK project using this connector as a starting point, run: + +``` +fivetran init --template zigpoll +``` + +`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit). + +> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters. + ## Features + - Fetches survey response data from all accessible Zigpoll accounts - Supports cursor-based pagination for efficient data retrieval - Incremental sync using timestamp-based checkpointing @@ -23,27 +38,30 @@ Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/co - Client-side timestamp filtering to prevent duplicate data processing ## Configuration file + The connector requires an API token to authenticate with Zigpoll's API and optionally accepts a start date for initial sync. Obtain your API token from your Zigpoll account settings. -``` +```json { "api_token": "", "start_date": "" } ``` -Configuration Parameters: +Configuration parameters: - `api_token` (required): Your Zigpoll API authentication token - `start_date` (optional): Date in YYYY-MM-DD format to start syncing data from (e.g., `"2023-01-01"`). If not provided, sync will start from EPOCH time -Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information. +> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information. ## Requirements file + The `requirements.txt` file specifies Python libraries required by the connector. This connector uses only the pre-installed packages in the Fivetran environment. -Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. +> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. ## Authentication + The connector uses API token authentication. To obtain your API token: 1. Log in to your Zigpoll account. 2. Navigate to account settings or API settings. @@ -51,6 +69,7 @@ The connector uses API token authentication. To obtain your API token: 4. Add the token to your `configuration.json` file. ## Pagination + The connector handles cursor-based pagination automatically through modular functions: - `get_account_ids` function retrieves all accessible account IDs - `process_responses_page` function handles individual page processing with client-side filtering @@ -59,6 +78,7 @@ The connector handles cursor-based pagination automatically through modular func The pagination processes responses in batches of 100 records per request and uses the `endCursor` from each response to fetch subsequent pages. The connector includes safeguards against infinite loops by detecting when the API returns the same cursor consecutively, which can occur when using timestamp filters with the Zigpoll API. ## Data handling + Survey response data is processed and transformed through a modular approach: - Primary data structure preserved with `_id` as the primary key - Metadata dictionaries are flattened into separate columns with `metadata_` prefix @@ -68,6 +88,7 @@ Survey response data is processed and transformed through a modular approach: - State management with checkpointing enables incremental syncing and resumability ## Error handling + The connector implements comprehensive error handling through multiple layers: - Configuration validation ensures required parameters are present - API requests include retry logic with exponential backoff for transient errors @@ -77,6 +98,7 @@ The connector implements comprehensive error handling through multiple layers: - Runtime error wrapping for sync failures to provide clear error messages ## Tables created + This connector creates a single table, `RESPONSE`, which contains all survey response data from Zigpoll accounts. | Column | Type | Description | @@ -93,4 +115,5 @@ This connector creates a single table, `RESPONSE`, which contains all survey res | metadata_* | STRING | Flattened metadata fields with prefix | ## Additional considerations -The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. \ No newline at end of file + +The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team.