Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
1579f09
docs: improve README files for batch 01
fivetran-JenasVimal Apr 7, 2026
2468cdd
fix: correct fivetran init template paths for nested connector direct…
fivetran-JenasVimal Apr 7, 2026
b51e111
fix: rename aws_rds_oracle/readme.md to README.md for consistency
fivetran-JenasVimal Apr 7, 2026
993fe50
fix: update fivetran init doc link to correct URL
fivetran-JenasVimal Apr 8, 2026
53eee6d
revert: remove template changes from batch 01
fivetran-JenasVimal Apr 8, 2026
d1872d5
Apply suggestions from code review
fivetran-JenasVimal Apr 8, 2026
de204c0
Update aws_dynamo_db_authentication/README.md
fivetran-JenasVimal Apr 8, 2026
4d7ef9a
Added the correct couchbase magma connector link
fivetran-JenasVimal Apr 8, 2026
2c097ec
Added placeholders
fivetran-JenasVimal Apr 8, 2026
26e1be0
Apply suggestions from code review
fivetran-JenasVimal Apr 8, 2026
f90c4bf
Update couchbase_capella/README.md
fivetran-JenasVimal Apr 8, 2026
7fd03f9
Update couchbase_capella/README.md
fivetran-JenasVimal Apr 8, 2026
a601be9
Update couchbase_magma/README.md
fivetran-JenasVimal Apr 8, 2026
82d869a
Update documentdb/README.md
fivetran-JenasVimal Apr 8, 2026
d1298e3
Apply suggestions from code review
fivetran-JenasVimal Apr 8, 2026
2499884
docs: update pre-installed packages note to link
fivetran-JenasVimal Apr 11, 2026
c480463
docs: revert template README change from batch commit
fivetran-JenasVimal Apr 13, 2026
0437029
Apply suggestions from code review
fivetran-JenasVimal Apr 13, 2026
7449da1
Apply suggestions from code review
fivetran-JenasVimal Apr 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 28 additions & 20 deletions apache_hbase/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,25 @@ This connector shows how to sync data from Apache HBase databases using Fivetran

## Requirements

* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
Comment thread
fivetran-sahilkhirwal marked this conversation as resolved.
* Operating system:
* Windows: 10 or later (64-bit only)
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
- Operating system:
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## Getting started

Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template apache_hbase
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
Comment thread
fivetran-sahilkhirwal marked this conversation as resolved.

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

Expand All @@ -26,18 +36,18 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se

## Configuration file

The connector requires the following configuration parameters:
The connector requires the following configuration parameters:

```
```json
{
"hostname": "<YOUR_HBASE_HOSTNAME>",
"port": "<YOUR_HBASE_PORT>",
"table_name": "<YOUR_HBASE_TABLE_NAME>",
"table_name": "<YOUR_HBASE_TABLE_NAME>",
"column_family": "<YOUR_HBASE_COLUMN_FAMILY>"
}
```

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

Expand All @@ -47,7 +57,7 @@ This connector requires the happybase library to communicate with Apache HBase:
happybase==1.2.0
```

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

Expand All @@ -61,8 +71,7 @@ Refer to the `execute_query_and_upsert` function, which implements batched data

## Data handling


The connector processes data from HBase in the following way:
The connector processes data from HBase in the following way:
- Creates a connection to the HBase server using the `happybase` library.
- Scans the specified table with a filter based on the `created_at` timestamp.
- Decodes and transforms each row into a structured format
Expand All @@ -71,26 +80,25 @@ The connector processes data from HBase in the following way:

## Error handling

The connector implements error handling at multiple levels:
The connector implements error handling at multiple levels:
- Connection errors: Captured in the `create_hbase_connection` function, raising a descriptive RuntimeError
- Data processing errors: The `execute_query_and_upsert` function uses try-except blocks to handle missing columns in row data, logging warnings and continuing execution without failing the entire sync

## Tables Created
## Tables created


The connector creates one table:
The connector creates one table:
- profile_table

The schema of the created table is as follows:

```
```json
{
"table": "profile_table",
"primary_key": ["id"],
"columns": {
"id": "STRING",
"created_at": "UTC_DATETIME",
},
"created_at": "UTC_DATETIME"
}
}
```

Expand Down
36 changes: 23 additions & 13 deletions apache_hive/using_pyhive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,26 @@ This connector shows how to fetch data from Apache Hive using the `PyHive` and `

## Requirements

* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
* Operating system:
* Windows: 10 or later (64-bit only)
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
- Operating system:
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## Getting started

Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template apache_hive/using_pyhive
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
Comment thread
fivetran-sahilkhirwal marked this conversation as resolved.

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

- Direct connection to Apache Hive data source.
Expand All @@ -26,7 +36,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co

This connector requires the following configuration parameters to establish a connection to your Hive instance:

```
```json
{
"hostname": "YOUR_HIVE_HOSTNAME",
"port": "<YOUR_HIVE_PORT>",
Expand All @@ -36,7 +46,7 @@ This connector requires the following configuration parameters to establish a co
}
Comment thread
fivetran-JenasVimal marked this conversation as resolved.
```

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

Expand All @@ -48,11 +58,11 @@ thrift_sasl
sasl
```

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
- `hostname`: The address of your Hive server
- `port`: The port number Hive is listening on (typically 10000)
- `username`: Your Hive username
Expand All @@ -62,25 +72,25 @@ Authentication is handled in the `create_hive_connection` function.

## Data handling

The connector performs the following data handling operations:
The connector performs the following data handling operations:
- Fetching: Data is retrieved from Apache Hive using SQL queries with timestamp-based filtering.
- Processing: The `process_row` function converts raw Hive data into dictionary format suitable for Fivetran.
- Column names are extracted and mapped to their values.
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
- State management: The connector tracks the latest created timestamp to enable incremental syncs.

## Tables created

The connector creates a table named `PEOPLE` with the following schema:

```
```json
{
"table": "people",
"primary_key": ["id"],
"columns": {
"id": "INT",
"created_at": "UTC_DATETIME"
},
}
}
```

Expand Down
39 changes: 25 additions & 14 deletions apache_hive/using_sqlalchemy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,26 @@ This connector demonstrates how to fetch data from Apache Hive using `SQLAlchemy

## Requirements

* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
* Operating system:
* Windows: 10 or later (64-bit only)
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
- Operating system:
- Windows: 10 or later (64-bit only)
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)

## Getting started

Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.

To initialize a new Connector SDK project using this connector as a starting point, run:

```
fivetran init --template apache_hive/using_sqlalchemy
```

`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
Comment thread
fivetran-sahilkhirwal marked this conversation as resolved.

> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.

## Features

- Connection to Apache Hive data source using SQLAlchemy ORM.
Expand All @@ -26,16 +36,16 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co

This connector requires the following configuration parameters to establish a connection to your Hive instance:

```
```json
{
"hostname": "<YOUR_HIVE_HOSTNAME>",
"port": "<YOUR_HIVE_PORT>",
"username": "<YOUR_HIVE_USERNAME>",
"database": "<YOUR_HIVE_DATABASE>",
"database": "<YOUR_HIVE_DATABASE>"
}
```

Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.

## Requirements file

Expand All @@ -47,13 +57,14 @@ PyHive==0.7.0
thrift_sasl
sasl
```

`PyHive` is required for actual dialect implementation for Hive along with the `SQLAlchemy` ORM. The `thrift_sasl` and `sasl` packages are necessary for SASL authentication, which is commonly used with Hive.

Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.

## Authentication

The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
- `hostname`: The address of your Hive server
- `port`: The port number Hive is listening on (typically 10000)
- `username`: Your Hive username
Expand All @@ -63,7 +74,7 @@ Authentication is handled in the `create_hive_connection` function.

## Data handling

The connector performs the following data handling operations:
The connector performs the following data handling operations:
- Fetching: Data is retrieved from Apache Hive using SQLAlchemy with raw SQL queries and stream options.
- Processing: The `process_row` function converts raw Hive data into a dictionary format suitable for Fivetran.
- Column names are extracted and mapped to their values.
Expand All @@ -74,17 +85,17 @@ The connector performs the following data handling operations:

The connector creates a table named `PEOPLE` with the following schema:

```
```json
{
"table": "people",
"primary_key": ["id"],
"columns": {
"id": "INT",
"created_at": "UTC_DATETIME"
},
}
}
```

## Additional considerations

The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/)..
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/).
Loading
Loading