Skip to content

Commit 89e5501

Browse files
docs: improve README files for batch 01 (apache_hbase → gnews) (#6)
1 parent b19952a commit 89e5501

36 files changed

Lines changed: 1079 additions & 686 deletions

File tree

apache_hbase/README.md

Lines changed: 28 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,25 @@ This connector shows how to sync data from Apache HBase databases using Fivetran
66

77
## Requirements
88

9-
* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10-
* Operating system:
11-
* Windows: 10 or later (64-bit only)
12-
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13-
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
9+
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10+
- Operating system:
11+
- Windows: 10 or later (64-bit only)
12+
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13+
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
1414

1515
## Getting started
1616

17-
Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
17+
Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
18+
19+
To initialize a new Connector SDK project using this connector as a starting point, run:
20+
21+
```
22+
fivetran init --template apache_hbase
23+
```
24+
25+
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26+
27+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
1828
1929
## Features
2030

@@ -26,18 +36,18 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se
2636

2737
## Configuration file
2838

29-
The connector requires the following configuration parameters:
39+
The connector requires the following configuration parameters:
3040

31-
```
41+
```json
3242
{
3343
"hostname": "<YOUR_HBASE_HOSTNAME>",
3444
"port": "<YOUR_HBASE_PORT>",
35-
"table_name": "<YOUR_HBASE_TABLE_NAME>",
45+
"table_name": "<YOUR_HBASE_TABLE_NAME>",
3646
"column_family": "<YOUR_HBASE_COLUMN_FAMILY>"
3747
}
3848
```
3949

40-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
50+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
4151
4252
## Requirements file
4353

@@ -47,7 +57,7 @@ This connector requires the happybase library to communicate with Apache HBase:
4757
happybase==1.2.0
4858
```
4959

50-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
60+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
5161
5262
## Authentication
5363

@@ -61,8 +71,7 @@ Refer to the `execute_query_and_upsert` function, which implements batched data
6171

6272
## Data handling
6373

64-
65-
The connector processes data from HBase in the following way:
74+
The connector processes data from HBase in the following way:
6675
- Creates a connection to the HBase server using the `happybase` library.
6776
- Scans the specified table with a filter based on the `created_at` timestamp.
6877
- Decodes and transforms each row into a structured format
@@ -71,26 +80,25 @@ The connector processes data from HBase in the following way:
7180

7281
## Error handling
7382

74-
The connector implements error handling at multiple levels:
83+
The connector implements error handling at multiple levels:
7584
- Connection errors: Captured in the `create_hbase_connection` function, raising a descriptive RuntimeError
7685
- Data processing errors: The `execute_query_and_upsert` function uses try-except blocks to handle missing columns in row data, logging warnings and continuing execution without failing the entire sync
7786

78-
## Tables Created
87+
## Tables created
7988

80-
81-
The connector creates one table:
89+
The connector creates one table:
8290
- profile_table
8391

8492
The schema of the created table is as follows:
8593

86-
```
94+
```json
8795
{
8896
"table": "profile_table",
8997
"primary_key": ["id"],
9098
"columns": {
9199
"id": "STRING",
92-
"created_at": "UTC_DATETIME",
93-
},
100+
"created_at": "UTC_DATETIME"
101+
}
94102
}
95103
```
96104

apache_hive/using_pyhive/README.md

Lines changed: 23 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,26 @@ This connector shows how to fetch data from Apache Hive using the `PyHive` and `
66

77
## Requirements
88

9-
* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10-
* Operating system:
11-
* Windows: 10 or later (64-bit only)
12-
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13-
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
9+
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10+
- Operating system:
11+
- Windows: 10 or later (64-bit only)
12+
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13+
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
1414

1515
## Getting started
1616

1717
Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
1818

19+
To initialize a new Connector SDK project using this connector as a starting point, run:
20+
21+
```
22+
fivetran init --template apache_hive/using_pyhive
23+
```
24+
25+
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26+
27+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
28+
1929
## Features
2030

2131
- Direct connection to Apache Hive data source.
@@ -26,7 +36,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co
2636

2737
This connector requires the following configuration parameters to establish a connection to your Hive instance:
2838

29-
```
39+
```json
3040
{
3141
"hostname": "YOUR_HIVE_HOSTNAME",
3242
"port": "<YOUR_HIVE_PORT>",
@@ -36,7 +46,7 @@ This connector requires the following configuration parameters to establish a co
3646
}
3747
```
3848

39-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
49+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
4050
4151
## Requirements file
4252

@@ -48,11 +58,11 @@ thrift_sasl
4858
sasl
4959
```
5060

51-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
61+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
5262
5363
## Authentication
5464

55-
The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
65+
The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
5666
- `hostname`: The address of your Hive server
5767
- `port`: The port number Hive is listening on (typically 10000)
5868
- `username`: Your Hive username
@@ -62,25 +72,25 @@ Authentication is handled in the `create_hive_connection` function.
6272

6373
## Data handling
6474

65-
The connector performs the following data handling operations:
75+
The connector performs the following data handling operations:
6676
- Fetching: Data is retrieved from Apache Hive using SQL queries with timestamp-based filtering.
6777
- Processing: The `process_row` function converts raw Hive data into dictionary format suitable for Fivetran.
6878
- Column names are extracted and mapped to their values.
69-
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
79+
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
7080
- State management: The connector tracks the latest created timestamp to enable incremental syncs.
7181

7282
## Tables created
7383

7484
The connector creates a table named `PEOPLE` with the following schema:
7585

76-
```
86+
```json
7787
{
7888
"table": "people",
7989
"primary_key": ["id"],
8090
"columns": {
8191
"id": "INT",
8292
"created_at": "UTC_DATETIME"
83-
},
93+
}
8494
}
8595
```
8696

apache_hive/using_sqlalchemy/README.md

Lines changed: 25 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,26 @@ This connector demonstrates how to fetch data from Apache Hive using `SQLAlchemy
66

77
## Requirements
88

9-
* [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10-
* Operating system:
11-
* Windows: 10 or later (64-bit only)
12-
* macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13-
* Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
9+
- [Supported Python versions](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/README.md#requirements)
10+
- Operating system:
11+
- Windows: 10 or later (64-bit only)
12+
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13+
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
1414

1515
## Getting started
1616

1717
Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
1818

19+
To initialize a new Connector SDK project using this connector as a starting point, run:
20+
21+
```
22+
fivetran init --template apache_hive/using_sqlalchemy
23+
```
24+
25+
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26+
27+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
28+
1929
## Features
2030

2131
- Connection to Apache Hive data source using SQLAlchemy ORM.
@@ -26,16 +36,16 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co
2636

2737
This connector requires the following configuration parameters to establish a connection to your Hive instance:
2838

29-
```
39+
```json
3040
{
3141
"hostname": "<YOUR_HIVE_HOSTNAME>",
3242
"port": "<YOUR_HIVE_PORT>",
3343
"username": "<YOUR_HIVE_USERNAME>",
34-
"database": "<YOUR_HIVE_DATABASE>",
44+
"database": "<YOUR_HIVE_DATABASE>"
3545
}
3646
```
3747

38-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
48+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
3949
4050
## Requirements file
4151

@@ -47,13 +57,14 @@ PyHive==0.7.0
4757
thrift_sasl
4858
sasl
4959
```
60+
5061
`PyHive` is required for actual dialect implementation for Hive along with the `SQLAlchemy` ORM. The `thrift_sasl` and `sasl` packages are necessary for SASL authentication, which is commonly used with Hive.
5162

52-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
63+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages) are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
5364
5465
## Authentication
5566

56-
The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
67+
The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
5768
- `hostname`: The address of your Hive server
5869
- `port`: The port number Hive is listening on (typically 10000)
5970
- `username`: Your Hive username
@@ -63,7 +74,7 @@ Authentication is handled in the `create_hive_connection` function.
6374

6475
## Data handling
6576

66-
The connector performs the following data handling operations:
77+
The connector performs the following data handling operations:
6778
- Fetching: Data is retrieved from Apache Hive using SQLAlchemy with raw SQL queries and stream options.
6879
- Processing: The `process_row` function converts raw Hive data into a dictionary format suitable for Fivetran.
6980
- Column names are extracted and mapped to their values.
@@ -74,17 +85,17 @@ The connector performs the following data handling operations:
7485

7586
The connector creates a table named `PEOPLE` with the following schema:
7687

77-
```
88+
```json
7889
{
7990
"table": "people",
8091
"primary_key": ["id"],
8192
"columns": {
8293
"id": "INT",
8394
"created_at": "UTC_DATETIME"
84-
},
95+
}
8596
}
8697
```
8798

8899
## Additional considerations
89100

90-
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/)..
101+
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/).

0 commit comments

Comments
 (0)