You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13
+
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
14
14
15
15
## Getting started
16
16
17
-
Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
17
+
Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
18
+
19
+
To initialize a new Connector SDK project using this connector as a starting point, run:
20
+
21
+
```
22
+
fivetran init --template apache_hbase
23
+
```
24
+
25
+
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26
+
27
+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
18
28
19
29
## Features
20
30
@@ -26,18 +36,18 @@ Refer to the [Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/se
26
36
27
37
## Configuration file
28
38
29
-
The connector requires the following configuration parameters:
39
+
The connector requires the following configuration parameters:
30
40
31
-
```
41
+
```json
32
42
{
33
43
"hostname": "<YOUR_HBASE_HOSTNAME>",
34
44
"port": "<YOUR_HBASE_PORT>",
35
-
"table_name": "<YOUR_HBASE_TABLE_NAME>",
45
+
"table_name": "<YOUR_HBASE_TABLE_NAME>",
36
46
"column_family": "<YOUR_HBASE_COLUMN_FAMILY>"
37
47
}
38
48
```
39
49
40
-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
50
+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
41
51
42
52
## Requirements file
43
53
@@ -47,7 +57,7 @@ This connector requires the happybase library to communicate with Apache HBase:
47
57
happybase==1.2.0
48
58
```
49
59
50
-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
60
+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages)are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
51
61
52
62
## Authentication
53
63
@@ -61,8 +71,7 @@ Refer to the `execute_query_and_upsert` function, which implements batched data
61
71
62
72
## Data handling
63
73
64
-
65
-
The connector processes data from HBase in the following way:
74
+
The connector processes data from HBase in the following way:
66
75
- Creates a connection to the HBase server using the `happybase` library.
67
76
- Scans the specified table with a filter based on the `created_at` timestamp.
68
77
- Decodes and transforms each row into a structured format
@@ -71,26 +80,25 @@ The connector processes data from HBase in the following way:
71
80
72
81
## Error handling
73
82
74
-
The connector implements error handling at multiple levels:
83
+
The connector implements error handling at multiple levels:
75
84
- Connection errors: Captured in the `create_hbase_connection` function, raising a descriptive RuntimeError
76
85
- Data processing errors: The `execute_query_and_upsert` function uses try-except blocks to handle missing columns in row data, logging warnings and continuing execution without failing the entire sync
- macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64])
13
+
- Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64)
14
14
15
15
## Getting started
16
16
17
17
Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started.
18
18
19
+
To initialize a new Connector SDK project using this connector as a starting point, run:
20
+
21
+
```
22
+
fivetran init --template apache_hive/using_pyhive
23
+
```
24
+
25
+
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26
+
27
+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
28
+
19
29
## Features
20
30
21
31
- Direct connection to Apache Hive data source.
@@ -26,7 +36,7 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co
26
36
27
37
This connector requires the following configuration parameters to establish a connection to your Hive instance:
28
38
29
-
```
39
+
```json
30
40
{
31
41
"hostname": "YOUR_HIVE_HOSTNAME",
32
42
"port": "<YOUR_HIVE_PORT>",
@@ -36,7 +46,7 @@ This connector requires the following configuration parameters to establish a co
36
46
}
37
47
```
38
48
39
-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
49
+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
40
50
41
51
## Requirements file
42
52
@@ -48,11 +58,11 @@ thrift_sasl
48
58
sasl
49
59
```
50
60
51
-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
61
+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages)are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
52
62
53
63
## Authentication
54
64
55
-
The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
65
+
The connector supports `CUSTOM` authentication for Apache Hive. You need to provide:
56
66
-`hostname`: The address of your Hive server
57
67
-`port`: The port number Hive is listening on (typically 10000)
58
68
-`username`: Your Hive username
@@ -62,25 +72,25 @@ Authentication is handled in the `create_hive_connection` function.
62
72
63
73
## Data handling
64
74
65
-
The connector performs the following data handling operations:
75
+
The connector performs the following data handling operations:
66
76
- Fetching: Data is retrieved from Apache Hive using SQL queries with timestamp-based filtering.
67
77
- Processing: The `process_row` function converts raw Hive data into dictionary format suitable for Fivetran.
68
78
- Column names are extracted and mapped to their values.
69
-
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
79
+
- Batching: Data is processed in configurable batches (1000 rows by default) to prevent memory overflow.
70
80
- State management: The connector tracks the latest created timestamp to enable incremental syncs.
71
81
72
82
## Tables created
73
83
74
84
The connector creates a table named `PEOPLE` with the following schema:
`fivetran init` initializes a new Connector SDK project by setting up the project structure, configuration files, and a connector you can run immediately with `fivetran debug`. For more information on `fivetran init`, refer to the [Connector SDK `init` documentation](https://fivetran.com/docs/connector-sdk/connector-development-and-configuration/connector-sdk-commands#fivetraninit).
26
+
27
+
> Note: Ensure you have updated the `configuration.json` file with the necessary parameters before running `fivetran debug`. See the [Configuration file](#configuration-file) section for details on the required configuration parameters.
28
+
19
29
## Features
20
30
21
31
- Connection to Apache Hive data source using SQLAlchemy ORM.
@@ -26,16 +36,16 @@ Refer to the [Connector SDK setup guide](https://fivetran.com/docs/connectors/co
26
36
27
37
This connector requires the following configuration parameters to establish a connection to your Hive instance:
28
38
29
-
```
39
+
```json
30
40
{
31
41
"hostname": "<YOUR_HIVE_HOSTNAME>",
32
42
"port": "<YOUR_HIVE_PORT>",
33
43
"username": "<YOUR_HIVE_USERNAME>",
34
-
"database": "<YOUR_HIVE_DATABASE>",
44
+
"database": "<YOUR_HIVE_DATABASE>"
35
45
}
36
46
```
37
47
38
-
Note: Ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
48
+
> Note: When submitting connector code as a [Community Connector](https://github.com/fivetran/fivetran-csdk-connectors/tree/main) in the open-source [Connector SDK repository](https://github.com/fivetran/fivetran-csdk-connectors/tree/main), ensure the `configuration.json` file has placeholder values. When adding the connector to your production repository, ensure that the `configuration.json` file is not checked into version control to protect sensitive information.
39
49
40
50
## Requirements file
41
51
@@ -47,13 +57,14 @@ PyHive==0.7.0
47
57
thrift_sasl
48
58
sasl
49
59
```
60
+
50
61
`PyHive` is required for actual dialect implementation for Hive along with the `SQLAlchemy` ORM. The `thrift_sasl` and `sasl` packages are necessary for SASL authentication, which is commonly used with Hive.
51
62
52
-
Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
63
+
> Note: [Some packages](https://fivetran.com/docs/connector-sdk/technical-reference#preinstalledpackages)are pre-installed in the Connector SDK runtime environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`.
53
64
54
65
## Authentication
55
66
56
-
The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
67
+
The connector supports authentication for Apache Hive through SQLAlchemy. You need to provide the following:
57
68
-`hostname`: The address of your Hive server
58
69
-`port`: The port number Hive is listening on (typically 10000)
59
70
-`username`: Your Hive username
@@ -63,7 +74,7 @@ Authentication is handled in the `create_hive_connection` function.
63
74
64
75
## Data handling
65
76
66
-
The connector performs the following data handling operations:
77
+
The connector performs the following data handling operations:
67
78
- Fetching: Data is retrieved from Apache Hive using SQLAlchemy with raw SQL queries and stream options.
68
79
- Processing: The `process_row` function converts raw Hive data into a dictionary format suitable for Fivetran.
69
80
- Column names are extracted and mapped to their values.
@@ -74,17 +85,17 @@ The connector performs the following data handling operations:
74
85
75
86
The connector creates a table named `PEOPLE` with the following schema:
76
87
77
-
```
88
+
```json
78
89
{
79
90
"table": "people",
80
91
"primary_key": ["id"],
81
92
"columns": {
82
93
"id": "INT",
83
94
"created_at": "UTC_DATETIME"
84
-
},
95
+
}
85
96
}
86
97
```
87
98
88
99
## Additional considerations
89
100
90
-
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/)..
101
+
The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our [Support team](https://support.fivetran.com/).
0 commit comments