Skip to content

Commit 68ebf15

Browse files
zax1314157zax1314157
authored
[Feature][Connector-V2] Add aerospike sink connector (#8821)
1 parent d3a9ff7 commit 68ebf15

File tree

27 files changed

+1577
-2
lines changed

27 files changed

+1577
-2
lines changed

Diff for: ‎.github/workflows/labeler/label-scope-conf.yml

+5
Original file line numberDiff line numberDiff line change
@@ -306,3 +306,8 @@ sls:
306306
- changed-files:
307307
- any-glob-to-any-file: seatunnel-connectors-v2/connector-sls/**
308308
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(sls)/**'
309+
aerospike:
310+
- all:
311+
- changed-files:
312+
- any-glob-to-any-file: seatunnel-connectors-v2/connector-aerospike/**
313+
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(aerospike)/**'

Diff for: ‎NOTICE

+6-1
Original file line numberDiff line numberDiff line change
@@ -102,4 +102,9 @@ The class com.hazelcast.internal.util.ConcurrentReferenceHashMap contains code w
102102
and updated within the WildFly project (https://github.com/wildfly/wildfly).
103103

104104
The class org.apache.calcite.linq4j.tree.ConstantExpression contains code
105-
originating from the Calcite project (https://github.com/apache/calcite).
105+
originating from the Calcite project (https://github.com/apache/calcite).
106+
107+
Aerospike Sink Connector
108+
Copyright 2023 The original authors.
109+
Contains Aerospike Client Library (https://www.aerospike.com/)
110+
which is licensed under the AGPL 3.0 License (https://www.aerospike.com/terms/download/3rd-party-licenses)

Diff for: ‎docs/en/connector-v2/changelog/connector-aerospike.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
<details><summary> Change Log </summary>
2+
3+
| Change | Commit | Version |
4+
|------------------------------------------| --- | --- |
5+
| [Improve][connector][aerospike] add support sink connector e2e doc dist (#8821) |https://github.com/apache/seatunnel/pull/8821| dev |
6+
</details>

Diff for: ‎docs/en/connector-v2/sink/Aerospike.md

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
import ChangeLog from '../changelog/connector-aerospike.md';
2+
3+
# Aerospike
4+
5+
> Aerospike sink connector
6+
7+
## Support Those Engines
8+
9+
> Spark<br/>
10+
> Flink<br/>
11+
> Seatunnel Zeta<br/>
12+
13+
## License Compatibility Notice
14+
15+
This connector depends on Aerospike Client Library which is licensed under AGPL 3.0.
16+
When using this connector, you need to comply with AGPL 3.0 license terms.
17+
18+
## Key Features
19+
20+
- [ ] [exactly-once](../../concept/connector-v2-features.md)
21+
- [ ] [cdc](../../concept/connector-v2-features.md)
22+
23+
## Description
24+
25+
Sink connector for Aerospike database.
26+
27+
## Supported DataSource Info
28+
29+
| Datasource | Supported Versions | Maven |
30+
|------------|-----------------|----------------------------------------------------------------------------------------|
31+
| Aerospike | 4.4.17+ | [Download](https://mvnrepository.com/artifact/com.aerospike/aerospike-client) |
32+
33+
## Data Type Mapping
34+
35+
| SeaTunnel Data Type | Aerospike Data Type | Storage Format |
36+
|---------------------|---------------------|--------------------------------------------------------------------------------|
37+
| STRING | STRING | Direct string storage |
38+
| INT | INTEGER | 32-bit integer |
39+
| BIGINT | LONG | 64-bit integer |
40+
| DOUBLE | DOUBLE | 64-bit floating point |
41+
| BOOLEAN | BOOLEAN | Stored as true/false values |
42+
| ARRAY | BYTEARRAY | Only support byte array type |
43+
| LIST | LIST | Support generic list types |
44+
| DATE | LONG | Converted to epoch milliseconds |
45+
| TIMESTAMP | LONG | Converted to epoch milliseconds |
46+
47+
Note:
48+
- When using ARRAY type, SeaTunnel's array elements must be byte type
49+
- LIST type supports any element types that can be serialized
50+
- DATE/TIMESTAMP conversion uses system default time zone
51+
52+
## Options
53+
54+
| Name | Type | Required | Default | Description |
55+
|----------------|--------|----------|---------|-----------------------------------------------------------------------------|
56+
| host | string | Yes | - | Aerospike server hostname or IP address |
57+
| port | int | No | 3000 | Aerospike server port |
58+
| namespace | string | Yes | - | Namespace in Aerospike |
59+
| set | string | Yes | - | Set name in Aerospike |
60+
| username | string | No | - | Username for authentication |
61+
| password | string | No | - | Password for authentication |
62+
| key | string | Yes | - | Field name to use as Aerospike primary key |
63+
| bin_name | string | No | - | Bin name for storing data |
64+
| data_format | string | No | string | Data storage format: map/string/kv |
65+
| write_timeout | int | No | 200 | Write operation timeout in milliseconds |
66+
| schema.field | map | No | {} | Field type mappings (e.g. {"name":"STRING","age":"INTEGER"}) |
67+
68+
### data_format Options
69+
- **map**: Store data as JSON map
70+
- **string**: Store data as JSON string
71+
- **kv**: Store each field as separate bin
72+
73+
## Task Example
74+
75+
### Simple Example
76+
77+
```hocon
78+
env {
79+
parallelism = 2
80+
job.mode = "BATCH"
81+
}
82+
83+
source {
84+
FakeSource {
85+
row.num = 10
86+
schema = {
87+
fields {
88+
id = "int"
89+
name = "string"
90+
age = "int"
91+
address = "string"
92+
}
93+
}
94+
}
95+
}
96+
97+
sink {
98+
Aerospike {
99+
host = "localhost"
100+
port = 3000
101+
namespace = "test_namespace"
102+
set = "user_data"
103+
key = "id"
104+
data_format = "map"
105+
write_timeout = 300
106+
schema.field = {
107+
id = "INTEGER"
108+
name = "STRING"
109+
age = "INTEGER"
110+
address = "STRING"
111+
}
112+
}
113+
}
114+
```
115+
## Changelog
116+
117+
<ChangeLog />

Diff for: ‎docs/zh/connector-v2/changelog/connector-aerospike.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
<details><summary> Change Log </summary>
2+
3+
| Change | Commit | Version |
4+
|------------------------------------------| --- | --- |
5+
| [Improve][connector][aerospike] add support sink connector e2e doc dist (#8821) |https://github.com/apache/seatunnel/pull/8821| dev |
6+
</details>

Diff for: ‎docs/zh/connector-v2/sink/Aerospike.md

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
import ChangeLog from '../changelog/connector-aerospike.md';
2+
3+
# Aerospike
4+
5+
> Aerospike 数据写入连接器
6+
7+
## 许可证兼容性通知
8+
9+
此连接器依赖于根据AGPL 3.0许可的Aerospike客户端库。
10+
使用此连接器时，您需要遵守AGPL 3.0许可条款。
11+
12+
## 支持引擎
13+
14+
> Spark<br/>
15+
> Flink<br/>
16+
> Seatunnel Zeta<br/>
17+
18+
## 主要特性
19+
20+
- [ ] [精确一次](../../concept/connector-v2-features.md)
21+
- [ ] [CDC](../../concept/connector-v2-features.md)
22+
23+
## 描述
24+
25+
用于向 Aerospike 数据库写入数据的连接器。
26+
27+
## 支持的数据源
28+
29+
| 数据源 | 支持版本 | Maven 依赖 |
30+
|------------|---|-------------------------------------------------------------------------|
31+
| Aerospike | 4.4.17+ | [下载](https://mvnrepository.com/artifact/com.aerospike/aerospike-client) |
32+
33+
## 数据类型映射
34+
35+
| SeaTunnel 数据类型 | Aerospike 数据类型 | 存储格式 |
36+
|----------------|--------------------|------------------------------------------------------------------------------|
37+
| STRING | STRING | 直接存储字符串 |
38+
| INT | INTEGER | 32位整型 |
39+
| BIGINT | LONG | 64位整型 |
40+
| DOUBLE | DOUBLE | 64位浮点数 |
41+
| BOOLEAN | BOOLEAN | 存储为 true/false 值 |
42+
| ARRAY | BYTEARRAY | 仅支持字节数组类型 |
43+
| LIST | LIST | 支持泛型列表类型 |
44+
| DATE | LONG | 转换为纪元时间毫秒数 |
45+
| TIMESTAMP | LONG | 转换为纪元时间毫秒数 |
46+
47+
注意事项：
48+
- 使用ARRAY类型时，SeaTunnel数组元素必须是byte类型
49+
- LIST类型支持可序列化的任意元素类型
50+
- DATE/TIMESTAMP转换使用系统默认时区
51+
52+
## 配置选项
53+
54+
| 参数名称 | 类型 | 必填 | 默认值 | 说明 |
55+
|----------------|---------|------|---------|---------------------------------------------------------------------|
56+
| host | string | 是 | - | Aerospike 服务器主机名或IP地址 |
57+
| port | int | 否 | 3000 | Aerospike 服务器端口 |
58+
| namespace | string | 是 | - | Aerospike 命名空间 |
59+
| set | string | 是 | - | Aerospike 集合名称 |
60+
| username | string | 否 | - | 认证用户名 |
61+
| password | string | 否 | - | 认证密码 |
62+
| key | string | 是 | - | 用作 Aerospike 主键的字段名称 |
63+
| bin_name | string | 否 | - | 数据存储的 bin 名称 |
64+
| data_format | string | 否 | string | 数据存储格式：map/string/kv |
65+
| write_timeout | int | 否 | 200 | 写入操作超时时间（毫秒） |
66+
| schema.field | map | 否 | {} | 字段类型映射（示例：{"name":"STRING","age":"INTEGER"}） |
67+
68+
### data_format 选项说明
69+
- **map**: 以JSON对象格式存储
70+
- **string**: 以JSON字符串格式存储
71+
- **kv**: 每个字段存储为独立的bin
72+
73+
## 任务示例
74+
75+
### 简单示例
76+
77+
```hocon
78+
env {
79+
parallelism = 2
80+
job.mode = "BATCH"
81+
}
82+
83+
source {
84+
FakeSource {
85+
row.num = 10
86+
schema = {
87+
fields {
88+
id = "int"
89+
name = "string"
90+
age = "int"
91+
address = "string"
92+
}
93+
}
94+
}
95+
}
96+
97+
sink {
98+
Aerospike {
99+
host = "localhost"
100+
port = 3000
101+
namespace = "test_namespace"
102+
set = "user_data"
103+
key = "id"
104+
data_format = "map"
105+
write_timeout = 300
106+
schema.field = {
107+
id = "INTEGER"
108+
name = "STRING"
109+
age = "INTEGER"
110+
address = "STRING"
111+
}
112+
}
113+
}
114+
```
115+
## Changelog
116+
117+
<ChangeLog />

Diff for: ‎plugin-mapping.properties

+1
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,7 @@ seatunnel.sink.Sls = connector-sls
141141
seatunnel.source.Typesense = connector-typesense
142142
seatunnel.sink.Typesense = connector-typesense
143143
seatunnel.source.Opengauss-CDC = connector-cdc-opengauss
144+
seatunnel.sink.Aerospike = connector-aerospike
144145

145146
seatunnel.transform.Sql = seatunnel-transforms-v2
146147
seatunnel.transform.FieldMapper = seatunnel-transforms-v2

Diff for: ‎seatunnel-connectors-v2/connector-aerospike/pom.xml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Licensed to the Apache Software Foundation (ASF) under one or more
5+
contributor license agreements. See the NOTICE file distributed with
6+
this work for additional information regarding copyright ownership.
7+
The ASF licenses this file to You under the Apache License, Version 2.0
8+
(the "License"); you may not use this file except in compliance with
9+
the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing, software
14+
distributed under the License is distributed on an "AS IS" BASIS,
15+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16+
See the License for the specific language governing permissions and
17+
limitations under the License.
18+
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
21+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
22+
<modelVersion>4.0.0</modelVersion>
23+
<parent>
24+
<groupId>org.apache.seatunnel</groupId>
25+
<artifactId>seatunnel-connectors-v2</artifactId>
26+
<version>${revision}</version>
27+
</parent>
28+
29+
<artifactId>connector-aerospike</artifactId>
30+
<name>SeaTunnel : Connectors V2 : Aerospike</name>
31+
32+
<dependencies>
33+
<dependency>
34+
<groupId>org.apache.seatunnel</groupId>
35+
<artifactId>connector-common</artifactId>
36+
<version>${project.version}</version>
37+
</dependency>
38+
<dependency>
39+
<groupId>org.apache.seatunnel</groupId>
40+
<artifactId>seatunnel-format-json</artifactId>
41+
<version>${project.version}</version>
42+
</dependency>
43+
<dependency>
44+
<groupId>com.aerospike</groupId>
45+
<artifactId>aerospike-client</artifactId>
46+
<version>4.4.17</version>
47+
</dependency>
48+
<dependency>
49+
<groupId>com.alibaba</groupId>
50+
<artifactId>fastjson</artifactId>
51+
<version>2.0.33</version>
52+
</dependency>
53+
</dependencies>
54+
</project>

Diff for: ‎seatunnel-connectors-v2/connector-aerospike/src/main/java/org/apache/seatunnel/connectors/seatunnel/aerospike/config/AerospikeDataType.java

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* https://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
package org.apache.seatunnel.connectors.seatunnel.aerospike.config;
19+
20+
public enum AerospikeDataType {
21+
STRING,
22+
INTEGER,
23+
LONG,
24+
DOUBLE,
25+
BOOLEAN,
26+
BYTEARRAY,
27+
LIST
28+
}

0 commit comments

Comments
 (0)