Skip to content

Commit eef6588

Browse files
committed
Add lakehouse connector
1 parent 02f549a commit eef6588

34 files changed

+4203
-0
lines changed

Diff for: core/trino-server/src/main/provisio/trino.xml

+6
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,12 @@
160160
</artifact>
161161
</artifactSet>
162162

163+
<artifactSet to="plugin/lakehouse">
164+
<artifact id="${project.groupId}:trino-lakehouse:zip:${project.version}">
165+
<unpack />
166+
</artifact>
167+
</artifactSet>
168+
163169
<artifactSet to="plugin/loki">
164170
<artifact id="${project.groupId}:trino-loki:zip:${project.version}">
165171
<unpack />

Diff for: docs/src/main/sphinx/connector.md

+1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ Iceberg <connector/iceberg>
2525
Ignite <connector/ignite>
2626
JMX <connector/jmx>
2727
Kafka <connector/kafka>
28+
Lakehouse <connector/lakehouse>
2829
Loki <connector/loki>
2930
MariaDB <connector/mariadb>
3031
Memory <connector/memory>

Diff for: docs/src/main/sphinx/connector/lakehouse.md

+94
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Lakehouse connector
2+
3+
The Lakehouse connector combines the features of the
4+
[Hive](/connector/hive), [Iceberg](/connector/iceberg),
5+
[Delta Lake](/connector/delta-lake), and [Hudi](/connector/hudi)
6+
connectors into a single connector. It allows you to query or write
7+
to data stored in multiple table types (also known as table formats)
8+
that all share the same file system and metastore service.
9+
10+
## General configuration
11+
12+
To configure the Lakehouse connector, create a catalog properties file
13+
`etc/catalog/example.properties` with the following content, replacing the
14+
properties as appropriate:
15+
16+
```text
17+
connector.name=lakehouse
18+
```
19+
20+
You must configure a [metastore for metadata](/object-storage/metastores).
21+
The `hive.metastore` property will also configure the Iceberg catalog.
22+
Do not specify `iceberg.catalog.type`.
23+
24+
You must select and configure one of the
25+
[supported file systems](lakehouse-file-system-configuration).
26+
27+
## Configuration properties
28+
29+
The following configuration properties are available:
30+
31+
:::{list-table}
32+
:widths: 30, 58, 12
33+
:header-rows: 1
34+
35+
* - Property name
36+
- Description
37+
- Default
38+
* - `lakehouse.table-type`
39+
- The default table type for newly created tables when the `format`
40+
table property is not specified. Possible values:
41+
* `HIVE`
42+
* `ICEBERG`
43+
* `DELTA`
44+
- `ICEBERG`
45+
:::
46+
47+
(lakehouse-file-system-configuration)=
48+
## File system access configuration
49+
50+
The connector supports accessing the following file systems:
51+
52+
* [](/object-storage/file-system-azure)
53+
* [](/object-storage/file-system-gcs)
54+
* [](/object-storage/file-system-s3)
55+
* [](/object-storage/file-system-hdfs)
56+
57+
You must enable and configure the specific file system access.
58+
59+
## Examples
60+
61+
Create an Iceberg table:
62+
63+
```sql
64+
CREATE TABLE iceberg_table (
65+
c1 INTEGER,
66+
c2 DATE,
67+
c3 DOUBLE
68+
)
69+
WITH (
70+
type = 'ICEBERG'
71+
format = 'PARQUET',
72+
partitioning = ARRAY['c1', 'c2'],
73+
sorted_by = ARRAY['c3']
74+
);
75+
```
76+
77+
Create a Hive table:
78+
79+
```sql
80+
CREATE TABLE hive_page_views (
81+
view_time TIMESTAMP,
82+
user_id BIGINT,
83+
page_url VARCHAR,
84+
ds DATE,
85+
country VARCHAR
86+
)
87+
WITH (
88+
type = 'HIVE',
89+
format = 'ORC',
90+
partitioned_by = ARRAY['ds', 'country'],
91+
bucketed_by = ARRAY['user_id'],
92+
bucket_count = 50
93+
)
94+
```

0 commit comments

Comments
 (0)