|
| 1 | +# Lakehouse connector |
| 2 | + |
| 3 | +The Lakehouse connector combines the features of the |
| 4 | +[Hive](/connector/hive), [Iceberg](/connector/iceberg), |
| 5 | +[Delta Lake](/connector/delta-lake), and [Hudi](/connector/hudi) |
| 6 | +connectors into a single connector. It allows you to query or write |
| 7 | +to data stored in multiple table types (also known as table formats) |
| 8 | +that all share the same file system and metastore service. |
| 9 | + |
| 10 | +## General configuration |
| 11 | + |
| 12 | +To configure the Lakehouse connector, create a catalog properties file |
| 13 | +`etc/catalog/example.properties` with the following content, replacing the |
| 14 | +properties as appropriate: |
| 15 | + |
| 16 | +```text |
| 17 | +connector.name=lakehouse |
| 18 | +``` |
| 19 | + |
| 20 | +You must configure a [metastore for metadata](/object-storage/metastores). |
| 21 | +The `hive.metastore` property will also configure the Iceberg catalog. |
| 22 | +Do not specify `iceberg.catalog.type`. |
| 23 | + |
| 24 | +You must select and configure one of the |
| 25 | +[supported file systems](lakehouse-file-system-configuration). |
| 26 | + |
| 27 | +## Configuration properties |
| 28 | + |
| 29 | +The following configuration properties are available: |
| 30 | + |
| 31 | +:::{list-table} |
| 32 | +:widths: 30, 58, 12 |
| 33 | +:header-rows: 1 |
| 34 | + |
| 35 | +* - Property name |
| 36 | + - Description |
| 37 | + - Default |
| 38 | +* - `lakehouse.table-type` |
| 39 | + - The default table type for newly created tables when the `format` |
| 40 | + table property is not specified. Possible values: |
| 41 | + * `HIVE` |
| 42 | + * `ICEBERG` |
| 43 | + * `DELTA` |
| 44 | + - `ICEBERG` |
| 45 | +::: |
| 46 | + |
| 47 | +(lakehouse-file-system-configuration)= |
| 48 | +## File system access configuration |
| 49 | + |
| 50 | +The connector supports accessing the following file systems: |
| 51 | + |
| 52 | +* [](/object-storage/file-system-azure) |
| 53 | +* [](/object-storage/file-system-gcs) |
| 54 | +* [](/object-storage/file-system-s3) |
| 55 | +* [](/object-storage/file-system-hdfs) |
| 56 | + |
| 57 | +You must enable and configure the specific file system access. |
| 58 | + |
| 59 | +## Examples |
| 60 | + |
| 61 | +Create an Iceberg table: |
| 62 | + |
| 63 | +```sql |
| 64 | +CREATE TABLE iceberg_table ( |
| 65 | + c1 INTEGER, |
| 66 | + c2 DATE, |
| 67 | + c3 DOUBLE |
| 68 | +) |
| 69 | +WITH ( |
| 70 | + type = 'ICEBERG' |
| 71 | + format = 'PARQUET', |
| 72 | + partitioning = ARRAY['c1', 'c2'], |
| 73 | + sorted_by = ARRAY['c3'] |
| 74 | +); |
| 75 | +``` |
| 76 | + |
| 77 | +Create a Hive table: |
| 78 | + |
| 79 | +```sql |
| 80 | +CREATE TABLE hive_page_views ( |
| 81 | + view_time TIMESTAMP, |
| 82 | + user_id BIGINT, |
| 83 | + page_url VARCHAR, |
| 84 | + ds DATE, |
| 85 | + country VARCHAR |
| 86 | +) |
| 87 | +WITH ( |
| 88 | + type = 'HIVE', |
| 89 | + format = 'ORC', |
| 90 | + partitioned_by = ARRAY['ds', 'country'], |
| 91 | + bucketed_by = ARRAY['user_id'], |
| 92 | + bucket_count = 50 |
| 93 | +) |
| 94 | +``` |
0 commit comments