From d92a19ce052d8ec9570882f17a4ba76929284cb5 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 19 Mar 2025 12:18:06 -0700 Subject: [PATCH 01/62] Iceberg Integration --- .../snapshots/snowflake.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 content/shared/influxdb3-query-guides/snapshots/snowflake.md diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md new file mode 100644 index 0000000000..a974c53317 --- /dev/null +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -0,0 +1,152 @@ +# Integrating InfluxDB with Snowflake Using Apache Iceberg + +## Overview + +Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. + +### Key Benefits + +- **Efficient data access**: Query InfluxDB data directly from Snowflake. +- **Cost-effective storage**: Optimize data retention and minimize storage costs. +- **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. + +## Prerequisites + +Before you begin, ensure you have the following: + +- A **Snowflake account** with necessary permissions. +- Access to an **external object store** (such as AWS S3). +- Familiarity with **Apache Iceberg** and **Snowflake**. + +## Step-by-step guide + +### Step 1: Configure external storage + +Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. + +#### Example: Configure an S3 stage in Snowflake + +```sql +CREATE STAGE my_s3_stage +URL='s3://my-bucket/' +STORAGE_INTEGRATION=my_storage_integration; +``` + +For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). + +### Step 2: Set up a catalog integration in Snowflake + +Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. + +#### Example: Create a catalog integration in Snowflake + +```sql +CREATE CATALOG INTEGRATION my_catalog_integration + CATALOG_SOURCE = 'OBJECT_STORE' + TABLE_FORMAT = 'ICEBERG' + ENABLED = TRUE; +``` + +For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + +### Step 3: Export InfluxDB data to Iceberg format + +Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. + +#### Example: Export data using the Iceberg Exporter + +```sh +# Clone the Iceberg exporter repository +git clone https://github.com/influxdata/influxdb_iox.git +cd influxdb_iox/iceberg_exporter +``` + +- Configure the exporter with your InfluxDB data source and target Iceberg table location. +- Run the exporter to generate Iceberg-compatible Parquet files. + +For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). + +### Step 4: Create an Iceberg table in Snowflake + +After exporting the data, create an Iceberg table in Snowflake. + +#### Example: Create an Iceberg table in Snowflake + +```sql +CREATE ICEBERG TABLE my_iceberg_table + EXTERNAL_VOLUME = 'my_external_volume' + METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; +``` + +Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. + +### Step 5: Query the Iceberg table from Snowflake + +Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. + +#### Example: Query the Iceberg table + +```sql +SELECT * FROM my_iceberg_table +WHERE timestamp > '2025-01-01'; +``` + +## Interfaces for using Iceberg integration + +- **CLI**: `Influx CTL` enables users to trigger snapshot exports. +- **API**: Provides REST endpoints to manage and configure snapshots. +- **SQL (Snowflake)**: Query Iceberg tables using standard SQL. + +## CLI and API reference + +### CLI commands + +#### Example: Enable Iceberg feature and export a snapshot + +```sh +# Enable Iceberg feature +influxctl enable-iceberg + +# Export a snapshot +influxctl export --namespace foo --table bar +``` + +### API endpoints + +#### Example: Export a snapshot + +- **Method**: `POST` +- **Endpoint**: `/snapshots/export` +- **Request body**: + +```json +{ + "namespace": "foo", + "table": "bar" +} +``` + +#### Example: Check snapshot status + +- **Method**: `GET` +- **Endpoint**: `/snapshots/status` + +## Considerations and limitations + +- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. +- **Performance**: Query performance may vary based on data size and query complexity. +- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. + +## Next steps + +- Enhance REST Catalog support. +- Optimize AWS S3 access control and security. +- Improve Grafana dashboards and alerting. +- Expand compatibility testing with other Iceberg engines. + +## References + +- **InfluxDB Iceberg Exporter**: [GitHub Repository](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter) +- **Snowflake Iceberg Tables**: [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg) + + From 5d7bc7f2a161988c8d348417eb023792bf27cec2 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:12 -0700 Subject: [PATCH 02/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- .../influxdb3-query-guides/snapshots/snowflake.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a974c53317..91afffb3f2 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -137,16 +137,5 @@ influxctl export --namespace foo --table bar - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. -## Next steps - -- Enhance REST Catalog support. -- Optimize AWS S3 access control and security. -- Improve Grafana dashboards and alerting. -- Expand compatibility testing with other Iceberg engines. - -## References - -- **InfluxDB Iceberg Exporter**: [GitHub Repository](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter) -- **Snowflake Iceberg Tables**: [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg) From f873263274e41e9414c41cf850eafe4c61bee3b8 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:21 -0700 Subject: [PATCH 03/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 91afffb3f2..0495ba4870 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -111,7 +111,9 @@ influxctl enable-iceberg influxctl export --namespace foo --table bar ``` -### API endpoints +### Use the API to manage and configure snapshots + +Use the {{% product-name %}} HTTP API to export snapshots and check status. #### Example: Export a snapshot From 8391ce799bc89758e84a7b4117cb51514ed86803 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:39 -0700 Subject: [PATCH 04/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0495ba4870..4199425d50 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,5 @@ # Integrating InfluxDB with Snowflake Using Apache Iceberg -## Overview Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. From 8b454e066ee966278fd5595f8b7814cd22799388 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:45:56 -0700 Subject: [PATCH 05/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 4199425d50..fdfe8f9a20 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,7 +1,8 @@ # Integrating InfluxDB with Snowflake Using Apache Iceberg -Snapshots for Snowflake enable users to export time-series data from InfluxDB into a structured format using Apache Iceberg. This integration facilitates efficient data sharing between InfluxDB and Snowflake without the need for complex ETL processes. +Export time-series data snapshots from InfluxDB into Apache Iceberg format. +Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. ### Key Benefits From 57c024c9a74a945207b53e2eb22d1f61ec300377 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:15 -0700 Subject: [PATCH 06/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index fdfe8f9a20..6a4bf29d7f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,4 @@ -# Integrating InfluxDB with Snowflake Using Apache Iceberg +# Integrate with Snowflake using Apache Iceberg Export time-series data snapshots from InfluxDB into Apache Iceberg format. From 9563143f9e48a7d9f6a42e3f66793c8a13a502a5 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:23 -0700 Subject: [PATCH 07/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6a4bf29d7f..f3a92ec76c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -18,7 +18,6 @@ Before you begin, ensure you have the following: - Access to an **external object store** (such as AWS S3). - Familiarity with **Apache Iceberg** and **Snowflake**. -## Step-by-step guide ### Step 1: Configure external storage From 3f82fb63e2f36dfbf9b8c835e49df8b36de6ce1d Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:35 -0700 Subject: [PATCH 08/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index f3a92ec76c..1079ec9446 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -19,7 +19,7 @@ Before you begin, ensure you have the following: - Familiarity with **Apache Iceberg** and **Snowflake**. -### Step 1: Configure external storage +## Step 1: Configure external storage Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. From 0478e058001c78f3ffd4fab8e7cc561e917b8271 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:47 -0700 Subject: [PATCH 09/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1079ec9446..4a10a87b48 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -92,9 +92,9 @@ WHERE timestamp > '2025-01-01'; ## Interfaces for using Iceberg integration -- **CLI**: `Influx CTL` enables users to trigger snapshot exports. -- **API**: Provides REST endpoints to manage and configure snapshots. -- **SQL (Snowflake)**: Query Iceberg tables using standard SQL. +- [Use the CLI to trigger snapshot exports](#use-the-CLI-to-trigger-snapshot-exports) +- [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) +- [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) ## CLI and API reference From 4a1f036d795865b42b3c7ecfe89121a4714573f3 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:46:54 -0700 Subject: [PATCH 10/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 4a10a87b48..c4151a105f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -98,7 +98,7 @@ WHERE timestamp > '2025-01-01'; ## CLI and API reference -### CLI commands +### Use the CLI to trigger snapshot exports #### Example: Enable Iceberg feature and export a snapshot From 091c31503fc63e95d7757300f09d21c436263684 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Wed, 19 Mar 2025 13:47:06 -0700 Subject: [PATCH 11/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c4151a105f..a66cb2272d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -96,8 +96,6 @@ WHERE timestamp > '2025-01-01'; - [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) - [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) -## CLI and API reference - ### Use the CLI to trigger snapshot exports #### Example: Enable Iceberg feature and export a snapshot From a32d724aff4e7013e871dc812c16a53087071255 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:27:51 -0700 Subject: [PATCH 12/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a66cb2272d..62c9e1c22a 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -23,7 +23,7 @@ Before you begin, ensure you have the following: Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. -#### Example: Configure an S3 stage in Snowflake +### Example: Configure an S3 stage in Snowflake ```sql CREATE STAGE my_s3_stage From d22f3fbdd4c3916eeb44bdd503160bae55398a65 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:02 -0700 Subject: [PATCH 13/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 62c9e1c22a..494168403d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -33,7 +33,7 @@ STORAGE_INTEGRATION=my_storage_integration; For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). -### Step 2: Set up a catalog integration in Snowflake +## Step 2: Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. From 0a5d3f56156559ec01d59cae46f650516f0f50e5 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:13 -0700 Subject: [PATCH 14/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 494168403d..0a63768b75 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -37,7 +37,7 @@ For more details, refer to the [Snowflake documentation](https://docs.snowflake. Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -#### Example: Create a catalog integration in Snowflake +### Example: Create a catalog integration in Snowflake ```sql CREATE CATALOG INTEGRATION my_catalog_integration From 7bc3352994eb92a322f745e69d4a3a274ce3003b Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:22 -0700 Subject: [PATCH 15/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0a63768b75..58309eb2b7 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -46,7 +46,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration ENABLED = TRUE; ``` -For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). ### Step 3: Export InfluxDB data to Iceberg format From 22c19be4667a12684ec74cf29fe52432ae9ee49d Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:31 -0700 Subject: [PATCH 16/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 58309eb2b7..3a9f55d7db 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -48,7 +48,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -### Step 3: Export InfluxDB data to Iceberg format +## Step 3: Export InfluxDB data to Iceberg format Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. From e9345afc0cb8a3c7cd938344d8a5d25f1337353c Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:28:57 -0700 Subject: [PATCH 17/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3a9f55d7db..40d4616812 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -50,7 +50,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Step 3: Export InfluxDB data to Iceberg format -Use InfluxDB's Iceberg Exporter to convert and export your time-series data into the Iceberg table format. +Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. #### Example: Export data using the Iceberg Exporter From 4693ecec749b9185d095ef61333061ba66c5b0a1 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:07 -0700 Subject: [PATCH 18/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 40d4616812..be3907ed25 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -52,7 +52,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. -#### Example: Export data using the Iceberg Exporter +### Example: Export data using the Iceberg exporter ```sh # Clone the Iceberg exporter repository From 8ea52b75e3358290e55f058e4cf1be7168d7e4b4 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:33 -0700 Subject: [PATCH 19/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index be3907ed25..fbcd27d256 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -79,7 +79,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -### Step 5: Query the Iceberg table from Snowflake +## Step 5: Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From 836e31b9244a5fe99293c73871697abdad0124e3 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 12:29:50 -0700 Subject: [PATCH 20/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index fbcd27d256..6dacdbe00c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -83,7 +83,7 @@ Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external st Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. -#### Example: Query the Iceberg table +### Example: Query the Iceberg table ```sql SELECT * FROM my_iceberg_table From 3283130a3bfe97ad539dee038f0b98bd5edab60d Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 17:12:58 -0700 Subject: [PATCH 21/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6dacdbe00c..b835158553 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -6,7 +6,7 @@ Integrate data with Snowflake and other Iceberg-compatible tools without the nee ### Key Benefits -- **Efficient data access**: Query InfluxDB data directly from Snowflake. +- **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. From 6c55f87a14fe70311e7d135a7ff846237a040d1c Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Thu, 20 Mar 2025 17:13:16 -0700 Subject: [PATCH 22/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Jason Stirnaman --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index b835158553..9b4c7f5067 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -50,7 +50,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Step 3: Export InfluxDB data to Iceberg format -Use the InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. +Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. ### Example: Export data using the Iceberg exporter From 7c6a6305190af00788e33e447119da89d9a6779a Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 20 Mar 2025 18:10:09 -0700 Subject: [PATCH 23/62] updating headings --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 9b4c7f5067..5aebcfc240 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -19,7 +19,7 @@ Before you begin, ensure you have the following: - Familiarity with **Apache Iceberg** and **Snowflake**. -## Step 1: Configure external storage +## Configure external storage Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. @@ -33,7 +33,7 @@ STORAGE_INTEGRATION=my_storage_integration; For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). -## Step 2: Set up a catalog integration in Snowflake +## Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -48,7 +48,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -## Step 3: Export InfluxDB data to Iceberg format +## Export InfluxDB data to Iceberg format Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. @@ -65,7 +65,7 @@ cd influxdb_iox/iceberg_exporter For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). -### Step 4: Create an Iceberg table in Snowflake +### Create an Iceberg table in Snowflake After exporting the data, create an Iceberg table in Snowflake. @@ -79,7 +79,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -## Step 5: Query the Iceberg table from Snowflake +## Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From d353d01a3a3c37dc04e292331bec34ae4e06ca1c Mon Sep 17 00:00:00 2001 From: meelahme Date: Thu, 20 Mar 2025 18:18:42 -0700 Subject: [PATCH 24/62] Adding a numbered-list TOC --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5aebcfc240..eb8f316660 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -10,6 +10,16 @@ Integrate data with Snowflake and other Iceberg-compatible tools without the nee - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. +## Implementation steps + +Follow these steps to integrate InfluxDB 3 with Snowflake using Apache Iceberg: + +1. [Configure external storage](#configure-external-storage) +2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) +3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) +4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +5. [Query your data in Snowflake](#query-your-data-in-snowflake) + ## Prerequisites Before you begin, ensure you have the following: From 677ed17db3d1f400ad2b5f185db355fd9acbb1fe Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 10:13:59 -0700 Subject: [PATCH 25/62] Added an export command example --- .../snapshots/snowflake.md | 57 ++++++++++++++++--- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index eb8f316660..1118ca50b9 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -62,18 +62,57 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. -### Example: Export data using the Iceberg exporter +### Example: Export data using Iceberg exporter -```sh -# Clone the Iceberg exporter repository -git clone https://github.com/influxdata/influxdb_iox.git -cd influxdb_iox/iceberg_exporter -``` +This example assumes: +- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data) +- You've configured compaction to trigger more quickly with these environment variables: + + - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` + - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` +- You have a `config.json` -- Configure the exporter with your InfluxDB data source and target Iceberg table location. -- Run the exporter to generate Iceberg-compatible Parquet files. +#### Example `config.json` -For more details, refer to the [Iceberg Exporter README](https://github.com/influxdata/influxdb_iox/tree/main/iceberg_exporter). +```json +{ + "exports": [ + { + "namespace": "company_sensors", + "table_name": "cpu" + } + ] +} +``` +#### Running the export command + +```console +$ influxdb_iox iceberg export \ + --catalog-dsn postgresql://postgres@localhost:5432/postgres \ + --source-object-store file + --source-data-dir ~/.influxdb_iox/object_store \ + --sink-object-store file \ + --sink-data-dir /tmp/iceberg \ + --export-config-path config.json +``` +The export command outputs an absolute path to an Iceberg metadata file: + +`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json +` +#### Example: Querying the exported metadata using duckdb + +```console +$ duckdb +D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; +┌───────────┬──────────────────────┬─────────────────────┬─────────────┬───┬────────────┬───────────────┬─────────────┬────────────────────┬────────────────────┐ +│ cpu │ host │ time │ usage_guest │ … │ usage_nice │ usage_softirq │ usage_steal │ usage_system │ usage_user │ +│ varchar │ varchar │ timestamp │ double │ │ double │ double │ double │ double │ double │ +├───────────┼──────────────────────┼─────────────────────┼─────────────┼───┼────────────┼───────────────┼─────────────┼────────────────────┼────────────────────┤ +│ cpu-total │ Andrews-MBP.hsd1.m… │ 2020-06-11 16:52:00 │ 0.0 │ … │ 0.0 │ 0.0 │ 0.0 │ 1.1173184357541899 │ 0.9435133457479826 │ +├───────────┴──────────────────────┴─────────────────────┴─────────────┴───┴────────────┴───────────────┴─────────────┴────────────────────┴────────────────────┤ +│ 1 rows 13 columns (9 shown) │ +└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ +``` ### Create an Iceberg table in Snowflake From accaffecb7fe86f781e7d73adfc167dba2b256bd Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 10:52:16 -0700 Subject: [PATCH 26/62] Proof read: improving grammar and clarity --- .../snapshots/snowflake.md | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1118ca50b9..bbc075708b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,5 @@ # Integrate with Snowflake using Apache Iceberg - Export time-series data snapshots from InfluxDB into Apache Iceberg format. Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. @@ -60,17 +59,17 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ## Export InfluxDB data to Iceberg format -Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster into the Iceberg table format. +Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. ### Example: Export data using Iceberg exporter -This example assumes: -- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data) -- You've configured compaction to trigger more quickly with these environment variables: +This example assumes the following: +- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data). +- You've configured compaction to trigger more quickly with these environment variables: - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` -- You have a `config.json` +- You have a `config.json`. #### Example `config.json` @@ -84,6 +83,7 @@ This example assumes: ] } ``` + #### Running the export command ```console @@ -95,11 +95,12 @@ $ influxdb_iox iceberg export \ --sink-data-dir /tmp/iceberg \ --export-config-path config.json ``` + The export command outputs an absolute path to an Iceberg metadata file: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json ` -#### Example: Querying the exported metadata using duckdb +#### Example: Querying the exported metadata using DuckDB ```console $ duckdb @@ -114,6 +115,8 @@ D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` +Next, create an Iceberg table in Snowflake. + ### Create an Iceberg table in Snowflake After exporting the data, create an Iceberg table in Snowflake. @@ -151,10 +154,10 @@ WHERE timestamp > '2025-01-01'; ```sh # Enable Iceberg feature -influxctl enable-iceberg +$ influxctl enable-iceberg # Export a snapshot -influxctl export --namespace foo --table bar +$ influxctl export --namespace foo --table bar ``` ### Use the API to manage and configure snapshots From 8b8257b4e2b5d135211bf0fc9862f465f03bdb9e Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 11:28:30 -0700 Subject: [PATCH 27/62] Adding explanations for examples --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index bbc075708b..6e75d61f70 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -166,6 +166,8 @@ Use the {{% product-name %}} HTTP API to export snapshots and check status. #### Example: Export a snapshot +This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. + - **Method**: `POST` - **Endpoint**: `/snapshots/export` - **Request body**: @@ -176,17 +178,21 @@ Use the {{% product-name %}} HTTP API to export snapshots and check status. "table": "bar" } ``` +The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. #### Example: Check snapshot status +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. + - **Method**: `GET` - **Endpoint**: `/snapshots/status` +In this example, the GET request to the /snapshots/status endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. + ## Considerations and limitations +When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: + - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. - - - From ffc80146ab1c923f4e36195cf9c3345c330d2e56 Mon Sep 17 00:00:00 2001 From: meelahme Date: Fri, 21 Mar 2025 11:41:50 -0700 Subject: [PATCH 28/62] Updating examples --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 6e75d61f70..5c5d5a397e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -182,12 +182,12 @@ The `POST` request to the `/snapshots/export` endpoint triggers the export of da #### Example: Check snapshot status -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - **Method**: `GET` - **Endpoint**: `/snapshots/status` -In this example, the GET request to the /snapshots/status endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. +The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. ## Considerations and limitations From 05d48584122d5afcf504f81fce6493a5f2b84466 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:58:51 -0700 Subject: [PATCH 29/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5c5d5a397e..99b22c0fe4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,3 @@ -# Integrate with Snowflake using Apache Iceberg Export time-series data snapshots from InfluxDB into Apache Iceberg format. Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. From aa6b36a80f9dc5a2346dfdfe547645a258ba77b9 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:12 -0700 Subject: [PATCH 30/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 99b22c0fe4..d081697511 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,6 @@ -Export time-series data snapshots from InfluxDB into Apache Iceberg format. -Integrate data with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. +Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. +Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. ### Key Benefits From 7e260b530c238cc345878f64c8bec520710a983a Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:27 -0700 Subject: [PATCH 31/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../snapshots/snowflake.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index d081697511..a0dfea7088 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -8,16 +8,6 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. -## Implementation steps - -Follow these steps to integrate InfluxDB 3 with Snowflake using Apache Iceberg: - -1. [Configure external storage](#configure-external-storage) -2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) -3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) -4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -5. [Query your data in Snowflake](#query-your-data-in-snowflake) - ## Prerequisites Before you begin, ensure you have the following: @@ -26,6 +16,14 @@ Before you begin, ensure you have the following: - Access to an **external object store** (such as AWS S3). - Familiarity with **Apache Iceberg** and **Snowflake**. +## Integrate InfluxDB 3 with Snowflake + +1. [Configure external storage](#configure-external-storage) +2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) +3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) +4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +5. [Query your data in Snowflake](#query-your-data-in-snowflake) + ## Configure external storage From bd472c84fd7b77806a2a24e9c14f18796d8ede65 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 12:59:49 -0700 Subject: [PATCH 32/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index a0dfea7088..3846ae0746 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -25,19 +25,17 @@ Before you begin, ensure you have the following: 5. [Query your data in Snowflake](#query-your-data-in-snowflake) -## Configure external storage +### Create a Snowflake external stage -Set up an external storage location (such as AWS S3) to store Iceberg table data and metadata. +Use the `CREATE STAGE` Snowflake SQL command to set up an external storage location +(such as AWS S3) to store Iceberg table data and metadata--for example: -### Example: Configure an S3 stage in Snowflake +#### Example: Configure an S3 stage in Snowflake ```sql CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -``` - -For more details, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-object-storage). ## Set up a catalog integration in Snowflake From 1ad0ffa51553b8b0af69f32503b85b0703180a23 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:00:07 -0700 Subject: [PATCH 33/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3846ae0746..8452aa1483 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -37,11 +37,11 @@ CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -## Set up a catalog integration in Snowflake +### Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -### Example: Create a catalog integration in Snowflake +#### Example: Create a catalog integration in Snowflake ```sql CREATE CATALOG INTEGRATION my_catalog_integration From b022d2536fa8daee2eaf258b9639edbd0db82ce8 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:01:43 -0700 Subject: [PATCH 34/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8452aa1483..e441e1c71f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -52,7 +52,7 @@ CREATE CATALOG INTEGRATION my_catalog_integration For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -## Export InfluxDB data to Iceberg format +### Export InfluxDB time series data to Iceberg format Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. From b71285148ea65085ae0f5e6538afb0bf1894f5fb Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:01:57 -0700 Subject: [PATCH 35/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index e441e1c71f..5d3db0f74e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -54,7 +54,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format -Use InfluxData's Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. ### Example: Export data using Iceberg exporter From c5d58eef903e6a85f223d99acef8c2ca752bcc46 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:02:11 -0700 Subject: [PATCH 36/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 5d3db0f74e..8d8edec05d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -56,7 +56,7 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -### Example: Export data using Iceberg exporter +#### Example: Export data using Iceberg exporter This example assumes the following: From 9adfbdaab530546d724e5607b0a4a79a69cdac56 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:02:30 -0700 Subject: [PATCH 37/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8d8edec05d..accdb4af6d 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -60,7 +60,6 @@ Use the InfluxDB Iceberg exporter to convert and export your time-series data fr This example assumes the following: -- You have followed the example for [writing and querying data in the IOx README](https://github.com/influxdata/influxdb_iox/blob/main/README.md#write-and-read-data). - You've configured compaction to trigger more quickly with these environment variables: - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` From 37888c0a6a3a465f6c9b35c344f1a29fc37c63bd Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:03 -0700 Subject: [PATCH 38/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index accdb4af6d..63bcdaffe4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -78,7 +78,7 @@ This example assumes the following: } ``` -#### Running the export command +### Run the export command ```console $ influxdb_iox iceberg export \ From 4e6e03428519a6202376a9f1eb0bdf81905849b9 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:22 -0700 Subject: [PATCH 39/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- .../snapshots/snowflake.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 63bcdaffe4..54c82b8437 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -94,22 +94,6 @@ The export command outputs an absolute path to an Iceberg metadata file: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json ` -#### Example: Querying the exported metadata using DuckDB - -```console -$ duckdb -D SELECT * FROM iceberg_scan('/tmp/iceberg/metadata/v1.metadata.json') LIMIT 1; -┌───────────┬──────────────────────┬─────────────────────┬─────────────┬───┬────────────┬───────────────┬─────────────┬────────────────────┬────────────────────┐ -│ cpu │ host │ time │ usage_guest │ … │ usage_nice │ usage_softirq │ usage_steal │ usage_system │ usage_user │ -│ varchar │ varchar │ timestamp │ double │ │ double │ double │ double │ double │ double │ -├───────────┼──────────────────────┼─────────────────────┼─────────────┼───┼────────────┼───────────────┼─────────────┼────────────────────┼────────────────────┤ -│ cpu-total │ Andrews-MBP.hsd1.m… │ 2020-06-11 16:52:00 │ 0.0 │ … │ 0.0 │ 0.0 │ 0.0 │ 1.1173184357541899 │ 0.9435133457479826 │ -├───────────┴──────────────────────┴─────────────────────┴─────────────┴───┴────────────┴───────────────┴─────────────┴────────────────────┴────────────────────┤ -│ 1 rows 13 columns (9 shown) │ -└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ -``` - -Next, create an Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From e79725d87e1e1220e13ea7fa6e9de4f9ffe3e018 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:35 -0700 Subject: [PATCH 40/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 54c82b8437..44097d32f2 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -109,7 +109,7 @@ CREATE ICEBERG TABLE my_iceberg_table Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. -## Query the Iceberg table from Snowflake +### Query the Iceberg table from Snowflake Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. From a4e2b80c2a3053ecefc766a900aa8a1ec3f2af77 Mon Sep 17 00:00:00 2001 From: Jameelah Mercer <36314199+MeelahMe@users.noreply.github.com> Date: Fri, 21 Mar 2025 13:03:46 -0700 Subject: [PATCH 41/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 44097d32f2..3318eb80fc 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -113,7 +113,7 @@ Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external st Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. -### Example: Query the Iceberg table +#### Example: Query the Iceberg table ```sql SELECT * FROM my_iceberg_table From aebe4417bd143a0cfe9f27479bf6f0b7e51485c9 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 10:48:33 -0700 Subject: [PATCH 42/62] removing run export example and adding --- .../snapshots/snowflake.md | 21 +++---------------- 1 file changed, 3 insertions(+), 18 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 3318eb80fc..af8f2e3e0a 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,4 +1,3 @@ - Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. @@ -60,9 +59,6 @@ Use the InfluxDB Iceberg exporter to convert and export your time-series data fr This example assumes the following: -- You've configured compaction to trigger more quickly with these environment variables: - - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L0_FILES_TO_COMPACT=1` - - `INFLUXDB_IOX_COMPACTION_MIN_NUM_L1_FILES_TO_COMPACT=1` - You have a `config.json`. #### Example `config.json` @@ -78,22 +74,11 @@ This example assumes the following: } ``` -### Run the export command - -```console -$ influxdb_iox iceberg export \ - --catalog-dsn postgresql://postgres@localhost:5432/postgres \ - --source-object-store file - --source-data-dir ~/.influxdb_iox/object_store \ - --sink-object-store file \ - --sink-data-dir /tmp/iceberg \ - --export-config-path config.json -``` +After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: -The export command outputs an absolute path to an Iceberg metadata file: +`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` -`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json -` +This metadata file is what you'll reference when creating your Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From 277fcde268fc098c45953e1ee997e85fd427b9d9 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:04:51 -0700 Subject: [PATCH 43/62] Changes to- Example: Export data using Iceberg exporter --- .../influxdb3-query-guides/snapshots/snowflake.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index af8f2e3e0a..c2de54b1c9 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -55,13 +55,12 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -#### Example: Export data using Iceberg exporter +#### Creating a configuration file -This example assumes the following: +Before running the export command, you need to create a configuration file that specifies which tables to export: -- You have a `config.json`. - -#### Example `config.json` +1. Create a file named `config.json` in your working directory +2. Add the following JSON content, adjusting the namespace and table names to match your data: ```json { @@ -74,6 +73,8 @@ This example assumes the following: } ``` +This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. + After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: `/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` From 07f655171af067e80babb37d2befcc90cc9d7621 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:32:05 -0700 Subject: [PATCH 44/62] Adding note to- Export InfluxDB time series data to Iceberg format --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c2de54b1c9..52e29f2830 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -54,7 +54,13 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + +### Export InfluxDB time series data to Iceberg format +> **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. + +Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. #### Creating a configuration file Before running the export command, you need to create a configuration file that specifies which tables to export: From 4b2ee68164675e8e68db039903dff4a39c07c235 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:39:38 -0700 Subject: [PATCH 45/62] removing reduandant sections and information --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 52e29f2830..8ffd7f0564 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -53,14 +53,11 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl ### Export InfluxDB time series data to Iceberg format -Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). - -### Export InfluxDB time series data to Iceberg format - > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). + #### Creating a configuration file Before running the export command, you need to create a configuration file that specifies which tables to export: From 0c31daa4f7f634b40d2911cef3f35c0786e06f49 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 11:53:21 -0700 Subject: [PATCH 46/62] removing IOX references --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8ffd7f0564..1b5e1517d0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -56,7 +56,6 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). #### Creating a configuration file @@ -78,11 +77,7 @@ Before running the export command, you need to create a configuration file that This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. -After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file at a location similar to: - -`/tmp/iceberg/company_sensors/cpu/metadata/v1.metadata.json` - -This metadata file is what you'll reference when creating your Iceberg table in Snowflake. +After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. ### Create an Iceberg table in Snowflake From 6cd3c8265949ed3a8a911d9449b2c7d3bdd6abc3 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:03:01 -0700 Subject: [PATCH 47/62] Rewriting section to focus and clarify that Iceberg table is done with support assistance --- .../influxdb3-query-guides/snapshots/snowflake.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 1b5e1517d0..55aa53dec5 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -81,9 +81,11 @@ After configuring the export settings in the `config.json` file, the system auto ### Create an Iceberg table in Snowflake -After exporting the data, create an Iceberg table in Snowflake. +After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: -#### Example: Create an Iceberg table in Snowflake +1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. + +2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: ```sql CREATE ICEBERG TABLE my_iceberg_table @@ -91,7 +93,9 @@ CREATE ICEBERG TABLE my_iceberg_table METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; ``` -Ensure that `EXTERNAL_VOLUME` and `METADATA_FILE_PATH` point to your external storage and metadata file. +3. Your support engineer will help ensure the EXTERNAL_VOLUME and METADATA_FILE_PATH parameters correctly point to your external storage and metadata file. + +This creates a table in Snowflake that reads directly from the Iceberg-formatted data exported from your InfluxDB instance. ### Query the Iceberg table from Snowflake From 8adec7291f551c45f1b38d4a2241b5acadd99052 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:11:48 -0700 Subject: [PATCH 48/62] removing entier section: Interfaces for using Iceberg integration section --- .../snapshots/snowflake.md | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 55aa53dec5..8c048b67ca 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -108,24 +108,6 @@ SELECT * FROM my_iceberg_table WHERE timestamp > '2025-01-01'; ``` -## Interfaces for using Iceberg integration - -- [Use the CLI to trigger snapshot exports](#use-the-CLI-to-trigger-snapshot-exports) -- [Use the API to manage and configure snapshots](#use-the-api-to-manage-and-configure-snapshots) -- [Use SQL in Snowflake to query Iceberg tables](#use-sql-in-snowflake-to-query-iceberg-tables) - -### Use the CLI to trigger snapshot exports - -#### Example: Enable Iceberg feature and export a snapshot - -```sh -# Enable Iceberg feature -$ influxctl enable-iceberg - -# Export a snapshot -$ influxctl export --namespace foo --table bar -``` - ### Use the API to manage and configure snapshots Use the {{% product-name %}} HTTP API to export snapshots and check status. From 92208e494e9a0391260a3db0d6c91c2a08bf1cf1 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:28:12 -0700 Subject: [PATCH 49/62] Adding read-only note to section --- .../shared/influxdb3-query-guides/snapshots/snowflake.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8c048b67ca..733cd6aafa 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -81,8 +81,14 @@ After configuring the export settings in the `config.json` file, the system auto ### Create an Iceberg table in Snowflake +After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: + After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: +> **⚠️ Important**: **Tables created through this integration are read-only.** +> **You cannot write directly to these tables using Snowflake or any other engine.** +> They serve as an analytics interface to your InfluxDB data. + 1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. 2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: From c9e26fae6d94448978d7812b57e281feb1611576 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 12:56:27 -0700 Subject: [PATCH 50/62] updating TOC numbe4r list --- .../influxdb3-query-guides/snapshots/snowflake.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 733cd6aafa..b621908cb5 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -17,11 +17,10 @@ Before you begin, ensure you have the following: ## Integrate InfluxDB 3 with Snowflake -1. [Configure external storage](#configure-external-storage) -2. [Set up a catalog integration in Snowflake](#set-up-a-catalog-integration-in-snowflake) -3. [Export InfluxDB data to Iceberg format](#export-influxdb-data-to-iceberg-format) -4. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -5. [Query your data in Snowflake](#query-your-data-in-snowflake) +1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) +2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) +3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) +4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) ### Create a Snowflake external stage @@ -36,7 +35,7 @@ CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; -### Set up a catalog integration in Snowflake +### Set up a catalog integration in Snowflake {#set-up-a-catalog-integration-in-snowflake} Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -79,7 +78,7 @@ This configuration file tells the exporter which InfluxDB tables to convert to I After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. -### Create an Iceberg table in Snowflake +### Create an Iceberg table in Snowflake After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: From cb023207f32c6144d11e61601cc7feb468f64fbc Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 13:06:51 -0700 Subject: [PATCH 51/62] Updating who to contact for Snowflake Integration --- .../influxdb3-query-guides/snapshots/snowflake.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index b621908cb5..db5589c5f6 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,6 +1,10 @@ +# Snowflake Integration + Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. +> **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. + ### Key Benefits - **Efficient data access**: Query your data directly from Snowflake. @@ -17,16 +21,16 @@ Before you begin, ensure you have the following: ## Integrate InfluxDB 3 with Snowflake +Once you've contacted sales and enabled Iceberg integration, your InfluxData support engineers will help you with the following steps: + 1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) 2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) 3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) 4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) - ### Create a Snowflake external stage -Use the `CREATE STAGE` Snowflake SQL command to set up an external storage location -(such as AWS S3) to store Iceberg table data and metadata--for example: +Your support engineer will guide you through setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command. This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. #### Example: Configure an S3 stage in Snowflake @@ -34,8 +38,9 @@ Use the `CREATE STAGE` Snowflake SQL command to set up an external storage locat CREATE STAGE my_s3_stage URL='s3://my-bucket/' STORAGE_INTEGRATION=my_storage_integration; +``` -### Set up a catalog integration in Snowflake {#set-up-a-catalog-integration-in-snowflake} +### Set up a catalog integration in Snowflake Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. @@ -84,7 +89,7 @@ After the export process is complete, you can work with your InfluxData support After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: -> **⚠️ Important**: **Tables created through this integration are read-only.** +> **Note**: **Tables created through this integration are read-only.** > **You cannot write directly to these tables using Snowflake or any other engine.** > They serve as an analytics interface to your InfluxDB data. From 4d259fb305e71127ebee6571cf0780423f263f97 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:39:40 -0700 Subject: [PATCH 52/62] Restructuring, reving, and proof reading edit --- .../snapshots/snowflake.md | 103 ++++++++++++------ 1 file changed, 69 insertions(+), 34 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index db5589c5f6..59bc2957c3 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -15,9 +15,22 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q Before you begin, ensure you have the following: -- A **Snowflake account** with necessary permissions. -- Access to an **external object store** (such as AWS S3). -- Familiarity with **Apache Iceberg** and **Snowflake**. +- **InfluxDB Cloud Dedicated plan** or compatible environment +- **Snowflake account** with appropriate permissions +- **External object store** (AWS S3, Azure Blob Storage, or GCP) + +## Request Iceberg Integration + +1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. + +2. Be prepared to provide: + - Your organization ID + - Customer name + - Cloud provider and region + - External storage details (bucket name, region, permissions) + - Technical contact information + +> **Note**: Iceberg integration is a premium feature that requires setup by the InfluxData team. ## Integrate InfluxDB 3 with Snowflake @@ -42,7 +55,7 @@ STORAGE_INTEGRATION=my_storage_integration; ### Set up a catalog integration in Snowflake -Set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +Your support engineer will help you set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. #### Example: Create a catalog integration in Snowflake @@ -59,7 +72,58 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. -Use the InfluxDB Iceberg exporter to convert and export your time-series data from your {{% product-name omit="Clustered" %}} cluster to the Iceberg table format. +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#tab-cli) +[API](#tab-api) +{{% /tabs %}} + +{{% tab-content %}} +{{% tab-pane id="tab-cli" %}} + +#### Using the CLI + +Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: + +```sh +influxctl snapshot export --namespace foo --table bar +``` + +{{% /tab-pane %}} +{{% tab-pane id="tab-api" %}} + +#### Using the API + +Use the {{% product-name %}} HTTP API to export snapshots and check status. + +##### Example: Export a snapshot + +This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. + +- **Method**: `POST` +- **Endpoint**: `/snapshots/export` +- **Request body**: + +```json +{ + "namespace": "foo", + "table": "bar" +} +``` +The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. + +##### Example: Check snapshot status + +This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. + +- **Method**: `GET` +- **Endpoint**: `/snapshots/status` + +The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. + +{{% /tab-pane %}} +{{% /tab-content %}} +{{< /tabs-wrapper >}} #### Creating a configuration file @@ -118,35 +182,6 @@ SELECT * FROM my_iceberg_table WHERE timestamp > '2025-01-01'; ``` -### Use the API to manage and configure snapshots - -Use the {{% product-name %}} HTTP API to export snapshots and check status. - -#### Example: Export a snapshot - -This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. - -- **Method**: `POST` -- **Endpoint**: `/snapshots/export` -- **Request body**: - -```json -{ - "namespace": "foo", - "table": "bar" -} -``` -The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. - -#### Example: Check snapshot status - -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - -- **Method**: `GET` -- **Endpoint**: `/snapshots/status` - -The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. - ## Considerations and limitations When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: From a8badd6cc0f9813bbcbde9ef57a2fd4b1c0fb9c7 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:42:36 -0700 Subject: [PATCH 53/62] google dev style read through --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 59bc2957c3..0a6572d8f4 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,11 +1,11 @@ -# Snowflake Integration +# Snowflake integration -Integrate {{< product-name >}} with Snowflake and other Iceberg-compatible tools without the need for complex ETL processes. -Export time series data snapshots from InfluxDB into Apache Iceberg format and query it from Snowflake. +Integrate {{< product-name >}} with Snowflake without complex ETL processes. +Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. > **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. -### Key Benefits +## Key benefits - **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. @@ -15,11 +15,11 @@ Export time series data snapshots from InfluxDB into Apache Iceberg format and q Before you begin, ensure you have the following: -- **InfluxDB Cloud Dedicated plan** or compatible environment +- **{{< product-name >}} Cloud Dedicated plan** or compatible environment - **Snowflake account** with appropriate permissions - **External object store** (AWS S3, Azure Blob Storage, or GCP) -## Request Iceberg Integration +## Request Iceberg integration 1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. From 83b65701088ac30b40336c3e8cfd19972e87c6ee Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 15:47:09 -0700 Subject: [PATCH 54/62] updating prereqs --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 0a6572d8f4..c65e207d2c 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -16,8 +16,6 @@ Export time series data snapshots into Apache Iceberg format and query directly Before you begin, ensure you have the following: - **{{< product-name >}} Cloud Dedicated plan** or compatible environment -- **Snowflake account** with appropriate permissions -- **External object store** (AWS S3, Azure Blob Storage, or GCP) ## Request Iceberg integration From 6eed23c6d5c39a80f5fa7f135011409ff201cf29 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 16:00:07 -0700 Subject: [PATCH 55/62] wip: removing tab shortcode to figure out error --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index c65e207d2c..570f99827e 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -70,15 +70,6 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. -{{< tabs-wrapper >}} -{{% tabs %}} -[CLI](#tab-cli) -[API](#tab-api) -{{% /tabs %}} - -{{% tab-content %}} -{{% tab-pane id="tab-cli" %}} - #### Using the CLI Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: @@ -87,9 +78,6 @@ Use the `influxctl` command to export InfluxDB time-series data to Iceberg forma influxctl snapshot export --namespace foo --table bar ``` -{{% /tab-pane %}} -{{% tab-pane id="tab-api" %}} - #### Using the API Use the {{% product-name %}} HTTP API to export snapshots and check status. From a013770f9141e6ad902bf3e39e3533cc2f9d0eb3 Mon Sep 17 00:00:00 2001 From: meelahme Date: Wed, 26 Mar 2025 16:36:06 -0700 Subject: [PATCH 56/62] adding shortcode structure back --- .../influxdb3-query-guides/snapshots/snowflake.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 570f99827e..ec669a045f 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -70,6 +70,14 @@ For more information, refer to the [Snowflake documentation](https://docs.snowfl > **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. +{{< tabs-wrapper >}} +{{% tabs %}} +[CLI](#tab-cli) +[API](#tab-api) +{{% /tabs %}} + +{{% tab-content %}} +{{% tab-pane id="tab-cli" %}} #### Using the CLI Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: @@ -77,7 +85,8 @@ Use the `influxctl` command to export InfluxDB time-series data to Iceberg forma ```sh influxctl snapshot export --namespace foo --table bar ``` - +{{% /tab-pane %}} +{{% tab-pane id="tab-api" %}} #### Using the API Use the {{% product-name %}} HTTP API to export snapshots and check status. @@ -106,7 +115,6 @@ This example shows how to check the status of an ongoing or completed snapshot e - **Endpoint**: `/snapshots/status` The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. - {{% /tab-pane %}} {{% /tab-content %}} {{< /tabs-wrapper >}} From 905edc3984e005e3d0485297ae1af02e6c34f5b3 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 12:23:34 -0500 Subject: [PATCH 57/62] feat(dedicated): iceberg export for snowflake: - Add directory and frontmatter - Revise content to be Cloud Dedicated specific, omitting implementation and config detail that will be handled by InfluxData Support - Fix code samples - Organize, clean up headings - TODO: Comment out influxctl and API code if necessary (until features are completed) - TODO: Verify details with PM - TODO: Move content as noted in the iceberg-for-snowflake.md comment --- .../process-data/export/_index.md | 16 ++ .../export/iceberg-for-snowflake.md | 19 ++ .../snapshots/snowflake.md | 198 +++++++----------- 3 files changed, 113 insertions(+), 120 deletions(-) create mode 100644 content/influxdb3/cloud-dedicated/process-data/export/_index.md create mode 100644 content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md diff --git a/content/influxdb3/cloud-dedicated/process-data/export/_index.md b/content/influxdb3/cloud-dedicated/process-data/export/_index.md new file mode 100644 index 0000000000..f00e2ed286 --- /dev/null +++ b/content/influxdb3/cloud-dedicated/process-data/export/_index.md @@ -0,0 +1,16 @@ +--- +title: Export data +description: > + Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems + and avoid complex ETL processes. +menu: + influxdb3_cloud_dedicated: + parent: Process & visualize data +weight: 103 +--- +Use InfluxData export tools to: + +- Integrate time series data stored in {{% product-name %}} with other tools and systems +- Avoid complex ETL (Extract, Transform, Load) processes + +{{< children >}} \ No newline at end of file diff --git a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md new file mode 100644 index 0000000000..6cbf1447d3 --- /dev/null +++ b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md @@ -0,0 +1,19 @@ +--- +title: Export in Iceberg format for Snowflake +seotitle: Export time series data to Iceberg format for Snowflake +description: > + Integrate {{< product-name >}} with Snowflake without complex ETL processes. + Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. +menu: + influxdb3_cloud_dedicated: + name: Snowflake with Iceberg + parent: Export data +weight: 101 +influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] +source: /shared/influxdb3-query-guides/snapshots/snowflake.md +--- + + + + diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index ec669a045f..07d4a9663b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -1,185 +1,143 @@ -# Snowflake integration - Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. -> **Note**: Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to enable this capability for your organization. - ## Key benefits - **Efficient data access**: Query your data directly from Snowflake. - **Cost-effective storage**: Optimize data retention and minimize storage costs. - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. -## Prerequisites +> [!Note] +> #### Contact sales to enable Iceberg integration +> +> Iceberg integration is a premium feature that requires setup by the InfluxData team. +> [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. + +- [Prerequisite](#prerequisite) +- [Request Iceberg integration](#request-iceberg-integration) +- [Integrate InfluxDB 3 with Snowflake](#integrate-influxdb-3-with-snowflake) +- [Export data to Iceberg format](#export-data-to-iceberg-format) +- [Check snapshot status](#check-snapshot-status) +- [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) +- [Considerations and limitations](#considerations-and-limitations) -Before you begin, ensure you have the following: +### Prerequisite -- **{{< product-name >}} Cloud Dedicated plan** or compatible environment +Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/cloud-dedicated/get-started/setup/). -## Request Iceberg integration +### Request Iceberg integration -1. Contact [InfluxData sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration. +1. [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration for your {{% product-name %}} cluster. -2. Be prepared to provide: +2. Be prepared to provide the following information: - Your organization ID - Customer name - Cloud provider and region - External storage details (bucket name, region, permissions) - Technical contact information -> **Note**: Iceberg integration is a premium feature that requires setup by the InfluxData team. - -## Integrate InfluxDB 3 with Snowflake - -Once you've contacted sales and enabled Iceberg integration, your InfluxData support engineers will help you with the following steps: - -1. [Create a Snowflake external stage](#create-a-snowflake-external-stage) -2. [Export InfluxDB time series data to Iceberg format](#export-influxdb-time-series-data-to-iceberg-format) -3. [Create an Iceberg table in Snowflake](#create-an-iceberg-table-in-snowflake) -4. [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake) - -### Create a Snowflake external stage - -Your support engineer will guide you through setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command. This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. - -#### Example: Configure an S3 stage in Snowflake - -```sql -CREATE STAGE my_s3_stage -URL='s3://my-bucket/' -STORAGE_INTEGRATION=my_storage_integration; -``` +### Setup Snowflake and configure your cluster -### Set up a catalog integration in Snowflake +Iceberg integration for {{% product-name %}} requires setup by the InfluxData team. +After you have contacted sales and enabled Iceberg integration, you'll specify which +tables you want to make available for exporting. +InfluxData support engineers will help you with the integration steps, which include: -Your support engineer will help you set up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details. +This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. +- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. +For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export. -#### Example: Create a catalog integration in Snowflake +After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake. -```sql -CREATE CATALOG INTEGRATION my_catalog_integration - CATALOG_SOURCE = 'OBJECT_STORE' - TABLE_FORMAT = 'ICEBERG' - ENABLED = TRUE; -``` +> [!Important] +> #### Export tables are read-only in Snowflake +> +> Tables created through this integration are _read-only_. +> You cannot write directly to these tables using Snowflake or any other engine. +> -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +## Export data to Iceberg format -### Export InfluxDB time series data to Iceberg format +Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg format. -> **Note**: Before exporting InfluxDB time series data to Iceberg format, ensure that the relevant InfluxDB tables are properly set up. Please reach out to your support engineers to configure the tables that need to be exported. +> [!Important] +> Before you can export data to Iceberg format, the InfluxData Support Team needs +> to have set up your tables for exporting. +> For more information, [contact the InfluxData Sales team]({{< cta-link >}}). {{< tabs-wrapper >}} {{% tabs %}} -[CLI](#tab-cli) -[API](#tab-api) +[CLI](#cli) +[API](#api) {{% /tabs %}} - {{% tab-content %}} -{{% tab-pane id="tab-cli" %}} -#### Using the CLI -Use the `influxctl` command to export InfluxDB time-series data to Iceberg format: - -```sh -influxctl snapshot export --namespace foo --table bar +{{% code-placeholders "NAMESPACE|TABLE_NAME" %}} +```bash +influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME ``` -{{% /tab-pane %}} -{{% tab-pane id="tab-api" %}} -#### Using the API - -Use the {{% product-name %}} HTTP API to export snapshots and check status. +{{% /code-placeholders %}} -##### Example: Export a snapshot +Replace the following: -This example demonstrates how to export a snapshot of your data from InfluxDB to an Iceberg table using the HTTP API. +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export -- **Method**: `POST` -- **Endpoint**: `/snapshots/export` -- **Request body**: - -```json -{ - "namespace": "foo", - "table": "bar" -} -``` -The `POST` request to the `/snapshots/export` endpoint triggers the export of data from the specified namespace and table in InfluxDB to an Iceberg table. The request body specifies the namespace (`foo`) and the table (`bar`) to be exported. - -##### Example: Check snapshot status - -This example shows how to check the status of an ongoing or completed snapshot export using the HTTP API. - -- **Method**: `GET` -- **Endpoint**: `/snapshots/status` - -The `GET` request to the `/snapshots/status` endpoint retrieves the status of the snapshot export. This can be used to monitor the progress of the export or verify its completion. -{{% /tab-pane %}} {{% /tab-content %}} -{{< /tabs-wrapper >}} - -#### Creating a configuration file +{{% tab-content %}} -Before running the export command, you need to create a configuration file that specifies which tables to export: +**Endpoint**: -1. Create a file named `config.json` in your working directory -2. Add the following JSON content, adjusting the namespace and table names to match your data: +{{% api-endpoint method="POST" endpoint="/snapshots/export" %}} +**Request body**: + ```json { - "exports": [ - { - "namespace": "company_sensors", - "table_name": "cpu" - } - ] + "namespace": "NAMESPACE", + "table": "TABLE_NAME" } ``` -This configuration file tells the exporter which InfluxDB tables to convert to Iceberg format. You can list multiple tables by adding more objects to the exports array. +The following example shows how to use cURL with the HTTP API: -After configuring the export settings in the `config.json` file, the system automatically handles the export process. The export generates an Iceberg metadata file that you'll reference when creating your Iceberg table in Snowflake. +{{% code-placeholders "NAMESPACE|TABLE_NAME" %}} +```bash +curl -X POST https://{{% influxdb/host %}}/snapshots/export \ + -d '{"namespace": "NAMESPACE", "table": "TABLE_NAME"}' +``` +{{% /code-placeholders %}} -### Create an Iceberg table in Snowflake +Replace the following: -After the export process is complete, you can work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. If you do not have access to a support engineer, refer to the Snowflake documentation or consult your database administrator for assistance. Here's what happens during this step: +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export -After the export process is complete, you'll work with your InfluxData support engineer to create an Iceberg table in Snowflake that references your exported data. Here's what happens during this step: +{{% /tab-content %}} +{{< /tabs-wrapper >}} -> **Note**: **Tables created through this integration are read-only.** -> **You cannot write directly to these tables using Snowflake or any other engine.** -> They serve as an analytics interface to your InfluxDB data. +## Check snapshot status -1. Your support engineer will provide you with the location of the Iceberg metadata file generated during the export process. +Use the HTTP API to check the status of an ongoing or completed snapshot export. -2. Using this information, you or your database administrator will execute a SQL command in Snowflake similar to: +{{% api-endpoint method="GET" endpoint="/snapshots/status" %}} -```sql -CREATE ICEBERG TABLE my_iceberg_table - EXTERNAL_VOLUME = 'my_external_volume' - METADATA_FILE_PATH = 's3://my-bucket/path/to/metadata.json'; +```bash +curl -X GET https://{{% influxdb/host %}}/snapshots/status ``` -3. Your support engineer will help ensure the EXTERNAL_VOLUME and METADATA_FILE_PATH parameters correctly point to your external storage and metadata file. - -This creates a table in Snowflake that reads directly from the Iceberg-formatted data exported from your InfluxDB instance. - -### Query the Iceberg table from Snowflake +## Query the Iceberg table from Snowflake -Once the Iceberg table is set up, you can query it using standard SQL in Snowflake. - -#### Example: Query the Iceberg table - -```sql -SELECT * FROM my_iceberg_table -WHERE timestamp > '2025-01-01'; -``` +Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}. ## Considerations and limitations When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: +- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine. - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. + From b4b4a7e67e4b1d7dce673c983776b6403ac4cf03 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:51:38 -0500 Subject: [PATCH 58/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 07d4a9663b..65f2065af0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -64,7 +64,7 @@ Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg f > [!Important] > Before you can export data to Iceberg format, the InfluxData Support Team needs -> to have set up your tables for exporting. +> to set up your tables for exporting. > For more information, [contact the InfluxData Sales team]({{< cta-link >}}). {{< tabs-wrapper >}} From c65597149775d1b49ba5155c1a6865f473fc8b1c Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:51:56 -0500 Subject: [PATCH 59/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 65f2065af0..8ca9a72a42 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -8,7 +8,7 @@ Export time series data snapshots into Apache Iceberg format and query directly - **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake. > [!Note] -> #### Contact sales to enable Iceberg integration +> #### Contact sales to enable the InfluxDB Iceberg integration > > Iceberg integration is a premium feature that requires setup by the InfluxData team. > [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. From 6034b91df8f0865810ec600268cc30865f1c322a Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:52:16 -0500 Subject: [PATCH 60/62] Update content/shared/influxdb3-query-guides/snapshots/snowflake.md Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index 8ca9a72a42..da56408d2b 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -10,7 +10,7 @@ Export time series data snapshots into Apache Iceberg format and query directly > [!Note] > #### Contact sales to enable the InfluxDB Iceberg integration > -> Iceberg integration is a premium feature that requires setup by the InfluxData team. +> The InfluxDB Iceberg integration is a premium feature. > [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account. - [Prerequisite](#prerequisite) From b11788dcd57bffd4824652ec6cfa23dc81e61df9 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 15:54:04 -0500 Subject: [PATCH 61/62] Apply suggestions from code review Co-authored-by: Scott Anderson --- content/shared/influxdb3-query-guides/snapshots/snowflake.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/shared/influxdb3-query-guides/snapshots/snowflake.md index da56408d2b..a7c06ca894 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/shared/influxdb3-query-guides/snapshots/snowflake.md @@ -38,7 +38,7 @@ Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/c ### Setup Snowflake and configure your cluster -Iceberg integration for {{% product-name %}} requires setup by the InfluxData team. +The {{% product-name %}} Iceberg integration requires setup by the InfluxData team. After you have contacted sales and enabled Iceberg integration, you'll specify which tables you want to make available for exporting. InfluxData support engineers will help you with the integration steps, which include: @@ -60,7 +60,7 @@ After the setup is complete, you can export data snapshots to Iceberg format and ## Export data to Iceberg format -Use the `influxctl` CLI command or the HTTP API to export snapshots to Iceberg format. +Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format. > [!Important] > Before you can export data to Iceberg format, the InfluxData Support Team needs From f5d465b3f148525bca155d7063e8134244eb5cf9 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Thu, 27 Mar 2025 16:17:08 -0500 Subject: [PATCH 62/62] feat(dedicated): iceberg for snowflake: - Merge fixes - Provide instruction for checking snapshot status - Cleanup --- .../export/iceberg-for-snowflake.md | 19 ------- .../use-integrations}/_index.md | 7 ++- .../iceberg-for-snowflake.md} | 57 +++++++++++++++---- 3 files changed, 49 insertions(+), 34 deletions(-) delete mode 100644 content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md rename content/influxdb3/cloud-dedicated/{process-data/export => query-data/use-integrations}/_index.md (73%) rename content/{shared/influxdb3-query-guides/snapshots/snowflake.md => influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md} (64%) diff --git a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md b/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md deleted file mode 100644 index 6cbf1447d3..0000000000 --- a/content/influxdb3/cloud-dedicated/process-data/export/iceberg-for-snowflake.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Export in Iceberg format for Snowflake -seotitle: Export time series data to Iceberg format for Snowflake -description: > - Integrate {{< product-name >}} with Snowflake without complex ETL processes. - Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. -menu: - influxdb3_cloud_dedicated: - name: Snowflake with Iceberg - parent: Export data -weight: 101 -influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] -source: /shared/influxdb3-query-guides/snapshots/snowflake.md ---- - - - - diff --git a/content/influxdb3/cloud-dedicated/process-data/export/_index.md b/content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md similarity index 73% rename from content/influxdb3/cloud-dedicated/process-data/export/_index.md rename to content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md index f00e2ed286..59425da14f 100644 --- a/content/influxdb3/cloud-dedicated/process-data/export/_index.md +++ b/content/influxdb3/cloud-dedicated/query-data/use-integrations/_index.md @@ -1,14 +1,15 @@ --- -title: Export data +title: Use integrations to export and query data description: > Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems and avoid complex ETL processes. menu: influxdb3_cloud_dedicated: - parent: Process & visualize data + parent: Query data + name: Use integrations weight: 103 --- -Use InfluxData export tools to: +Use {{% product-name %}} integrations: - Integrate time series data stored in {{% product-name %}} with other tools and systems - Avoid complex ETL (Extract, Transform, Load) processes diff --git a/content/shared/influxdb3-query-guides/snapshots/snowflake.md b/content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md similarity index 64% rename from content/shared/influxdb3-query-guides/snapshots/snowflake.md rename to content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md index a7c06ca894..63aca4d2a0 100644 --- a/content/shared/influxdb3-query-guides/snapshots/snowflake.md +++ b/content/influxdb3/cloud-dedicated/query-data/use-integrations/iceberg-for-snowflake.md @@ -1,3 +1,17 @@ +--- +title: Export Iceberg for Snowflake +seotitle: Export time series data to Iceberg format for Snowflake +description: > + Integrate {{< product-name >}} with Snowflake without complex ETL processes. + Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. +menu: + influxdb3_cloud_dedicated: + name: Export Iceberg for Snowflake + parent: Use integrations +weight: 101 +influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export] +--- + Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake. @@ -40,16 +54,18 @@ Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/c The {{% product-name %}} Iceberg integration requires setup by the InfluxData team. After you have contacted sales and enabled Iceberg integration, you'll specify which -tables you want to make available for exporting. -InfluxData support engineers will help you with the integration steps, which include: +tables you want to make available for export. +InfluxData support engineers will work with you to set up the integration by: -- Setting up a Snowflake external stage using the `CREATE STAGE` Snowflake SQL command and your storage provider details. -This establishes an external storage location (such as AWS S3) to store Iceberg table data and metadata. -- Setting up a catalog integration in Snowflake to manage and load Iceberg tables efficiently. -For more information, refer to the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). -- Creating an export configuration that tells the InfluxDB Iceberg exporter which tables are available for export. +- Configuring a Snowflake external stage using the `CREATE STAGE` SQL command with your storage provider details. +This creates an external storage location (such as AWS S3) for Iceberg table data and metadata. +- Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables. +For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration). +- Creating an exports configuration that specifies which tables are available for export. + > [!Important] > #### Export tables are read-only in Snowflake @@ -60,6 +76,14 @@ After the setup is complete, you can export data snapshots to Iceberg format and ## Export data to Iceberg format +After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots. +The exported data is stored in Iceberg format in the external storage location you specified. + +_To refresh the exported data with new or changed data, submit a request to the +[InfluxData Support team](https://support.influxdata.com) each time a refresh is needed. +Currently, customer-triggered exports aren't supported._ + + -- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export {{% /tab-content %}} {{% tab-content %}} @@ -112,14 +136,19 @@ curl -X POST https://{{% influxdb/host %}}/snapshots/export \ Replace the following: -- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace of the data to export. -- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The table name to export +- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export +- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export {{% /tab-content %}} {{< /tabs-wrapper >}} +-- Customer-triggered exports aren't supported --> ## Check snapshot status +To check the status of a snapshot export, contact the [InfluxData Support team](https://support.influxdata.com). + + ## Query the Iceberg table from Snowflake @@ -136,8 +166,11 @@ Use SQL in Snowflake to query the Iceberg table exported from {{< product-name > When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind: +- **Export controls limited to InfluxData**: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes. + Submit a request to the [InfluxData Support team](https://support.influxdata.com) to: + - Update your Snowflake tables with new or changed data + - Check the status of a snapshot export - **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine. - **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB. - **Performance**: Query performance may vary based on data size and query complexity. - **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration. -