Skip to content

Latest commit

 

History

History
176 lines (131 loc) · 7.12 KB

iceberg-for-snowflake.md

File metadata and controls

176 lines (131 loc) · 7.12 KB
title seotitle description menu weight influxdb3/cloud-dedicated/tags
Export Iceberg for Snowflake
Export time series data to Iceberg format for Snowflake
Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
influxdb3_cloud_dedicated
name parent
Export Iceberg for Snowflake
Use integrations
101
integrations
snowflake
iceberg
export

Integrate {{< product-name >}} with Snowflake without complex ETL processes. Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.

Key benefits

  • Efficient data access: Query your data directly from Snowflake.
  • Cost-effective storage: Optimize data retention and minimize storage costs.
  • Supports AI and ML workloads: Enhance machine learning applications by making time-series data accessible in Snowflake.

Note

Contact sales to enable the InfluxDB Iceberg integration

The InfluxDB Iceberg integration is a premium feature. Contact InfluxData Sales to enable this capability for your account.

Prerequisite

Before you begin, ensure you have an {{< product-name >}} account.

Request Iceberg integration

  1. Contact InfluxData Sales to request Iceberg integration for your {{% product-name %}} cluster.

  2. Be prepared to provide the following information:

    • Your organization ID
    • Customer name
    • Cloud provider and region
    • External storage details (bucket name, region, permissions)
    • Technical contact information

Setup Snowflake and configure your cluster

The {{% product-name %}} Iceberg integration requires setup by the InfluxData team. After you have contacted sales and enabled Iceberg integration, you'll specify which tables you want to make available for export. InfluxData support engineers will work with you to set up the integration by:

  • Configuring a Snowflake external stage using the CREATE STAGE SQL command with your storage provider details. This creates an external storage location (such as AWS S3) for Iceberg table data and metadata.
  • Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables. For more information, see the Snowflake documentation.
  • Creating an exports configuration that specifies which tables are available for export.

Important

Export tables are read-only in Snowflake

Tables created through this integration are read-only.
You cannot write directly to these tables using Snowflake or any other engine.

Export data to Iceberg format

After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots. The exported data is stored in Iceberg format in the external storage location you specified.

To refresh the exported data with new or changed data, submit a request to the InfluxData Support team each time a refresh is needed. Currently, customer-triggered exports aren't supported.

Check snapshot status

To check the status of a snapshot export, contact the InfluxData Support team.

Query the Iceberg table from Snowflake

Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}.

Considerations and limitations

When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind:

  • Export controls limited to InfluxData: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes. Submit a request to the InfluxData Support team to:
    • Update your Snowflake tables with new or changed data
    • Check the status of a snapshot export
  • Exported data is read-only: Tables created through this integration are read-only. You cannot write directly to these tables using Snowflake or any other engine.
  • Data consistency: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB.
  • Performance: Query performance may vary based on data size and query complexity.
  • Feature support: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration.