Skip to content

Iceberg Integration documentation #5918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 68 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
d92a19c
Iceberg Integration
MeelahMe Mar 19, 2025
5d7bc7f
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
f873263
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
8391ce7
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
8b454e0
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
57c024c
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
9563143
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
3f82fb6
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
0478e05
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
4a1f036
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
091c315
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 19, 2025
a32d724
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
d22f3fb
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
0a5d3f5
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
7bc3352
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
22c19be
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
e9345af
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
4693ece
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
8ea52b7
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
836e31b
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 20, 2025
a4eefaa
Merge branch 'master' into inceberg
jstirnaman Mar 20, 2025
3283130
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
6c55f87
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
7c6a630
updating headings
MeelahMe Mar 21, 2025
d353d01
Adding a numbered-list TOC
MeelahMe Mar 21, 2025
677ed17
Added an export command example
MeelahMe Mar 21, 2025
accaffe
Proof read: improving grammar and clarity
MeelahMe Mar 21, 2025
8b8257b
Adding explanations for examples
MeelahMe Mar 21, 2025
ffc8014
Updating examples
MeelahMe Mar 21, 2025
bb7a0d3
Merge branch 'master' into inceberg
MeelahMe Mar 21, 2025
05d4858
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
aa6b36a
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
7e260b5
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
bd472c8
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
1ad0ffa
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
b022d25
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
b712851
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
c5d58ee
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
9adfbda
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
37888c0
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
4e6e034
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
e79725d
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
a4e2b80
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
MeelahMe Mar 21, 2025
aebe441
removing run export example and adding
MeelahMe Mar 26, 2025
277fcde
Changes to- Example: Export data using Iceberg exporter
MeelahMe Mar 26, 2025
07f6551
Adding note to- Export InfluxDB time series data to Iceberg format
MeelahMe Mar 26, 2025
4b2ee68
removing reduandant sections and information
MeelahMe Mar 26, 2025
0c31daa
removing IOX references
MeelahMe Mar 26, 2025
6cd3c82
Rewriting section to focus and clarify that Iceberg table is done wit…
MeelahMe Mar 26, 2025
8adec72
removing entier section: Interfaces for using Iceberg integration sec…
MeelahMe Mar 26, 2025
92208e4
Adding read-only note to section
MeelahMe Mar 26, 2025
c9e26fa
updating TOC numbe4r list
MeelahMe Mar 26, 2025
cb02320
Updating who to contact for Snowflake Integration
MeelahMe Mar 26, 2025
4d259fb
Restructuring, reving, and proof reading edit
MeelahMe Mar 26, 2025
a8badd6
google dev style read through
MeelahMe Mar 26, 2025
83b6570
updating prereqs
MeelahMe Mar 26, 2025
3b56fee
Merge branch 'master' into inceberg
MeelahMe Mar 26, 2025
6eed23c
wip: removing tab shortcode to figure out error
MeelahMe Mar 26, 2025
6c09ef4
Merge branch 'iceberg' of github.com:influxdata/docs-v2 into iceberg
MeelahMe Mar 26, 2025
a013770
adding shortcode structure back
MeelahMe Mar 26, 2025
905edc3
feat(dedicated): iceberg export for snowflake:
jstirnaman Mar 27, 2025
b4b4a7e
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
c655971
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
6034b91
Update content/shared/influxdb3-query-guides/snapshots/snowflake.md
jstirnaman Mar 27, 2025
b11788d
Apply suggestions from code review
jstirnaman Mar 27, 2025
6af2b3f
Merge pull request #5931 from influxdata/jts/iceberg
jstirnaman Mar 27, 2025
f5d465b
feat(dedicated): iceberg for snowflake:
jstirnaman Mar 27, 2025
3a44b1e
Merge branch 'master' into inceberg
MeelahMe Mar 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: Use integrations to export and query data
description: >
Use InfluxData export tools to integrate time series data stored in {{% product-name %}} with other tools and systems
and avoid complex ETL processes.
menu:
influxdb3_cloud_dedicated:
parent: Query data
name: Use integrations
weight: 103
---
Use {{% product-name %}} integrations:

- Integrate time series data stored in {{% product-name %}} with other tools and systems
- Avoid complex ETL (Extract, Transform, Load) processes

{{< children >}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
title: Export Iceberg for Snowflake
seotitle: Export time series data to Iceberg format for Snowflake
description: >
Integrate {{< product-name >}} with Snowflake without complex ETL processes.
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.
menu:
influxdb3_cloud_dedicated:
name: Export Iceberg for Snowflake
parent: Use integrations
weight: 101
influxdb3/cloud-dedicated/tags: [integrations, snowflake, iceberg, export]
---

Integrate {{< product-name >}} with Snowflake without complex ETL processes.
Export time series data snapshots into Apache Iceberg format and query directly from Snowflake.

## Key benefits

- **Efficient data access**: Query your data directly from Snowflake.
- **Cost-effective storage**: Optimize data retention and minimize storage costs.
- **Supports AI and ML workloads**: Enhance machine learning applications by making time-series data accessible in Snowflake.

> [!Note]
> #### Contact sales to enable the InfluxDB Iceberg integration
>
> The InfluxDB Iceberg integration is a premium feature.
> [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to enable this capability for your account.

- [Prerequisite](#prerequisite)
- [Request Iceberg integration](#request-iceberg-integration)
- [Integrate InfluxDB 3 with Snowflake](#integrate-influxdb-3-with-snowflake)
- [Export data to Iceberg format](#export-data-to-iceberg-format)
- [Check snapshot status](#check-snapshot-status)
- [Query the Iceberg table from Snowflake](#query-the-iceberg-table-from-snowflake)
- [Considerations and limitations](#considerations-and-limitations)

### Prerequisite

Before you begin, ensure you have an [{{< product-name >}} account](/influxdb3/cloud-dedicated/get-started/setup/).

### Request Iceberg integration

1. [Contact InfluxData Sales](https://www.influxdata.com/contact-sales/) to request Iceberg integration for your {{% product-name %}} cluster.

2. Be prepared to provide the following information:
- Your organization ID
- Customer name
- Cloud provider and region
- External storage details (bucket name, region, permissions)
- Technical contact information

### Setup Snowflake and configure your cluster

The {{% product-name %}} Iceberg integration requires setup by the InfluxData team.
After you have contacted sales and enabled Iceberg integration, you'll specify which
tables you want to make available for export.
InfluxData support engineers will work with you to set up the integration by:

- Configuring a Snowflake external stage using the `CREATE STAGE` SQL command with your storage provider details.
This creates an external storage location (such as AWS S3) for Iceberg table data and metadata.
- Establishing a catalog integration in Snowflake to efficiently manage and load Iceberg tables.
For more information, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration).
- Creating an exports configuration that specifies which tables are available for export.

<!-- Customer-triggered exports aren't yet supported.
After the setup is complete, you can export data snapshots to Iceberg format and query the Iceberg table from Snowflake.
-->

> [!Important]
> #### Export tables are read-only in Snowflake
>
> Tables created through this integration are _read-only_.
> You cannot write directly to these tables using Snowflake or any other engine.
>

## Export data to Iceberg format

After the setup is complete, the InfluxData Support team triggers the initial export of your data snapshots.
The exported data is stored in Iceberg format in the external storage location you specified.

_To refresh the exported data with new or changed data, submit a request to the
[InfluxData Support team](https://support.influxdata.com) each time a refresh is needed.
Currently, customer-triggered exports aren't supported._

<!-- Customer-triggered exports aren't yet supported.
Use the `influxctl` CLI or the HTTP API to export snapshots to Iceberg format.

> [!Important]
> Before you can export data to Iceberg format, the InfluxData Support Team needs
> to set up your tables for exporting.
> For more information, [contact the InfluxData Sales team]({{< cta-link >}}).

{{< tabs-wrapper >}}
{{% tabs %}}
[CLI](#cli)
[API](#api)
{{% /tabs %}}
{{% tab-content %}}

{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
```bash
influxctl snapshot export --namespace NAMESPACE --table TABLE_NAME
```
{{% /code-placeholders %}}

Replace the following:

- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export

{{% /tab-content %}}
{{% tab-content %}}

**Endpoint**:

{{% api-endpoint method="POST" endpoint="/snapshots/export" %}}

**Request body**:

```json
{
"namespace": "NAMESPACE",
"table": "TABLE_NAME"
}
```

The following example shows how to use cURL with the HTTP API:

{{% code-placeholders "NAMESPACE|TABLE_NAME" %}}
```bash
curl -X POST https://{{% influxdb/host %}}/snapshots/export \
-d '{"namespace": "NAMESPACE", "table": "TABLE_NAME"}'
```
{{% /code-placeholders %}}

Replace the following:

- {{% code-placeholder-key %}}`NAMESPACE`{{% /code-placeholder-key %}}: The namespace (database) that contains the tables to export
- {{% code-placeholder-key %}}`TABLE_NAME`{{% /code-placeholder-key %}}: The name of the table to export

{{% /tab-content %}}
{{< /tabs-wrapper >}}
-- Customer-triggered exports aren't supported -->

## Check snapshot status

To check the status of a snapshot export, contact the [InfluxData Support team](https://support.influxdata.com).

<!-- API isn't yet available to customers.

Use the HTTP API to check the status of an ongoing or completed snapshot export.

{{% api-endpoint method="GET" endpoint="/snapshots/status" %}}

```bash
curl -X GET https://{{% influxdb/host %}}/snapshots/status
```
-- API isn't yet available to customers -->

## Query the Iceberg table from Snowflake

Use SQL in Snowflake to query the Iceberg table exported from {{< product-name >}}.

## Considerations and limitations

When exporting data from InfluxDB to an Iceberg table, keep the following considerations and limitations in mind:

- **Export controls limited to InfluxData**: You can't export or refresh data to Iceberg format independently. The InfluxData Support Team handles set up of the initial export and subsequently handles all data refreshes.
Submit a request to the [InfluxData Support team](https://support.influxdata.com) to:
- Update your Snowflake tables with new or changed data
- Check the status of a snapshot export
- **Exported data is read-only**: Tables created through this integration are _read-only_. You cannot write directly to these tables using Snowflake or any other engine.
- **Data consistency**: Ensure that the exported data in the Iceberg table is consistent with the source data in InfluxDB.
- **Performance**: Query performance may vary based on data size and query complexity.
- **Feature support**: Some advanced features of InfluxDB may not be fully supported in Snowflake through Iceberg integration.
Empty file.