Skip to content

connecting-data.md

Latest commit

 

History

History
105 lines (74 loc) · 5.29 KB

connecting-data.md

File metadata and controls

105 lines (74 loc) · 5.29 KB
icon plug
description Connect your databases, warehouses, lakes, and APIs to Spice.ai Cloud.

Connecting Data

Overview

Spice.ai supports federated SQL queries across a wide range of data sources. Data connectors let you query external data directly or replicate it into Spice for accelerated access.

For the full list of connectors and configuration options, see Data Connectors.

Supported Data Sources

Category Connectors
Relational databases PostgreSQL, MySQL, MSSQL, ClickHouse
Data warehouses Snowflake, Databricks, Dremio
Data lakes & formats S3, Delta Lake, ABFS
APIs & other GraphQL, GitHub, SharePoint
Streaming Debezium CDC
Embedded / local DuckDB, SQLite

Adding a Data Source

1. Store credentials as secrets

Data source credentials should be stored as secrets in your app. In the Portal, go to your app → Secrets and create entries for connection strings, access keys, or tokens.

{% hint style="info" %} Secret values are write-only and cannot be retrieved after saving. {% endhint %}

2. Configure the dataset in your Spicepod

Add a dataset entry to your app's spicepod.yaml referencing the connector and secrets:

datasets:
  - from: postgres:my_schema.my_table
    name: my_table
    params:
      pg_host: my-database.example.com
      pg_port: "5432"
      pg_db: my_database
      pg_user: "${secrets:PG_USER}"
      pg_pass: "${secrets:PG_PASS}"

3. Deploy

Commit and deploy your app to make the dataset available for querying.

Data Acceleration

For faster queries, enable data acceleration to cache datasets in a local engine:

datasets:
  - from: postgres:my_schema.my_table
    name: my_table
    acceleration:
      engine: arrow  # or duckdb, sqlite, postgres
      refresh_interval: 10m

Available acceleration engines:

  • Arrow — In-memory, fastest performance.
  • DuckDB — Embedded analytical engine.
  • SQLite — Lightweight, persistent.
  • PostgreSQL — Full-featured relational engine.

Common Issues

Table not found

Spice normalizes unquoted SQL identifiers to lowercase (PostgreSQL-style). If your source uses mixed-case table or column names, double-quote them:

SELECT "ColumnName" FROM "MySchema"."MyTable"

Object store file format errors

When connecting to object stores (S3, ABFS):

  • Folder paths require an explicit file_format (e.g., parquet, csv).
  • Single file paths auto-detect the format from the file extension.

Schema changes break data refresh

Spice infers the schema at deployment time. If the source schema changes, refresh will fail. Redeploy the app to pick up the new schema.

Connection failures

  • Confirm the data source is network-accessible from Spice.ai Cloud (firewalls, IP allowlists).
  • Verify secrets are correct and referenced with ${secrets:SECRET_NAME} syntax.
  • Check connector-specific docs for required parameters.

Further Reading