dbt-flink-adapter

Build streaming and batch data pipelines with dbt on Apache Flink.

dbt-flink-adapter connects dbt to Apache Flink through the Flink SQL Gateway, bringing version-controlled SQL models, testing, documentation, and lineage to Flink's stream and batch processing engine. A companion CLI tool, dbt-flink-ververica, compiles dbt models and deploys them as managed streaming jobs on Ververica Cloud.

Highlights

• 6 materializations: table, view, streaming_table, incremental, materialized_table, ephemeral
• Streaming-first: watermarks, window TVFs (tumbling, hopping, session, cumulative), Kafka integration
• Batch support: bounded sources (datagen, Kafka, filesystem, JDBC) with batch-optimized macros
• Incremental strategies: append, insert_overwrite, merge (via upsert-capable connectors)
• Model contracts: full dbt 1.5+ schema enforcement and constraint validation
• Catalog introspection: dbt docs generate works with tables, views, and column metadata
• Ververica Cloud deployment: compile, transform, and deploy with a single CLI command

Quick Start

# Install
pip install dbt-flink-adapter

# Start local Flink
cd test-kit && docker compose up -d

# Create project and run
dbt init my_project   # select "flink" adapter
cd my_project && dbt run

Deploy to Ververica Cloud:

cd dbt-flink-ververica && pip install -e .
dbt-flink-ververica auth login --email you@example.com
dbt-flink-ververica workflow --name my-pipeline --workspace-id YOUR_ID --email you@example.com

Documentation

Full documentation lives in docs/:

Guide	Description
Installation	Install the adapter and CLI
Local Quickstart	First pipeline on local Flink in 15 minutes
Ververica Quickstart	End-to-end deploy to Ververica Cloud
Materializations	All 6 materializations with examples
Streaming Pipelines	Watermarks, windows, Kafka pipelines
Batch Processing	Bounded sources and batch macros
Incremental Models	Append, overwrite, merge strategies
Sources & Connectors	Source definitions, CDC, connector setup
Ververica Deployment	Production deployment patterns
CI/CD	GitHub Actions automation
Adapter Config	profiles.yml, dbt_project.yml, model config
CLI Reference	All CLI commands and flags
Macros	Window, watermark, and batch macros
TOML Config	dbt-flink-ververica.toml reference
SQL Transformation	Query hints to SET/DROP pipeline
Flink Compatibility	Version matrix and limitations
Troubleshooting	Common errors and solutions

Prerequisites

Component	Version
Python	3.9 -- 3.13
dbt-core	1.8, 1.9, or 1.10
Apache Flink	1.20+ with SQL Gateway
Docker	Latest (for local Flink)

Example

{{ config(
    materialized='streaming_table',
    execution_mode='streaming',
    columns='window_start TIMESTAMP(3), window_end TIMESTAMP(3), user_id STRING, event_count BIGINT',
    properties={'connector': 'blackhole'}
) }}

SELECT
    window_start, window_end, user_id,
    COUNT(*) as event_count
FROM TABLE(
    TUMBLE(TABLE {{ ref('datagen_source') }}, DESCRIPTOR(event_time), INTERVAL '1' MINUTE)
)
GROUP BY window_start, window_end, user_id

Project Structure

dbt-flink-adapter/
  dbt/include/flink/macros/     # Materializations, window, watermark, batch macros
  dbt-flink-ververica/          # Ververica Cloud CLI tool
  project_example/              # Example dbt project with streaming/batch/incremental models
  test-kit/                     # Docker Compose with Flink, Kafka, PostgreSQL, MySQL
  envs/flink-1.20/              # Minimal Flink 1.20 Docker environment
  scripts/                      # Deployment and utility scripts
  docs/                         # Full documentation site

HTTP Proxy Adapter (Experimental)

An alternative adapter in adapter/ submits compiled SQL directly to a long-running Flink application via a FastAPI HTTP proxy, bypassing the SQL Gateway. See adapter/README.md for details.

Contributing

See CHANGELOG.md for version history. Development documentation lives in .dev-docs/.

License

Apache License 2.0. See LICENSE.