meltano-state-backend-bigquery

This is a Meltano extension that provides a BigQuery state backend.

Installation

This package needs to be installed in the same Python environment as Meltano.

From GitHub

With uv

uv tool install --with git+https://github.com/meltano/meltano-state-backend-bigquery.git meltano

With pipx

pipx install meltano
pipx inject meltano git+https://github.com/meltano/meltano-state-backend-bigquery.git

Configuration

To store state in BigQuery, set the state_backend.uri setting to bigquery://<project>/<dataset>.

State will be stored in two tables that Meltano will create automatically:

• meltano_state - Stores the actual state data
• meltano_state_locks - Manages concurrency locks

To authenticate to BigQuery, you can use either:

1. Application Default Credentials (recommended for GCP environments)
2. Service account JSON key file

Using Application Default Credentials

state_backend:
  uri: bigquery://my-project/my-dataset
  bigquery:
    project: my-project
    dataset: my-dataset
    location: US  # Optional: defaults to US

Using Service Account JSON Key

state_backend:
  uri: bigquery://my-project/my-dataset
  bigquery:
    project: my-project
    dataset: my-dataset
    location: US  # Optional: defaults to US
    credentials_path: /path/to/service-account-key.json

Connection Parameters

• project: Your GCP project ID (required)
• dataset: The BigQuery dataset where state will be stored (required)
• location: The BigQuery dataset location (optional, defaults to US)
• credentials_path: Path to service account JSON key file (optional, uses Application Default Credentials if not specified)

Security Considerations

When storing credentials:

• Use environment variables for sensitive values in production
• Consider using Application Default Credentials when running on GCP
• Ensure the service account has the following BigQuery permissions:
  • bigquery.datasets.create (if dataset doesn't exist)
  • bigquery.tables.create (if tables don't exist)
  • bigquery.tables.get
  • bigquery.tables.updateData
  • bigquery.tables.getData

Example using environment variables:

export MELTANO_STATE_BACKEND_BIGQUERY_CREDENTIALS_PATH='/path/to/key.json'
meltano config set meltano state_backend.uri 'bigquery://my-project/my-dataset'

Development

Setup

uv sync

Run tests

Run all tests, type checks, linting, and coverage:

uvx --with tox-uv tox run-parallel

Bump the version

Using the GitHub CLI:

gh release create v<new-version>