dbt opening multiple connections to target database - how to restrict number of parallel connections

[josepaul1988]

Tested with different versions of dbt-bigquery as well as dbt-postgres adapters. It seems like even with threads=1 dbt is initating one master connection and another set of connections for each schema defined in the dbt_project.yaml. In our project we cannot use more two connections concurrently to target database. is there a way or configuration that we can use to ensure strictly the number of connections made to the target database. From logs we could see the connections are initiated like below

[0m16:42:27.610328 [debug] [MainThread]: Acquiring new bigquery connection 'master'
[0m16:42:27.671724 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_gdp-project-id'
[0m16:42:27.674011 [debug] [ThreadPool]: Opening a new connection, currently in state init
[0m16:42:32.247486 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_staging'
[0m16:42:32.249455 [debug] [ThreadPool]: Opening a new connection, currently in state init
[0m16:42:32.250455 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_pre_consumption'
[0m16:42:32.252455 [debug] [ThreadPool]: Acquiring new bigquery connection 'list_project-id_consumption'

[mateuscarestiato]

Hi @josepaul1988! This is a known behavior — the threads setting only controls model execution parallelism, not the catalog/schema introspection connections.

What's happening in your logs:

'master' connection: used for metadata operations (always created, 1 connection)
'list_' connections: dbt opens one connection per schema defined in your dbt_project.yml to introspect existing objects — this happens before any model runs
Why threads=1 doesn't help here: These listing connections run in dbt's internal thread pool for catalog building, independent of your thread count.

Options to reduce total connections:

Consolidate schemas — fewer distinct schema names in dbt_project.yml = fewer listing connections:
yaml

dbt_project.yml — avoid many different schemas

models:
my_project:
+schema: main # one schema instead of staging/pre_consumption/consumption
Use --select to scope the run — dbt only introspects schemas relevant to selected models:
bash
dbt run --select tag:daily # limits schema scanning to relevant models
Disable catalog listing with --no-populate-cache (dbt 1.6+):
bash
dbt run --no-populate-cache
⚠️ This skips relation caching — useful if you're hitting connection limits but may cause slower runs on large projects.

For BigQuery specifically — consider using list_relations_without_caching override via a macro if the adapter supports it.
Unfortunately there's no direct config like max_connections=2 — the closest workaround is --no-populate-cache + scoped --select. Would love to see a native config for this in a future dbt version!