Skip to content

[dagster-dbt] Honor DBT_PROFILE/DBT_TARGET in the in-process adapter #33937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Vamsi-klu wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[dagster-dbt] Honor DBT_PROFILE/DBT_TARGET in the in-process adapter #33937

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion python_modules/libraries/dagster-dbt/dagster_dbt/core/resource.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,16 @@ def _initialize_dbt_core_adapter(self, args: Sequence[str]) -> BaseAdapter:
set_from_args(Namespace(profiles_dir=profiles_dir), None)
flags = get_flags()

profile = load_profile(self.project_dir, cli_vars, self.profile, self.target)
# Mirror dbt's CLI flag/env-var resolution. dbt only reads DBT_PROFILE /
# DBT_TARGET at its Click CLI layer (dbt/cli/params.py), so the `dbt`
# subprocess this resource spawns honors them, but the in-process adapter
# used for metadata calls (e.g. column metadata) does not. Fall back to the
# env vars when the resource does not set profile/target explicitly so both
# paths resolve the same profile. Explicit resource config still wins.
# Fixes https://github.com/dagster-io/dagster/issues/33818
profile_override = self.profile or os.getenv("DBT_PROFILE")

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor DBT_PROFILES_DIR with DBT_PROFILE

When DBT_PROFILE is set together with DBT_PROFILES_DIR and the resource has no explicit profiles_dir, the subprocess inherits DBT_PROFILES_DIR, but _initialize_dbt_core_adapter still sets dbt flags to self.project_dir before calling load_profile. This new fallback can therefore look up the env-selected profile in the wrong profiles directory, either failing adapter initialization or using same-named credentials from the project directory while the actual dbt run uses the env profiles directory.

Useful? React with 👍 / 👎.

target_override = self.target or os.getenv("DBT_TARGET")
Comment on lines +396 to +397

@greptile-apps greptile-apps Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Using or for the fallback means an explicitly-set empty string (profile="") would be treated as falsy and the env-var would win instead of the explicit value. While pydantic doesn't enforce non-empty strings on these fields, a more defensive check makes the intent unambiguous and matches how other dbt tooling handles Optional[str] overrides.

Suggested change
profile_override = self.profile or os.getenv("DBT_PROFILE")
target_override = self.target or os.getenv("DBT_TARGET")
profile_override = self.profile if self.profile is not None else os.getenv("DBT_PROFILE")
target_override = self.target if self.target is not None else os.getenv("DBT_TARGET")

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +396 to +397

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Gate DBT env overrides by dbt version

In dbt-core 1.7.x, which validate_dbt_version still accepts, the dbt CLI options for --profile and --target do not read DBT_PROFILE / DBT_TARGET (those envvars were added in dbt 1.8), so the subprocess will keep using the project/default profile while these lines force the in-process adapter to use the environment override. In a supported 1.7 deployment with DBT_TARGET=prod set, metadata calls can now connect to a different target than the actual dbt invocation; only apply this fallback for dbt versions whose CLI honors the same envvars.

Useful? React with 👍 / 👎.

Comment on lines +396 to +397

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Respect CLI flags before env fallback

When a caller supplies --target or --profile in args while DBT_TARGET / DBT_PROFILE is also set, the subprocess receives those explicit flags via full_dbt_args and dbt/Click gives them precedence over envvars, but the in-process adapter never parses args and instead selects the env override here whenever the resource fields are unset. In that scenario, metadata fetching can connect to the env target/profile while the actual dbt run uses the CLI-specified one; parse the CLI flags before falling back to the environment.

Useful? React with 👍 / 👎.

profile = load_profile(self.project_dir, cli_vars, profile_override, target_override)
project = load_project(self.project_dir, False, profile, cli_vars)
config = RuntimeConfig.from_parts(project, profile, flags)

Expand Down
61 changes: 61 additions & 0 deletions python_modules/libraries/dagster-dbt/dagster_dbt_tests/core/test_resource.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,67 @@ def test_dbt_profile_configuration() -> None:
assert dbt_cli_invocation.is_successful()


@pytest.mark.parametrize(
("explicit_profile", "explicit_target", "env", "expected_profile", "expected_target"),
[
# DBT_PROFILE / DBT_TARGET are honored when the resource sets neither, so the
# in-process adapter resolves the same profile as the dbt subprocess.
(None, None, {"DBT_PROFILE": "env_profile", "DBT_TARGET": "prod"}, "env_profile", "prod"),
# Explicit resource config always wins over the env vars.
(
"arg_profile",
"arg_target",
{"DBT_PROFILE": "env_profile", "DBT_TARGET": "prod"},
"arg_profile",
"arg_target",
),
# Nothing set: pass None through so dbt applies its own profiles.yml default.
(None, None, {}, None, None),
],
ids=["env_var_honored", "explicit_arg_wins", "no_override"],
)
def test_initialize_dbt_core_adapter_honors_target_profile_env_vars(
monkeypatch: pytest.MonkeyPatch,
mocker: MockerFixture,
explicit_profile: str | None,
explicit_target: str | None,
env: dict[str, str],
expected_profile: str | None,
expected_target: str | None,
) -> None:
# https://github.com/dagster-io/dagster/issues/33818 - the in-process adapter
# used for metadata calls ignored DBT_PROFILE / DBT_TARGET while the dbt
# subprocess honored them (via dbt's own CLI layer), so the two paths could
# resolve different profiles/targets.
class _StopAdapterInit(Exception):
"""Raised to short-circuit adapter init right after profile resolution."""

monkeypatch.delenv("DBT_PROFILE", raising=False)
monkeypatch.delenv("DBT_TARGET", raising=False)
for key, value in env.items():
monkeypatch.setenv(key, value)

Comment on lines +299 to +306

@greptile-apps greptile-apps Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test may not reach the mocked load_profile

Several real dbt calls run inside _initialize_dbt_core_adapter before load_profile is invoked — specifically set_invocation_context, set_from_args, and get_flags. These are not mocked, so if dbt's internal state isn't clean between parametrize runs (e.g. flags or adapter registry left from a previous test), one of those earlier calls could raise an unexpected exception. pytest.raises(_StopAdapterInit) would not catch it, causing a confusing failure that looks unrelated to the env-var resolution being tested. Mocking or resetting dbt's flags/adapter state around the test body would make the isolation tighter.

# Capture the overrides Dagster passes to dbt, then stop before any real
# adapter / warehouse connection is created.
mock_load_profile = mocker.patch(
"dbt.config.runtime.load_profile", side_effect=_StopAdapterInit
)

resource = DbtCliResource(
project_dir=os.fspath(test_jaffle_shop_path),
profile=explicit_profile,
target=explicit_target,
)

with pytest.raises(_StopAdapterInit):
resource._initialize_dbt_core_adapter(["run"]) # noqa: SLF001

# load_profile(project_dir, cli_vars, profile_name_override, target_override)
_, _, passed_profile, passed_target = mock_load_profile.call_args.args
assert passed_profile == expected_profile
assert passed_target == expected_target


@pytest.mark.parametrize(
"profiles_dir", [None, test_jaffle_shop_path, os.fspath(test_jaffle_shop_path)]
)
Expand Down