diff --git a/.claude/implementation/data_sources.md b/.claude/implementation/data_sources.md new file mode 100644 index 0000000..f0f5c1b --- /dev/null +++ b/.claude/implementation/data_sources.md @@ -0,0 +1,106 @@ +# Data lookups module (`modules/data_lookups`) + +This module centralizes **read-only** account discovery used when YAML references resources that are **not** defined in the same Terraform state (for example connections that already exist in dbt Cloud, or GitHub App installation IDs that are account-specific). + +It mirrors the intent of `modules/projects_v2/data_sources.tf` on the importer branch, but as an explicit child module with clear inputs and outputs so root orchestration stays predictable. + +## When the module is instantiated + +Root enables `module.data_lookups` when **either**: + +- The merged project YAML contains at least one **`LOOKUP:`** global-connection placeholder (see below), **or** +- `var.dbt_pat` is set (so GitHub installations can be fetched from the dbt Cloud integrations API). + +Gating uses `local._lookup_connection_ref_strings` in `variables.tf`; keep that extraction **in sync** with the `lookup_connection_keys` logic in `modules/data_lookups/main.tf`. + +## `LOOKUP:` global connections + +### Syntax + +Use a **string** value that starts with `LOOKUP:` followed by the **exact display name** of an existing global connection in the target dbt Cloud account (the `name` field returned by `data.dbtcloud_global_connections`). + +Example: + +```yaml +environments: + - name: Prod + key: prod + type: deployment + connection_key: "LOOKUP:Snowflake Production" +``` + +The map key passed to `modules/environments` and `modules/profiles` is the **full placeholder string** (e.g. `LOOKUP:Snowflake Production`), not the name alone. + +### Where placeholders are scanned + +- **Environments**: `connection` if set, otherwise `connection_key` (same precedence as `modules/environments` resolution). +- **Profiles**: `connection_key` only. + +### Resolution + +1. `data.dbtcloud_global_connections` runs **only** when at least one such placeholder exists (avoids an unnecessary read). +2. `lookup_connection_ids` maps each placeholder to `tostring(connection.id)` where `connection.name == replace(placeholder, "LOOKUP:", "")`. +3. Root builds `local.global_connection_ids_effective`: + + `merge(lookup_connection_ids, managed_global_connection_ids)` + + **Managed Terraform connections win on key collision** (in practice YAML keys and `LOOKUP:…` keys should not overlap). + +### Validation (V-01) + +`validation.tf` **does not** require `LOOKUP:…` values to appear under `global_connections[]`. Placeholders are intentionally for **pre-existing** connections. If no matching name exists in the account, resolution yields `null` and apply can fail on the environment resource; fixing that is an operational/data issue, not schema validation. + +## GitHub App installations + +When `var.dbt_pat` is non-null, the module calls: + +`GET {dbt_host}/api/v2/integrations/github/installations/` + +with `Authorization: Bearer `. + +Outputs: + +- `github_installation_by_owner` — map of **lowercase** GitHub `account.login` → installation **numeric id**. +- `github_installation_fallback_id` — first installation in the filtered list when owner matching is not used. + +**Note:** Service tokens cannot use this API; use a PAT. Default host for the HTTP call is `coalesce(var.dbt_host_url, "https://cloud.getdbt.com")` with a trailing `/api` segment stripped if present. + +### Consumption in `modules/repository` + +Root passes `module.data_lookups[0].github_installation_by_owner` and `github_installation_fallback_id` into the repository module when `data_lookups` is enabled (same conditions as above). The repository module resolves **`github_installation_id`** in order: + +1. **`repository.github_installation_id`** from YAML, if set +2. **`github_installation_by_owner[lower(owner)]`** where `owner` is parsed from `remote_url` (`github.com//…` or `git@github.com:/…`) +3. **`github_installation_fallback_id`** (first installation returned for the account) + +**Auto-detect GitHub** (`remote_url` on github.com, no explicit `git_clone_strategy`) uses **`github_app`** only when a non-null resolved installation id exists **or** `dbt_pat` is set (discovery may fill the id at apply). Otherwise it uses **`deploy_key`**. + +Explicit **`git_clone_strategy: github_app`** follows the same rule: without YAML id, discovery map entry, fallback, or PAT, strategy downgrades to **`deploy_key`**. + +Root still exposes the GitHub outputs for debugging and for any external callers. + +## Repository `LOOKUP:` (scalar, legacy) + +If `project.repository` is a **scalar** string beginning with `LOOKUP:` (v2 / importer style), it is collected in `lookup_repository_keys`. There is **no** resolution here yet; repository linking for the current v1 object-shaped `repository` block is unchanged. + +## Root outputs + +| Output | Meaning | +|--------|---------| +| `connection_ids` | **Effective** map used by environments/profiles (managed + `LOOKUP:`). | +| `lookup_connection_ids` | Only the `LOOKUP:`-resolved entries. | +| `github_installation_by_owner` / `github_installation_fallback_id` | From integrations API when PAT is set. | + +## Dependencies + +- **Provider**: `hashicorp/http` (declared in root `providers.tf` and the module). +- **dbt Cloud**: `data.dbtcloud_global_connections` uses the default `dbtcloud` provider configuration at root (`dbt_token`, `dbt_account_id`). + +## Extending this module + +When adding new lookup types: + +1. Add **inputs** only if root cannot derive them from existing YAML/locals. +2. Gate **expensive** `data` sources with a `count` tied to a `local.needs_*` flag. +3. Expose stable **outputs**; merge at root if multiple modules need the same id map. +4. Update **this document** and, where relevant, `schemas/v1.json` descriptions. diff --git a/.claude/implementation/resource_metadata.md b/.claude/implementation/resource_metadata.md new file mode 100644 index 0000000..e69de29 diff --git a/.terraform.lock.hcl b/.terraform.lock.hcl index 8dc8cba..3281a62 100644 --- a/.terraform.lock.hcl +++ b/.terraform.lock.hcl @@ -2,26 +2,43 @@ # Manual edits may be lost in future updates. provider "registry.terraform.io/dbt-labs/dbtcloud" { - version = "1.8.2" + version = "1.9.1" constraints = "~> 1.8" hashes = [ - "h1:3Hw1in/qcHdAp2BRhEaFRr/ez9lFjBv3V5WIUxHXcSs=", - "h1:ASwPE1I3U7cufOLFvGV+yHmV4/24y2Jrmt8V3Pn9lEY=", - "h1:BIVjUC8FxluKSvwm2jvySD0AvSkFPKwGn/SHez082Oc=", - "h1:IpC3g4srpeI3iJ5GL3GcxSMO/+XHBktS/zumEm/8xbc=", - "zh:01df4a2592c7d66152aa18bae2fb6abcca205f2c0d75486a67e3c8307ea70af2", - "zh:020e8c80eb8973ac0571b8cb2c990d76c82d2032f590ffaec418fc6728a48594", - "zh:0c30dd1d7ac516efe921362143c0372b2dedab59950662c08513c8c1e0430ee4", - "zh:192bc343bfe193d79b4f2e5d5340a44230e169670d190c7034c7db67fd97b3e7", - "zh:24b8175f22a97844000facea8742bd2f0ca5eedb214c900e6b50153308874e86", - "zh:26ef5f40358401f7fadae72b16782d0a8a8761af325aac17b5507234148c19f9", - "zh:7b9cd9637bf607854dfdeceab06dd361686f54864050cfb770d4214a2d0265db", - "zh:8df586c709b83ca62056fd3d9ff5e9d73833acdcb3194d7e5bbc0cde98f87ed9", - "zh:8e561719c50ae83caefa0dd8ebf3cf4907a512d1be90ecb80eb5db6593f9e9b2", - "zh:9071202c4b459dd28fa792a62e229815bae41d637428fa66dca5eb7fa0fc5801", - "zh:926cca7812b272ab976ce49c0858de78bd5c4c6fcffe16cb2d33cdce63522bc2", - "zh:ae927fc0b059c800b0342b6c6f7217dd191349334a6d6ed1cad578f13f64722c", - "zh:b0e484d5c2dc79b1f573aa3bdac4150a2b4094b6fd9c92033b855e7b8f07045c", - "zh:f96b0ce96fe679653e5dd3820f15e7916855ce8deb832f6b92befedc2387f1f6", + "h1:OygRDw8+b5uCeREVO0TJVs71YjB+QOomqfO+zFUSQwg=", + "zh:01b731b2787f7fbbda8b94100059fe9bbfa613948e017712f433784056db9ae2", + "zh:134a842f52383ab3818e9bc75a285947d356248af5a5b38bbb1fca7c8b6e92ec", + "zh:492fde3092595af2af15ca05cf8718b99f5d722d175de8e7dc4fb81d1e1ff42b", + "zh:53dd23b590eda76f704c0f8acd3d1ebc03b42682c25ef8dc73e5366eaa141ff1", + "zh:578ecaa58259bacda28cec78521f7179e93ac07187c11c4cf2fc00a5907faf30", + "zh:5d2a68740b7b02d21a44cb06554afc8f1b458ecfe6b22f3485f1ba62d2e0ba98", + "zh:6f377f5debf908bb01e837bcc83bc5dc38ef23f1c4afb0922df8135e45e34ada", + "zh:7631851aea4cc39ac1625bf70dfd8bffcd1b9494dc9d4d727fd07a2704e2f27f", + "zh:78da15555a98178b07afce709f4386c5ed8ce75e7cbe9bd13434d96e301ab6fd", + "zh:8224a5d6ef02f888c7acf828f617ee2716984d86ea54e9f4f39ed10db777f55f", + "zh:c137fff71396e3a78910fcd83876e941fa420f2a9f3d9e6f960704bec0226a9c", + "zh:c5ad536754aa3488a354772c585d293d25c29eaab8ca634e8b6e5980b99ed2c4", + "zh:e5e897540808d918107f85caeb27baea55bb51b40cc9cb5aa55f70d8ad242241", + "zh:fd1f648cc0a3afdef1b34ff4a691022e5320f4af004c31fcdad4fd8bd13391b6", + ] +} + +provider "registry.terraform.io/hashicorp/http" { + version = "3.5.0" + constraints = "~> 3.0" + hashes = [ + "h1:dl73+8wzQR++HFGoJgDqY3mj3pm14HUuH/CekVyOj5s=", + "zh:047c5b4920751b13425efe0d011b3a23a3be97d02d9c0e3c60985521c9c456b7", + "zh:157866f700470207561f6d032d344916b82268ecd0cf8174fb11c0674c8d0736", + "zh:1973eb9383b0d83dd4fd5e662f0f16de837d072b64a6b7cd703410d730499476", + "zh:212f833a4e6d020840672f6f88273d62a564f44acb0c857b5961cdb3bbc14c90", + "zh:2c8034bc039fffaa1d4965ca02a8c6d57301e5fa9fff4773e684b46e3f78e76a", + "zh:5df353fc5b2dd31577def9cc1a4ebf0c9a9c2699d223c6b02087a3089c74a1c6", + "zh:672083810d4185076c81b16ad13d1224b9e6ea7f4850951d2ab8d30fa6e41f08", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:7b4200f18abdbe39904b03537e1a78f21ebafe60f1c861a44387d314fda69da6", + "zh:843feacacd86baed820f81a6c9f7bd32cf302db3d7a0f39e87976ebc7a7cc2ee", + "zh:a9ea5096ab91aab260b22e4251c05f08dad2ed77e43e5e4fadcdfd87f2c78926", + "zh:d02b288922811739059e90184c7f76d45d07d3a77cc48d0b15fd3db14e928623", ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index de98ec1..9b2e6f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Documentation now matches the **YAML schema version 1** layout: `version: 1`, `account`, `globals` (connections, service tokens, groups, notifications, PrivateLink), environment field **`connection`** (not `connection_key`), `project_artefacts` / `semantic_layer_config`, and job **`environment_variable_overrides`**. Examples and troubleshooting were updated accordingly. + ### Fixed ### Removed diff --git a/README.md b/README.md index 1ff6f98..90187a6 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ This downloads the [examples/basic/](examples/basic/) starter into `./my-dbt-clo Then: ```bash -cd my-dbt-platform +cd my-dbt-cloud cp .env.example .env # fill in your dbt Cloud credentials # edit dbt-config.yml # replace YOUR_ placeholders with your warehouse details source .env && terraform init && terraform apply @@ -69,7 +69,26 @@ module "dbt_cloud" { **2. Create `dbt-config.yml`** +Configuration uses **`version: 1`**, an **`account`** block (including `host_url` for the dbt Cloud region), shared resources under **`globals`** (connections, service tokens, groups, notifications, PrivateLink endpoints), and a **`projects`** list. Validate in your editor with [`schemas/v1.json`](docs/configuration/yaml-schema.md). + ```yaml +# yaml-language-server: $schema=https://raw.githubusercontent.com/trouze/terraform-dbtcloud-yaml/main/schemas/v1.json + +version: 1 +account: + name: Your Account + host_url: https://cloud.getdbt.com + +globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + details: + host: adb-1234567890.1.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + projects: - name: Analytics key: analytics @@ -81,7 +100,7 @@ projects: environments: - name: Production key: prod - connection_key: databricks_prod # references global_connections key below + connection: databricks_prod # globals.connections[].key (or numeric id / LOOKUP:…) deployment_type: production type: deployment custom_branch: main @@ -93,8 +112,12 @@ projects: - name: Development key: dev - connection_key: databricks_prod + connection: databricks_prod type: development + credential: + credential_type: databricks + catalog: main + schema: analytics_dev jobs: - name: Daily Build @@ -110,14 +133,6 @@ projects: schedule_type: days_of_week schedule_days: [1, 2, 3, 4, 5] schedule_hours: [6] - -global_connections: - - name: Databricks Production - key: databricks_prod - type: databricks - host: adb-1234567890.1.azuredatabricks.net - http_path: /sql/1.0/warehouses/abc123 - catalog: main ``` **3. Create `terraform.tfvars`** @@ -157,9 +172,9 @@ Sensitive values are never in the YAML file. They're passed as Terraform variabl | Variable | Key format | Matches | |---|---|---| -| `token_map` | `"my_token_name"` | `credential.token_name` in YAML (Databricks legacy) | +| `token_map` | `"my_token_name"` | `credential.token_name` (Databricks legacy) or `jobs[].environment_variable_overrides` values prefixed with `secret_` | | `environment_credentials` | `"project_key_env_key"` | Environment credential by composite key | -| `connection_credentials` | `"connection_key"` | `global_connections[].key` in YAML | +| `connection_credentials` | `"connection_key"` | `globals.connections[].key` in YAML | | `lineage_tokens` | `"project_key_integration_key"` | `lineage_integrations[].key` composite | | `oauth_client_secrets` | `"oauth_config_key"` | `oauth_configurations[].key` in YAML | @@ -169,27 +184,27 @@ The composite key for `environment_credentials` uses underscores: a project with ## What you can manage -**Account-level** +**Account-level** (optional unless noted; shared connections and RBAC live under `globals` in YAML) - `account_features` — advanced CI, partial parsing, repo caching flags -- `global_connections` — shared warehouse connections (Databricks, Snowflake, BigQuery, Postgres, Redshift) -- `service_tokens` — API tokens with scoped permissions -- `groups` — user groups with project/account permissions -- `user_groups` — user-to-group assignments -- `notifications` — email, Slack, PagerDuty, webhook alerts +- `globals.connections` — shared warehouse connections (Databricks, Snowflake, BigQuery, Postgres, Redshift, and other adapter types supported by the provider) +- `globals.service_tokens` — API tokens with scoped permissions +- `globals.groups` — user groups with project/account permissions +- `user_groups` — user-to-group assignments (document root) +- `globals.notifications` — job alerts (dbt Cloud user, Slack channel, or external email) - `oauth_configurations` — OAuth provider configs - `ip_restrictions` — IP allowlist/denylist rules **Per-project** -- `repository` — Git integration (GitHub App, GitLab deploy token, Azure DevOps, SSH) -- `environments` — deployment and development environments -- `credentials` — warehouse credentials (14 types: Databricks, Snowflake password/keypair, BigQuery, Postgres, Redshift, Athena, Fabric, Synapse, Starburst, Spark, Teradata) -- `jobs` — scheduled, CI, merge, and on-demand jobs -- `environment_variables` — project and environment-level dbt vars -- `extended_attributes` — connection-level overrides per environment -- `profiles` — links connection + credential + extended attributes -- `lineage_integrations` — Tableau/Looker lineage config -- `artefacts` — docs job and freshness job links -- `semantic_layer` — semantic layer configuration +- `repository` — Git integration (GitHub App, GitLab, Azure DevOps, deploy key/token) +- `environments` — deployment and development environments (reference a global connection with `connection`, or use `primary_profile_key` when using profiles) +- per-environment `credential` — warehouse credentials (many adapter types; secrets via `environment_credentials`) +- `jobs` — scheduled, CI, merge, and other job types; optional `environment_variable_overrides` for job-specific env vars +- `environment_variables` — project- and environment-scoped dbt vars (with map or list `environment_values` forms normalized at apply time) +- `extended_attributes` — connection-level override payloads linked from environments +- `profiles` — link connection, credentials, and extended attributes for deployment environments +- `lineage_integrations` — Tableau / Looker lineage config +- `project_artefacts` — docs job and freshness job keys +- `semantic_layer_config` — semantic layer target environment --- @@ -198,11 +213,12 @@ The composite key for `environment_credentials` uses underscores: a project with Set `protected: true` on any resource to prevent accidental deletion: ```yaml -global_connections: - - name: Databricks Production - key: databricks_prod - protected: true # terraform destroy will be blocked for this resource - ... +globals: + connections: + - name: Databricks Production + key: databricks_prod + protected: true # terraform destroy will be blocked for this resource + ... projects: - name: Analytics @@ -241,8 +257,6 @@ terraform apply -var="yaml_file=./configs/finance.yml" terraform apply -var="yaml_file=./configs/marketing.yml" ``` -**Backward compatibility:** If your existing YAML uses the singular `project:` key, it still works — the module automatically wraps it in a list. - --- ## Job scheduling diff --git a/docs/configuration/environment-variables.md b/docs/configuration/environment-variables.md index fd546ee..7b6d017 100644 --- a/docs/configuration/environment-variables.md +++ b/docs/configuration/environment-variables.md @@ -199,7 +199,7 @@ The key `"analytics_prod"` maps to a project with `key: analytics` and an enviro ### `connection_credentials` -Map of connection credential objects for global connections, keyed by `global_connections[].key`: +Map of connection credential objects for global connections, keyed by `globals.connections[].key`: ```bash export TF_VAR_connection_credentials='{ @@ -216,13 +216,16 @@ export TF_VAR_connection_credentials='{ ### `token_map` -Legacy Databricks token map, keyed by `credential.token_name` in YAML: +Used in two ways: + +1. **Legacy Databricks** — keyed by `credential.token_name` in an environment `credential` block. +2. **Job env var overrides** — when a job sets `environment_variable_overrides` and a value starts with `secret_`, the prefix is removed and the remainder is looked up in this map (see [YAML Schema](yaml-schema.md)). ```bash -export TF_VAR_token_map='{"my_databricks_token": "dapi_abc123"}' +export TF_VAR_token_map='{"my_databricks_token": "dapi_abc123", "ci_override_secret": "sensitive-value"}' ``` -This is the older pattern. Prefer `environment_credentials` for new setups. +For warehouse credentials, prefer `environment_credentials` over legacy `token_name` when possible. ### `lineage_tokens` diff --git a/docs/configuration/yaml-schema.md b/docs/configuration/yaml-schema.md index ce06f90..939289e 100644 --- a/docs/configuration/yaml-schema.md +++ b/docs/configuration/yaml-schema.md @@ -6,20 +6,27 @@ Complete field reference for `dbt-config.yml`. Every key the module reads is doc ## Full skeleton -The top-level keys available in any `dbt-config.yml`: +Terraform requires **`version: 1`**, **`account`**, and **`projects`**. Shared resources live under **`globals`**; the root module hoists `globals.connections`, `globals.privatelink_endpoints`, `globals.service_tokens`, `globals.groups`, and `globals.notifications` into the internal shapes modules consume. Optional account-level keys (`oauth_configurations`, `ip_restrictions`, `account_features`, `user_groups`) stay at the document root. Validate with `schemas/v1.json`. ```yaml -# ── Account-level (all optional — omit any section you don't need) ──────────── -account_features: { ... } -global_connections: [ ... ] -service_tokens: [ ... ] -groups: [ ... ] -user_groups: [ ... ] -notifications: [ ... ] +version: 1 +account: + name: ... + host_url: https://cloud.getdbt.com + +globals: + connections: [ ... ] + privatelink_endpoints: [ ... ] + service_tokens: [ ... ] + groups: [ ... ] + notifications: [ ... ] + oauth_configurations: [ ... ] ip_restrictions: [ ... ] +account_features: { ... } +user_groups: [ ... ] +metadata: { ... } -# ── Projects (required) ─────────────────────────────────────────────────────── projects: - name: Analytics key: analytics @@ -31,13 +38,10 @@ projects: extended_attributes: [ ... ] profiles: [ ... ] lineage_integrations: [ ... ] - artefacts: { ... } - semantic_layer: { ... } + project_artefacts: { ... } + semantic_layer_config: { ... } ``` -!!! note "Backward compatibility" - The singular `project:` key is accepted and automatically wrapped into a one-element list. Existing single-project configs work without change. - --- ## `account_features` @@ -59,16 +63,16 @@ account_features: --- -## `global_connections` +## Global connections (`globals.connections`) -Account-level warehouse connections shared across projects. Reference them from environments using `connection_key`. +Define warehouse connections under **`globals.connections`** in YAML. Terraform hoists them to an internal `global_connections` list. Environments reference a connection with **`connection`** (key, numeric id, or `LOOKUP:…`). ### Common fields | Field | Type | Required | Default | Description | |---|---|---|---|---| | `name` | string | **yes** | — | Display name in dbt Cloud | -| `key` | string | no | `name` | Unique identifier used in `connection_key` references | +| `key` | string | **yes** | — | Unique identifier used in `environments[].connection` references | | `type` | string | **yes** | — | Adapter type — see valid values below | | `private_link_endpoint_id` | string | no | null | PrivateLink endpoint ID | | `protected` | bool | no | false | Prevents `terraform destroy` | @@ -78,16 +82,17 @@ Account-level warehouse connections shared across projects. Reference them from ### Databricks ```yaml -global_connections: - - name: Databricks Production - key: databricks_prod - type: databricks - host: adb-1234567890123456.7.azuredatabricks.net - http_path: /sql/1.0/warehouses/abc1234def567890 - catalog: main # optional — Unity Catalog catalog name - private_link_endpoint_id: null # optional - protected: false - # OAuth credentials via connection_credentials["databricks_prod"] +globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + host: adb-1234567890123456.7.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc1234def567890 + catalog: main # optional — Unity Catalog catalog name + private_link_endpoint_id: null # optional + protected: false + # OAuth credentials via connection_credentials["databricks_prod"] ``` | Field | Type | Required | Default | @@ -99,17 +104,18 @@ global_connections: ### Snowflake ```yaml -global_connections: - - name: Snowflake Production - key: snowflake_prod - type: snowflake - account: xy12345.us-east-1 - database: ANALYTICS - warehouse: TRANSFORMING - role: TRANSFORMER # optional - allow_sso: false # optional - client_session_keep_alive: false # optional - # OAuth credentials via connection_credentials["snowflake_prod"] +globals: + connections: + - name: Snowflake Production + key: snowflake_prod + type: snowflake + account: xy12345.us-east-1 + database: ANALYTICS + warehouse: TRANSFORMING + role: TRANSFORMER # optional + allow_sso: false # optional + client_session_keep_alive: false # optional + # OAuth credentials via connection_credentials["snowflake_prod"] ``` | Field | Type | Required | Default | @@ -124,20 +130,21 @@ global_connections: ### BigQuery ```yaml -global_connections: - - name: BigQuery Production - key: bigquery_prod - type: bigquery - gcp_project_id: my-gcp-project-id - client_email: dbt-sa@my-project.iam.gserviceaccount.com # optional - client_id: "123456789012345678901" # optional - auth_uri: https://accounts.google.com/o/oauth2/auth # optional - token_uri: https://oauth2.googleapis.com/token # optional - auth_provider_x509_cert_url: https://www.googleapis.com/oauth2/v1/certs # optional - client_x509_cert_url: https://www.googleapis.com/... # optional - timeout_seconds: 300 # optional - location: US # optional - # private_key / private_key_id via connection_credentials["bigquery_prod"] +globals: + connections: + - name: BigQuery Production + key: bigquery_prod + type: bigquery + gcp_project_id: my-gcp-project-id + client_email: dbt-sa@my-project.iam.gserviceaccount.com # optional + client_id: "123456789012345678901" # optional + auth_uri: https://accounts.google.com/o/oauth2/auth # optional + token_uri: https://oauth2.googleapis.com/token # optional + auth_provider_x509_cert_url: https://www.googleapis.com/oauth2/v1/certs # optional + client_x509_cert_url: https://www.googleapis.com/... # optional + timeout_seconds: 300 # optional + location: US # optional + # private_key / private_key_id via connection_credentials["bigquery_prod"] ``` | Field | Type | Required | Default | @@ -155,13 +162,14 @@ global_connections: ### Postgres ```yaml -global_connections: - - name: Postgres Production - key: postgres_prod - type: postgres - hostname: my-host.rds.amazonaws.com - dbname: analytics - port: 5432 # optional — default 5432 +globals: + connections: + - name: Postgres Production + key: postgres_prod + type: postgres + hostname: my-host.rds.amazonaws.com + dbname: analytics + port: 5432 # optional — default 5432 ``` | Field | Type | Required | Default | @@ -173,13 +181,14 @@ global_connections: ### Redshift ```yaml -global_connections: - - name: Redshift Production - key: redshift_prod - type: redshift - hostname: my-cluster.abc123.us-east-1.redshift.amazonaws.com - dbname: analytics - port: 5439 # optional — default 5439 +globals: + connections: + - name: Redshift Production + key: redshift_prod + type: redshift + hostname: my-cluster.abc123.us-east-1.redshift.amazonaws.com + dbname: analytics + port: 5439 # optional — default 5439 ``` | Field | Type | Required | Default | @@ -192,7 +201,7 @@ global_connections: ## `service_tokens` -Account-level API service tokens. +Account-level API service tokens. In YAML they belong under **`globals.service_tokens`** (the root module hoists them for modules). | Field | Type | Required | Default | Description | |---|---|---|---|---| @@ -212,23 +221,24 @@ Account-level API service tokens. **Valid `permission_set` values:** `account_admin` · `git_admin` · `job_admin` · `job_runner` · `job_viewer` · `member` · `metadata_only` · `owner` · `readonly` · `seeker_user` · `webhook_admin` ```yaml -service_tokens: - - name: CI Service Token - key: ci_token - protected: false - permissions: - - permission_set: job_runner - all_projects: true - - permission_set: git_admin - all_projects: false - project_id: 12345 +globals: + service_tokens: + - name: CI Service Token + key: ci_token + protected: false + permissions: + - permission_set: job_runner + all_projects: true + - permission_set: git_admin + all_projects: false + project_id: 12345 ``` --- ## `groups` -Account-level user groups. +Account-level user groups. In YAML they belong under **`globals.groups`**. | Field | Type | Required | Default | Description | |---|---|---|---|---| @@ -244,22 +254,23 @@ Account-level user groups. Same structure as `service_tokens[].permissions[]` above. ```yaml -groups: - - name: Developers - key: developers - assign_by_default: false - sso_mapping_groups: - - "data-team-eng" - permissions: - - permission_set: job_runner - all_projects: true +globals: + groups: + - name: Developers + key: developers + assign_by_default: false + sso_mapping_groups: + - "data-team-eng" + permissions: + - permission_set: job_runner + all_projects: true ``` --- ## `user_groups` -Assigns existing dbt Cloud users to groups. `group_keys` references `groups[].key`. +Assigns existing dbt Cloud users to groups. `group_keys` references **`globals.groups[].key`**. | Field | Type | Required | Default | |---|---|---|---| @@ -278,7 +289,7 @@ user_groups: ## `notifications` -Job notification rules. +Job notification rules. In YAML they belong under **`globals.notifications`**. | Field | Type | Required | Default | Description | |---|---|---|---|---| @@ -303,30 +314,31 @@ Job notification rules. | `3` | External email address | ```yaml -notifications: - # dbt Cloud user notification - - name: prod-failures-user - key: prod_failures_user - notification_type: 1 - user_id: 12345 - on_failure: [1001, 1002] - on_success: [] - - # Slack channel notification - - name: prod-failures-slack - key: prod_failures_slack - notification_type: 2 - slack_channel_id: C0123456789 - slack_channel_name: "#dbt-alerts" - on_failure: [1001, 1002] - on_cancel: [1001] - - # External email notification - - name: prod-failures-email - key: prod_failures_email - notification_type: 3 - external_email: oncall@example.com - on_failure: [1001] +globals: + notifications: + # dbt Cloud user notification + - name: prod-failures-user + key: prod_failures_user + notification_type: 1 + user_id: 12345 + on_failure: [1001, 1002] + on_success: [] + + # Slack channel notification + - name: prod-failures-slack + key: prod_failures_slack + notification_type: 2 + slack_channel_id: C0123456789 + slack_channel_name: "#dbt-alerts" + on_failure: [1001, 1002] + on_cancel: [1001] + + # External email notification + - name: prod-failures-email + key: prod_failures_email + notification_type: 3 + external_email: oncall@example.com + on_failure: [1001] ``` --- @@ -483,7 +495,7 @@ The strategy is auto-detected from the presence of `github_installation_id`, `gi | `key` | string | no | `name` | Unique identifier used in `deferring_environment_key`, etc. | | `type` | string | **yes** | — | `deployment` or `development` | | `deployment_type` | string | conditional | — | `production` or `staging` — required when `type: deployment` | -| `connection_key` | string | no | null | References `global_connections[].key` | +| `connection` | string | conditional | null | Global connection key, numeric id, or `LOOKUP:…` — omit when using `primary_profile_key` (either `connection` or `primary_profile_key` must be set) | | `dbt_version` | string | no | null | Pin dbt Core version (e.g., `"1.9.0"`) | | `custom_branch` | string | no | null | Custom git branch (development envs) | | `enable_model_query_history` | bool | no | null | Enable query history tracking | @@ -497,7 +509,7 @@ environments: key: prod type: deployment deployment_type: production - connection_key: databricks_prod + connection: databricks_prod dbt_version: "1.9.0" custom_branch: main enable_model_query_history: false @@ -747,7 +759,7 @@ Jobs are defined at project level and reference environments by `key`. They are | `dbt_version` | string | no | null | Pin dbt Core version (overrides environment) | | `num_threads` | number | no | 4 | Thread count | | `target_name` | string | no | null | dbt target name (e.g., `"prod"`) | -| `timeout_seconds` | number | no | 0 | Job timeout — `0` means no timeout | +| `timeout_seconds` | number | no | `0` | Job timeout in seconds — `0` means no timeout (provider default) | | `is_active` | bool | no | true | Whether the job is enabled | | `job_type` | string | no | — | `"scheduled"` · `"ci"` · `"merge"` | | `generate_docs` | bool | no | false | Regenerate docs on each run | @@ -760,7 +772,25 @@ Jobs are defined at project level and reference environments by `key`. They are | `self_deferring` | bool | no | null | Defer to the job's own previous run | | `force_node_selection` | bool | no | auto | SAO — set automatically; null for CI/merge jobs | | `protected` | bool | no | false | Prevents `terraform destroy` | -| `env_var_overrides` | map(string) | no | {} | Job-level env var overrides | +| `environment_variable_overrides` | map(string) | no | `{}` | Job-level env var overrides — creates `dbtcloud_environment_variable_job_override` resources | + +#### `environment_variable_overrides` + +Map of variable name to string value. Plain strings are written as the override `raw_value`. If a value starts with `secret_`, the prefix is stripped and the rest is used as a key into the Terraform `token_map` variable (same pattern as legacy Databricks `credential.token_name`). + +```yaml +jobs: + - name: Production Daily + key: prod_daily + environment_key: prod + execute_steps: + - dbt build + triggers: + schedule: true + environment_variable_overrides: + DBT_SOME_FLAG: "1" + DBT_SECRET_TOKEN: secret_ci_dbt_token # resolved from token_map["ci_dbt_token"] +``` #### `triggers` @@ -805,7 +835,7 @@ jobs: deferring_environment_key: prod self_deferring: true protected: true - env_var_overrides: + environment_variable_overrides: DBT_TARGET: prod # CI job — triggered on PR @@ -884,26 +914,26 @@ environment_variables: ### `extended_attributes[]` -Per-project connection-level overrides applied at the environment level. Linked to environments via `extended_attributes_key`. +Per-project connection-level overrides applied at the environment level. Linked from environments via `extended_attributes_key`. Terraform JSON-encodes the inner `extended_attributes` object for the dbt Cloud API (adapter-specific keys such as `databricks`, `snowflake`, etc.). | Field | Type | Required | Description | |---|---|---|---| -| `name` | string | **yes** | Display name | -| `key` | string | no | Unique identifier — referenced by `environments[].extended_attributes_key` | -| `content` | map | no | Nested YAML object of connection overrides | +| `key` | string | **yes** (recommended) | Unique slug — referenced by `environments[].extended_attributes_key`. If omitted, Terraform falls back to `name`. | +| `name` | string | no | Human-readable label (can be used as the identifier when `key` is omitted) | +| `extended_attributes` | object | **yes** | Map of connection override fields (structure depends on warehouse type) | +| `protected` | bool | no | Prevents `terraform destroy` when `true` | +| `id` | number | no | Legacy dbt Cloud id for import / remap | ```yaml extended_attributes: - - name: Databricks HTTP Override - key: databricks_overrides - content: + - key: databricks_overrides + extended_attributes: databricks: http_path: /sql/1.0/warehouses/override-warehouse-id catalog: overridden_catalog - - name: Snowflake Warehouse Override - key: snowflake_overrides - content: + - key: snowflake_overrides + extended_attributes: snowflake: warehouse: HIGH_MEMORY_WH ``` @@ -918,10 +948,9 @@ Links a global connection + environment credential + extended attributes into a |---|---|---|---|---| | `name` | string | **yes** | — | Display name | | `key` | string | no | `name` | Unique identifier | -| `connection_key` | string | no | null | References `global_connections[].key` | -| `connection_id` | number | no | null | Numeric connection ID (alternative to `connection_key`) | -| `credential_key` | string | no | null | Composite key `"{project_key}_{env_key}"` | -| `credentials_id` | number | no | null | Numeric credential ID (alternative to `credential_key`) | +| `connection_key` | string | **yes** | — | References `globals.connections[].key` | +| `credentials_key` | string | **yes** | — | Credential key / composite reference (see profiles module) | +| `credentials_id` | number | no | null | Source credential id for import / remap | | `extended_attributes_key` | string | no | null | References `extended_attributes[].key` | ```yaml @@ -929,7 +958,7 @@ Links a global connection + environment credential + extended attributes into a - name: prod-profile key: prod_profile connection_key: databricks_prod - credential_key: analytics_prod + credentials_key: analytics_prod extended_attributes_key: databricks_overrides ``` @@ -962,37 +991,38 @@ Per-project lineage integrations (e.g., Tableau, Looker). --- -### `artefacts` +### `project_artefacts` Links the project's documentation and source freshness jobs. Both fields reference `jobs[].key`. | Field | Type | Required | Description | |---|---|---|---| -| `docs_job` | string | no | Job key for the documentation artifact | -| `freshness_job` | string | no | Job key for the source freshness artifact | +| `docs_job_key` | string | no | Job key for the documentation artifact | +| `freshness_job_key` | string | no | Job key for the source freshness artifact | ```yaml - artefacts: - docs_job: prod_daily - freshness_job: prod_daily + project_artefacts: + docs_job_key: prod_daily + freshness_job_key: prod_daily ``` --- -### `semantic_layer` +### `semantic_layer_config` Configures the dbt Semantic Layer for a project. | Field | Type | Required | Description | |---|---|---|---| -| `environment` | string | **yes** | References `environments[].key` | +| `environment_id` | string | no | Numeric environment id (pass-through) | +| `environment_key` / `environment` | string | no | References `environments[].key` | !!! warning "Create-only" The semantic layer configuration cannot be imported. It is created once and Terraform will not attempt to update it on subsequent runs if it already exists. ```yaml - semantic_layer: - environment: prod + semantic_layer_config: + environment_key: prod ``` --- @@ -1005,7 +1035,7 @@ Sensitive values are never written directly in YAML. Instead, they are passed as |---|---|---| | `token_map` | `"my_token_name"` | `credential.token_name` in YAML (legacy Databricks) | | `environment_credentials` | `"project_key_env_key"` | Environment `credential` block | -| `connection_credentials` | `"connection_key"` | `global_connections[].key` | +| `connection_credentials` | `"connection_key"` | `globals.connections[].key` | | `lineage_tokens` | `"project_key_integration_key"` | `lineage_integrations[].key` composite | | `oauth_client_secrets` | `"oauth_config_key"` | `oauth_configurations[].key` | diff --git a/docs/getting-started/examples.md b/docs/getting-started/examples.md index 0b75583..05fa092 100644 --- a/docs/getting-started/examples.md +++ b/docs/getting-started/examples.md @@ -49,6 +49,28 @@ terraform apply Store multiple projects in a single YAML file. All share one Terraform state. ```yaml title="dbt-config.yml" +version: 1 +account: + name: Your Account + host_url: https://cloud.getdbt.com + +globals: + connections: + - name: Databricks Shared + key: databricks_prod + type: databricks + details: + host: adb-1234567890.1.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + - name: Snowflake Shared + key: snowflake_prod + type: snowflake + details: + account: xy12345 + database: ANALYTICS + warehouse: TRANSFORMING + projects: - name: Finance Analytics key: finance @@ -60,7 +82,7 @@ projects: key: prod type: deployment deployment_type: production - connection_key: databricks_prod + connection: databricks_prod credential: credential_type: databricks catalog: main @@ -86,7 +108,7 @@ projects: key: prod type: deployment deployment_type: production - connection_key: snowflake_prod + connection: snowflake_prod credential: credential_type: snowflake auth_type: password @@ -161,6 +183,21 @@ terraform apply -var="yaml_file=./configs/marketing.yml" Development, staging, and production environments in one project, with job deferral: ```yaml title="dbt-config.yml" +version: 1 +account: + name: Your Account + host_url: https://cloud.getdbt.com + +globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + details: + host: adb-1234567890.1.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + projects: - name: Analytics key: analytics @@ -172,7 +209,7 @@ projects: - name: Development key: dev type: development - connection_key: databricks_prod + connection: databricks_prod custom_branch: develop credential: credential_type: databricks @@ -183,7 +220,7 @@ projects: key: staging type: deployment deployment_type: staging - connection_key: databricks_prod + connection: databricks_prod credential: credential_type: databricks catalog: main @@ -193,7 +230,7 @@ projects: key: prod type: deployment deployment_type: production - connection_key: databricks_prod + connection: databricks_prod protected: true credential: credential_type: databricks @@ -256,37 +293,45 @@ See [YAML Schema](../configuration/yaml-schema.md) for every field with types, d ### Quick Reference ```yaml -# Account-level (optional) +version: 1 +account: + name: Your Account + host_url: https://cloud.getdbt.com + +# Optional account-level flags account_features: advanced_ci: true partial_parsing: true -global_connections: - - name: Databricks Production - key: databricks_prod - type: databricks - host: adb-1234.azuredatabricks.net - http_path: /sql/1.0/warehouses/abc123 - -service_tokens: - - name: CI Service Token - key: ci_token - permissions: - - permission_set: job_runner - all_projects: true - -groups: - - name: Developers - key: developers - assign_by_default: false - -notifications: - - name: prod-failures - key: prod_failures - notification_type: 2 # 2 = Slack - slack_channel_id: C0123456789 - slack_channel_name: "#dbt-alerts" - on_failure: [] +globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + details: + host: adb-1234.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + + service_tokens: + - name: CI Service Token + key: ci_token + permissions: + - permission_set: job_runner + all_projects: true + + groups: + - name: Developers + key: developers + assign_by_default: false + + notifications: + - name: prod-failures + key: prod_failures + notification_type: 2 # 2 = Slack + slack_channel_id: C0123456789 + slack_channel_name: "#dbt-alerts" + on_failure: [] # Projects (required) projects: @@ -303,7 +348,7 @@ projects: key: prod type: deployment deployment_type: production - connection_key: databricks_prod + connection: databricks_prod protected: true credential: credential_type: databricks diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index a52afe8..a7fbac1 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -80,6 +80,23 @@ export TF_VAR_environment_credentials='{ Edit `dbt-config.yml` with your project details: ```yaml +# yaml-language-server: $schema=https://raw.githubusercontent.com/trouze/terraform-dbtcloud-yaml/main/schemas/v1.json + +version: 1 +account: + name: Your Account + host_url: https://cloud.getdbt.com + +globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + details: + host: adb-1234567890.1.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + projects: - name: Analytics key: analytics @@ -92,7 +109,7 @@ projects: key: prod type: deployment deployment_type: production - connection_key: databricks_prod # References global_connections[].key + connection: databricks_prod # globals.connections[].key (or id / LOOKUP:…) credential: credential_type: databricks catalog: main @@ -110,8 +127,8 @@ projects: schedule_hours: [6] # 6 AM UTC ``` -!!! info "connection_key" - Instead of a raw numeric `connection_id`, environments reference connections by `connection_key` — matching `global_connections[].key` in the YAML. This keeps your config portable and readable. +!!! info "`connection`" + Environments reference a global connection with **`connection`**: the `key` from `globals.connections`, a numeric dbt Cloud connection id, or a `LOOKUP:…` placeholder for existing account connections. Alternatively, set **`primary_profile_key`** to use a profile instead of `connection`. !!! info "Jobs at project level" Jobs are defined at the project level with an `environment_key` field, not nested inside environments. This makes them easier to read and reference by key for deferral. diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index f1a36d4..0cf23d3 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -190,22 +190,23 @@ Error: dbt Cloud API error: 403 Forbidden Error: Cannot find connection with key: "my_connection" ``` -**Cause:** The `connection_key` in your environment doesn't match any `global_connections[].key`. +**Cause:** The environment's `connection` value doesn't match any `globals.connections[].key` (and isn't a valid numeric connection id or `LOOKUP:…` reference). **Solutions:** -1. **Check your global_connections keys:** +1. **Check your global connection keys under `globals`:** ```yaml -global_connections: - - name: Databricks Production - key: databricks_prod # ← this is the key +globals: + connections: + - name: Databricks Production + key: databricks_prod # ← this is the key ``` -2. **Make sure your environment references it correctly:** +2. **Reference that key from the environment with `connection`:** ```yaml environments: - name: Production - connection_key: databricks_prod # ← must match exactly + connection: databricks_prod # ← must match globals.connections[].key (unless using id / LOOKUP) ``` --- @@ -277,7 +278,7 @@ environments: key: prod # Required type: deployment # Required deployment_type: production # Required for deployment envs - connection_key: my_conn # References global_connections[].key + connection: my_conn # References globals.connections[].key credential: credential_type: databricks schema: analytics diff --git a/docs/index.md b/docs/index.md index 30dcf04..1294bf4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -47,6 +47,23 @@ This module bridges that gap: you describe your dbt Cloud setup in YAML, and Ter === "Step 2: Create dbt-config.yml" ```yaml + # yaml-language-server: $schema=https://raw.githubusercontent.com/trouze/terraform-dbtcloud-yaml/main/schemas/v1.json + + version: 1 + account: + name: Your Account + host_url: https://cloud.getdbt.com + + globals: + connections: + - name: Databricks Production + key: databricks_prod + type: databricks + details: + host: adb-1234567890.1.azuredatabricks.net + http_path: /sql/1.0/warehouses/abc123 + catalog: main + projects: - name: Analytics key: analytics @@ -59,7 +76,7 @@ This module bridges that gap: you describe your dbt Cloud setup in YAML, and Ter key: prod type: deployment deployment_type: production - connection_key: databricks_prod # references global_connections[].key + connection: databricks_prod # globals.connections[].key credential: credential_type: databricks catalog: main @@ -115,8 +132,8 @@ This module bridges that gap: you describe your dbt Cloud setup in YAML, and Ter | Scope | Resources | |-------|-----------| -| Account | Projects, global connections, service tokens, groups, user groups, notifications, OAuth configurations, IP restrictions, account features | -| Project | Repository, environments, credentials (14 warehouse types), jobs, environment variables, extended attributes, profiles, lineage integrations, project artefacts, semantic layer | +| Account | Projects, `globals.connections`, `globals.service_tokens`, `globals.groups`, user groups, `globals.notifications`, OAuth configurations, IP restrictions, account features | +| Project | Repository, environments, credentials (many warehouse types), jobs (incl. `environment_variable_overrides`), environment variables, extended attributes, profiles, lineage integrations, `project_artefacts`, `semantic_layer_config` | ### Credential Types @@ -149,8 +166,8 @@ Sensitive values are never in the YAML. They're passed as Terraform variables an | Variable | Key format | Matches in YAML | |---|---|---| | `environment_credentials` | `"project_key_env_key"` | Environment `credential:` block | -| `connection_credentials` | `"connection_key"` | `global_connections[].key` | -| `token_map` | `"token_name"` | `credential.token_name` (legacy Databricks) | +| `connection_credentials` | `"connection_key"` | `globals.connections[].key` | +| `token_map` | `"token_name"` | `credential.token_name` (legacy Databricks) or `secret_*` values in `jobs[].environment_variable_overrides` | | `lineage_tokens` | `"project_key_integration_key"` | `lineage_integrations[].key` composite | | `oauth_client_secrets` | `"oauth_config_key"` | `oauth_configurations[].key` | diff --git a/docs/reference/module-environment_variable_job_overrides.md b/docs/reference/module-environment_variable_job_overrides.md index c49b7b6..a6b76f2 100644 --- a/docs/reference/module-environment_variable_job_overrides.md +++ b/docs/reference/module-environment_variable_job_overrides.md @@ -1,3 +1,5 @@ +Populated from each project’s `jobs[]` entries that set **`environment_variable_overrides`**: a map of variable name to string. The root module passes resolved job IDs and project IDs after jobs are created. See [YAML Schema — environment variable job overrides](../configuration/yaml-schema.md#environment_variable_overrides). + ## Requirements No requirements. diff --git a/docs/reference/terraform.md b/docs/reference/terraform.md index 035bab1..a337fbb 100644 --- a/docs/reference/terraform.md +++ b/docs/reference/terraform.md @@ -49,9 +49,9 @@ No resources. | [dbt\_token](#input\_dbt\_token) | dbt Cloud API token for authentication | `string` | n/a | yes | | [yaml\_file](#input\_yaml\_file) | Path to the YAML file defining dbt Cloud resources | `string` | n/a | yes | | [target\_name](#input\_target\_name) | Default target name for dbt jobs (e.g., 'prod') | `string` | `""` | no | -| [token\_map](#input\_token\_map) | Map of Databricks token names to values. Key corresponds to `credential.token_name` in YAML. | `map(string)` | `{}` | no | +| [token\_map](#input\_token\_map) | Map of token names to secret values. Keys match `credential.token_name` (legacy Databricks) or the suffix after `secret_` in `jobs[].environment_variable_overrides` values. | `map(string)` | `{}` | no | | [environment\_credentials](#input\_environment\_credentials) | Map of environment credential objects keyed by `"{project_key}_{env_key}"`. Each object must include `credential_type` and type-specific fields. Supports 14 warehouse types. | `map(any)` | `{}` | no | -| [connection\_credentials](#input\_connection\_credentials) | Map of global connection keys to OAuth/auth credential objects. Key corresponds to `global_connections[].key` in YAML. | `map(any)` | `{}` | no | +| [connection\_credentials](#input\_connection\_credentials) | Map of global connection keys to OAuth/auth credential objects. Key corresponds to `globals.connections[].key` in YAML. | `map(any)` | `{}` | no | | [lineage\_tokens](#input\_lineage\_tokens) | Map of lineage integration tokens keyed by `"{project_key}_{integration_key}"`. | `map(string)` | `{}` | no | | [oauth\_client\_secrets](#input\_oauth\_client\_secrets) | Map of OAuth configuration keys to their client secrets. Key corresponds to `oauth_configurations[].key` in YAML. | `map(string)` | `{}` | no | diff --git a/examples/README.md b/examples/README.md index 196bbcd..46bf08c 100644 --- a/examples/README.md +++ b/examples/README.md @@ -52,9 +52,9 @@ Sensitive values are never in the YAML. They're passed as Terraform variables an | Variable | Key format | Matches in YAML | |---|---|---| -| `token_map` | `"token_name"` | `credential.token_name` | +| `token_map` | `"token_name"` | `credential.token_name` (Databricks legacy) or `secret_*` values in `jobs[].environment_variable_overrides` | | `environment_credentials` | `"project_key_env_key"` | Environment `credential:` block | -| `connection_credentials` | `"connection_key"` | `global_connections[].key` | +| `connection_credentials` | `"connection_key"` | `globals.connections[].key` | | `lineage_tokens` | `"project_key_integration_key"` | `lineage_integrations[].key` composite | | `oauth_client_secrets` | `"oauth_config_key"` | `oauth_configurations[].key` | diff --git a/examples/basic/dbt-config.yml b/examples/basic/dbt-config.yml index a576932..2f5f6f2 100644 --- a/examples/basic/dbt-config.yml +++ b/examples/basic/dbt-config.yml @@ -1,72 +1,56 @@ # yaml-language-server: $schema=https://raw.githubusercontent.com/trouze/terraform-dbtcloud-yaml/refs/heads/main/schemas/v1.json ############################################################################## -# dbt Cloud Terraform YAML — Minimal Starter +# dbt Cloud Terraform YAML — Minimal Starter (version: 1) # # Fill in the placeholder values (search for YOUR_) and run: # terraform init && terraform apply -# -# For a full reference of every supported field see: -# https://github.com/trouze/terraform-dbtcloud-yaml/blob/main/docs/configuration/yaml-schema.md ############################################################################## -########################### -# Warehouse connection -# Shared across environments in this project. -# Supported types: databricks | snowflake | bigquery | postgres | redshift -########################### - -global_connections: - - name: YOUR_CONNECTION_NAME # e.g. "Databricks Production" - key: prod_connection # referenced by environments below - type: databricks # change to your warehouse type - host: YOUR_HOST # e.g. adb-1234567890.1.azuredatabricks.net - http_path: YOUR_HTTP_PATH # e.g. /sql/1.0/warehouses/abc123 - catalog: YOUR_CATALOG # e.g. main - protected: true # prevents accidental deletion +version: 1 +account: + name: YOUR_ACCOUNT_NAME + host_url: https://cloud.getdbt.com - # --- Snowflake alternative --- - # - name: YOUR_CONNECTION_NAME - # key: prod_connection - # type: snowflake - # account: xy12345.us-east-1 - # database: ANALYTICS - # warehouse: TRANSFORMING - # role: TRANSFORMER - -########################### -# Projects -########################### +globals: + connections: + - name: YOUR_CONNECTION_NAME + key: prod_connection + type: databricks + protected: true + details: + host: YOUR_HOST + http_path: YOUR_HTTP_PATH + catalog: YOUR_CATALOG projects: - - name: YOUR_PROJECT_NAME # e.g. "Analytics" - key: analytics # short identifier used in credential keys - + - name: YOUR_PROJECT_NAME + key: analytics repository: - remote_url: "YOUR_ORG/YOUR_REPO" # e.g. "acme/analytics" - github_installation_id: YOUR_GITHUB_APP_ID # GitHub App installation ID - # For GitLab, replace the two lines above with: - # remote_url: "your-org/group/repo" - # gitlab_project_id: 1234 + remote_url: "YOUR_ORG/YOUR_REPO" + github_installation_id: YOUR_GITHUB_APP_ID environments: - name: Production key: prod - connection_key: prod_connection # matches global_connections[].key - deployment_type: production type: deployment + deployment_type: production + connection: prod_connection custom_branch: main protected: true credential: credential_type: databricks - catalog: YOUR_CATALOG # e.g. main - schema: YOUR_SCHEMA # e.g. analytics - # token supplied via var.environment_credentials["analytics_prod"] + catalog: YOUR_CATALOG + schema: YOUR_SCHEMA - name: Development key: dev - connection_key: prod_connection type: development + connection: prod_connection + credential: + credential_type: databricks + catalog: YOUR_CATALOG + schema: YOUR_SCHEMA_DEV jobs: - name: Daily Build @@ -80,5 +64,5 @@ projects: git_provider_webhook: false on_merge: false schedule_type: days_of_week - schedule_days: [1, 2, 3, 4, 5] # Mon–Fri (0=Sun, 6=Sat) - schedule_hours: [6] # 6 AM UTC + schedule_days: [1, 2, 3, 4, 5] + schedule_hours: [6] diff --git a/main.tf b/main.tf index bfe5ff0..f34ca67 100644 --- a/main.tf +++ b/main.tf @@ -1,8 +1,7 @@ ############################################# # Root Module - dbt Cloud Resources # -# Orchestrates all dbt Cloud resources from a YAML file. -# Supports both single-project (project:) and multi-project (projects:) YAML. +# Orchestrates all dbt Cloud resources from a YAML file (version: 1; schemas/v1.json). # Account-scoped resources are created first, then project-scoped resources. ############################################# @@ -43,17 +42,20 @@ module "global_connections" { connections_data = try(local.yaml_content.global_connections, []) connection_credentials = var.connection_credentials + privatelink_endpoints = try(local.yaml_content.privatelink_endpoints, []) } ############################################# -# Account-Level: Service Tokens +# Account-Level: Data lookups (LOOKUP: + GitHub installations) ############################################# -module "service_tokens" { - count = length(try(local.yaml_content.service_tokens, [])) > 0 ? 1 : 0 - source = "./modules/service_tokens" +module "data_lookups" { + count = length(local._lookup_connection_ref_strings) > 0 || var.dbt_pat != null ? 1 : 0 + source = "./modules/data_lookups" - service_tokens_data = try(local.yaml_content.service_tokens, []) + projects = local.projects + dbt_pat = var.dbt_pat + dbt_host_url = local.dbt_host_url_effective } ############################################# @@ -64,7 +66,8 @@ module "groups" { count = length(try(local.yaml_content.groups, [])) > 0 ? 1 : 0 source = "./modules/groups" - groups_data = try(local.yaml_content.groups, []) + groups_data = try(local.yaml_content.groups, []) + skip_global_project_permissions = var.skip_global_project_permissions } ############################################# @@ -124,6 +127,20 @@ module "project" { target_name = var.target_name } +############################################# +# Account-Level: Service Tokens +# Declared after project so permissions[].project_key resolves via project_ids. +############################################# + +module "service_tokens" { + count = length(try(local.yaml_content.service_tokens, [])) > 0 ? 1 : 0 + source = "./modules/service_tokens" + + service_tokens_data = try(local.yaml_content.service_tokens, []) + project_ids = module.project.project_ids + skip_global_project_permissions = var.skip_global_project_permissions +} + ############################################# # Repository Configuration ############################################# @@ -134,17 +151,21 @@ module "repository" { dbtcloud = dbtcloud.pat_provider } - projects = local.projects - project_ids = module.project.project_ids - dbt_pat = var.dbt_pat - enable_gitlab_deploy_token = var.enable_gitlab_deploy_token + projects = local.projects + project_ids = module.project.project_ids + dbt_pat = var.dbt_pat + enable_gitlab_deploy_token = var.enable_gitlab_deploy_token + github_installation_by_owner = length(module.data_lookups) > 0 ? module.data_lookups[0].github_installation_by_owner : {} + github_installation_fallback_id = length(module.data_lookups) > 0 ? module.data_lookups[0].github_installation_fallback_id : null + privatelink_endpoints = try(local.yaml_content.privatelink_endpoints, []) } module "project_repository" { source = "./modules/project_repository" - project_ids = module.project.project_ids - repository_ids = module.repository.repository_ids + project_ids = module.project.project_ids + repository_ids = module.repository.repository_ids + protected_repository_keys = module.repository.protected_repository_keys } ############################################# @@ -172,6 +193,23 @@ module "credentials" { environment_credentials = var.environment_credentials } +############################################# +# Profiles (before environments when environments use primary_profile_key) +############################################# + +module "profiles" { + count = length(flatten([for p in local.projects : try(p.profiles, [])])) > 0 ? 1 : 0 + source = "./modules/profiles" + + projects = local.projects + project_ids = module.project.project_ids + global_connection_ids = local.global_connection_ids_effective + credential_ids = module.credentials.credential_ids + credential_ids_by_source_id = module.credentials.credential_ids_by_source_id + extended_attribute_ids = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids : {} + extended_attribute_ids_by_source_id = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids_by_source_id : {} +} + ############################################# # Environments ############################################# @@ -179,11 +217,13 @@ module "credentials" { module "environments" { source = "./modules/environments" - projects = local.projects - project_ids = module.project.project_ids - credential_ids = module.credentials.credential_ids - global_connection_ids = length(try(local.yaml_content.global_connections, [])) > 0 ? module.global_connections[0].connection_ids : {} - extended_attribute_ids = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids : {} + projects = local.projects + project_ids = module.project.project_ids + credential_ids = module.credentials.credential_ids + global_connection_ids = local.global_connection_ids_effective + extended_attribute_ids = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids : {} + extended_attribute_ids_by_source_id = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids_by_source_id : {} + profile_ids = length(flatten([for p in local.projects : try(p.profiles, [])])) > 0 ? module.profiles[0].profile_ids : {} } ############################################# @@ -193,9 +233,12 @@ module "environments" { module "jobs" { source = "./modules/jobs" - projects = local.projects - project_ids = module.project.project_ids - environment_ids = module.environments.environment_ids + projects = local.projects + project_ids = module.project.project_ids + environment_ids = module.environments.environment_ids + deployment_types = module.environments.deployment_types + + depends_on = [module.environments, module.environment_variables] } ############################################# @@ -222,23 +265,9 @@ module "environment_variable_job_overrides" { projects = local.projects project_ids = module.project.project_ids job_ids = module.jobs.job_ids + token_map = var.token_map - depends_on = [module.environment_variables] -} - -############################################# -# Profiles (links connection + credential + extended_attributes) -############################################# - -module "profiles" { - count = length(flatten([for p in local.projects : try(p.profiles, [])])) > 0 ? 1 : 0 - source = "./modules/profiles" - - projects = local.projects - project_ids = module.project.project_ids - global_connection_ids = length(try(local.yaml_content.global_connections, [])) > 0 ? module.global_connections[0].connection_ids : {} - credential_ids = module.credentials.credential_ids - extended_attribute_ids = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids : {} + depends_on = [module.environment_variables, module.jobs] } ############################################# @@ -259,7 +288,7 @@ module "lineage_integrations" { ############################################# module "project_artefacts" { - count = length([for p in local.projects : p if try(p.artefacts, null) != null]) > 0 ? 1 : 0 + count = length([for p in local.projects : p if try(p.project_artefacts, null) != null]) > 0 ? 1 : 0 source = "./modules/project_artefacts" projects = local.projects @@ -274,10 +303,15 @@ module "project_artefacts" { ############################################# module "semantic_layer" { - count = length([for p in local.projects : p if try(p.semantic_layer, null) != null]) > 0 ? 1 : 0 + count = length([ + for p in local.projects : p + if try(p.semantic_layer_config, null) != null + ]) > 0 ? 1 : 0 source = "./modules/semantic_layer" projects = local.projects project_ids = module.project.project_ids environment_ids = module.environments.environment_ids + + depends_on = [module.environments] } diff --git a/modules/account_features/main.tf b/modules/account_features/main.tf index 5315db1..c08bd0f 100644 --- a/modules/account_features/main.tf +++ b/modules/account_features/main.tf @@ -8,6 +8,13 @@ terraform { } } +############################################# +# Account Features (singleton) +# Boolean feature flags for the entire account. +# Private API — may not be available on all deployments. +# No import or delete support. +############################################# + resource "dbtcloud_account_features" "features" { count = var.features != null ? 1 : 0 diff --git a/modules/credentials/main.tf b/modules/credentials/main.tf index 78bb992..3aa700f 100644 --- a/modules/credentials/main.tf +++ b/modules/credentials/main.tf @@ -9,8 +9,8 @@ terraform { } locals { - # Flatten all environments across all projects that have credentials defined - all_credential_owners = flatten([ + # Environments that declare an inline credential block (warehouse config in YAML). + env_credential_owners_list = flatten([ for p in var.projects : [ for env in try(p.environments, []) : { project_key = try(p.key, p.name) @@ -23,12 +23,45 @@ locals { ] ]) - credential_owners_map = { - for item in local.all_credential_owners : + # Source credential IDs from YAML (migration / import) — used to exclude duplicate profile-owned credentials. + env_source_credential_ids = toset([ + for item in local.env_credential_owners_list : + tostring(item.cred_data.id) + if try(item.cred_data.id, null) != null + ]) + + credential_owners_from_environments = { + for item in local.env_credential_owners_list : + item.composite_key => item + } + + # COMPAT(v1-schema): v2/importer YAML may set profiles[].credential + profiles[].credentials_id to mint a + # Terraform-managed credential keyed by project_profile (secrets still via environment_credentials map). + standalone_profile_credential_owners = { + for item in flatten([ + for p in var.projects : [ + for profile in try(p.profiles, []) : { + project_key = try(p.key, p.name) + project_id = var.project_ids[try(p.key, p.name)] + composite_key = "${try(p.key, p.name)}_${try(profile.key, profile.name)}" + cred_data = try(profile.credential, null) + } + if try(profile.credential.credential_type, null) != null && + try(profile.credentials_id, null) != null && + !contains(local.env_source_credential_ids, tostring(profile.credentials_id)) + ] + ]) : item.composite_key => item } + all_credential_owners_map = merge( + local.credential_owners_from_environments, + local.standalone_profile_credential_owners + ) + # Non-sensitive helpers for for_each conditions + available_token_names = toset(nonsensitive(keys(var.token_map))) + available_env_cred_keys = toset(nonsensitive(keys(var.environment_credentials))) env_cred_types = nonsensitive({ @@ -41,6 +74,38 @@ locals { for k, v in var.environment_credentials : k => try(v.tenant_id, null) != null }) + + merged_credential_ids = merge( + { for k, c in dbtcloud_databricks_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_snowflake_credential.credentials_password : k => c.credential_id }, + { for k, c in dbtcloud_snowflake_credential.credentials_keypair : k => c.credential_id }, + { for k, c in dbtcloud_bigquery_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_postgres_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_redshift_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_athena_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_fabric_credential.credentials_sql : k => c.credential_id }, + { for k, c in dbtcloud_fabric_credential.credentials_sp : k => c.credential_id }, + { for k, c in dbtcloud_synapse_credential.credentials_sql : k => c.credential_id }, + { for k, c in dbtcloud_synapse_credential.credentials_sp : k => c.credential_id }, + { for k, c in dbtcloud_starburst_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_spark_credential.credentials : k => c.credential_id }, + { for k, c in dbtcloud_teradata_credential.credentials : k => c.credential_id }, + ) + + # Map legacy YAML credential.id (dbt Cloud) -> Terraform-managed credential_id after apply + # (environments and standalone profile credentials — COMPAT v2/importer). + credential_ids_by_source_id = merge( + { + for item in local.env_credential_owners_list : + tostring(item.cred_data.id) => local.merged_credential_ids[item.composite_key] + if try(item.cred_data.id, null) != null && try(local.merged_credential_ids[item.composite_key], null) != null + }, + { + for k, item in local.standalone_profile_credential_owners : + tostring(item.cred_data.id) => local.merged_credential_ids[k] + if try(item.cred_data.id, null) != null && try(local.merged_credential_ids[k], null) != null + } + ) } ############################################# @@ -49,14 +114,17 @@ locals { resource "dbtcloud_databricks_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if( contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "databricks" ) || ( !contains(local.available_env_cred_keys, k) && - try(item.cred_data.credential_type, try(item.cred_data.type, "databricks")) == "databricks" + try(item.cred_data.credential_type, try(item.cred_data.type, "databricks")) == "databricks" && + try(item.cred_data.token_name, null) != null && + contains(local.available_token_names, item.cred_data.token_name) && + try(item.cred_data.schema, null) != null ) } @@ -73,7 +141,7 @@ resource "dbtcloud_databricks_credential" "credentials" { resource "dbtcloud_snowflake_credential" "credentials_password" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "snowflake" && @@ -97,7 +165,7 @@ resource "dbtcloud_snowflake_credential" "credentials_password" { resource "dbtcloud_snowflake_credential" "credentials_keypair" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "snowflake" && @@ -122,7 +190,7 @@ resource "dbtcloud_snowflake_credential" "credentials_keypair" { resource "dbtcloud_bigquery_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "bigquery" @@ -139,7 +207,7 @@ resource "dbtcloud_bigquery_credential" "credentials" { resource "dbtcloud_postgres_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "postgres" @@ -160,7 +228,7 @@ resource "dbtcloud_postgres_credential" "credentials" { resource "dbtcloud_redshift_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "redshift" @@ -179,7 +247,7 @@ resource "dbtcloud_redshift_credential" "credentials" { resource "dbtcloud_athena_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "athena" @@ -197,7 +265,7 @@ resource "dbtcloud_athena_credential" "credentials" { resource "dbtcloud_fabric_credential" "credentials_sql" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "fabric" && @@ -218,7 +286,7 @@ resource "dbtcloud_fabric_credential" "credentials_sql" { resource "dbtcloud_fabric_credential" "credentials_sp" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "fabric" && @@ -240,7 +308,7 @@ resource "dbtcloud_fabric_credential" "credentials_sp" { resource "dbtcloud_synapse_credential" "credentials_sql" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "synapse" && @@ -262,7 +330,7 @@ resource "dbtcloud_synapse_credential" "credentials_sql" { resource "dbtcloud_synapse_credential" "credentials_sp" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "synapse" && @@ -285,7 +353,7 @@ resource "dbtcloud_synapse_credential" "credentials_sp" { resource "dbtcloud_starburst_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && contains(["starburst", "trino"], try(local.env_cred_types[k], "")) @@ -304,7 +372,7 @@ resource "dbtcloud_starburst_credential" "credentials" { resource "dbtcloud_spark_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && contains(["spark", "apache_spark"], try(local.env_cred_types[k], "")) @@ -321,7 +389,7 @@ resource "dbtcloud_spark_credential" "credentials" { resource "dbtcloud_teradata_credential" "credentials" { for_each = { - for k, item in local.credential_owners_map : + for k, item in local.all_credential_owners_map : k => item if contains(local.available_env_cred_keys, k) && try(local.env_cred_types[k], "") == "teradata" diff --git a/modules/credentials/outputs.tf b/modules/credentials/outputs.tf index 3a89d05..080ebf8 100644 --- a/modules/credentials/outputs.tf +++ b/modules/credentials/outputs.tf @@ -1,19 +1,9 @@ output "credential_ids" { - description = "Map of composite key (project_key_env_key) to credential ID. Merges all warehouse types." - value = merge( - { for k, c in dbtcloud_databricks_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_snowflake_credential.credentials_password : k => c.credential_id }, - { for k, c in dbtcloud_snowflake_credential.credentials_keypair : k => c.credential_id }, - { for k, c in dbtcloud_bigquery_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_postgres_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_redshift_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_athena_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_fabric_credential.credentials_sql : k => c.credential_id }, - { for k, c in dbtcloud_fabric_credential.credentials_sp : k => c.credential_id }, - { for k, c in dbtcloud_synapse_credential.credentials_sql : k => c.credential_id }, - { for k, c in dbtcloud_synapse_credential.credentials_sp : k => c.credential_id }, - { for k, c in dbtcloud_starburst_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_spark_credential.credentials : k => c.credential_id }, - { for k, c in dbtcloud_teradata_credential.credentials : k => c.credential_id }, - ) + description = "Map of composite key (project_key_env_key or project_key_profile_key) to credential ID. Merges all warehouse types." + value = local.merged_credential_ids +} + +output "credential_ids_by_source_id" { + description = "Maps YAML credential.id (environment or standalone profile credentials, legacy dbt Cloud ID) to Terraform-managed credential_id after apply (COMPAT v2/importer)." + value = local.credential_ids_by_source_id } diff --git a/modules/data_lookups/main.tf b/modules/data_lookups/main.tf new file mode 100644 index 0000000..902c8ba --- /dev/null +++ b/modules/data_lookups/main.tf @@ -0,0 +1,104 @@ +terraform { + required_version = ">= 1.7" + required_providers { + dbtcloud = { + source = "dbt-labs/dbtcloud" + version = "~> 1.8" + } + http = { + source = "hashicorp/http" + version = "~> 3.0" + } + } +} + +############################################# +# LOOKUP: global connection resolution +# Matches account global connections by name (suffix after LOOKUP:). +############################################# + +locals { + lookup_connection_keys = toset([ + for conn_ref in flatten([ + for p in var.projects : concat( + [ + for env in try(p.environments, []) : + try(env.connection, null) + if try(env.connection, null) != null && startswith(tostring(env.connection), "LOOKUP:") + ], + [ + for prof in try(p.profiles, []) : + try(prof.connection_key, null) + if try(prof.connection_key, null) != null && startswith(tostring(prof.connection_key), "LOOKUP:") + ] + ) + ]) : + tostring(conn_ref) if startswith(tostring(conn_ref), "LOOKUP:") + ]) + + needs_global_connections_data = length(local.lookup_connection_keys) > 0 + + # Base URL for dbt Cloud Admin API (integrations); strip accidental /api suffix. + dbt_host_url_raw = coalesce(var.dbt_host_url, "https://cloud.getdbt.com") + dbt_host_url = replace(local.dbt_host_url_raw, "/api", "") + + # Optional: repository field as a scalar LOOKUP (v2 / importer shape); object repos are ignored. + lookup_repository_keys = toset([ + for repo_ref in [ + for p in var.projects : + p.repository + if can(regex("^LOOKUP:", try(tostring(p.repository), ""))) + ] : try(tostring(repo_ref), "") if startswith(try(tostring(repo_ref), ""), "LOOKUP:") + ]) +} + +data "dbtcloud_global_connections" "all" { + count = local.needs_global_connections_data ? 1 : 0 +} + +locals { + lookup_connection_ids = { + for lookup_key in local.lookup_connection_keys : + lookup_key => try( + tostring([ + for conn in data.dbtcloud_global_connections.all[0].connections : + conn.id if try(conn.name, null) == replace(lookup_key, "LOOKUP:", "") + ][0]), + null + ) + } +} + +############################################# +# GitHub App installations (account integrations API) +# Requires PAT — not available on service tokens. +############################################# + +data "http" "github_installations" { + count = var.dbt_pat != null ? 1 : 0 + + url = format("%s/api/v2/integrations/github/installations/", local.dbt_host_url) + request_headers = { + Authorization = format("Bearer %s", var.dbt_pat) + } +} + +locals { + github_installations_raw = length(data.http.github_installations) > 0 ? try( + tolist(jsondecode(data.http.github_installations[0].response_body)), + [] + ) : [] + + github_installations = [ + for inst in local.github_installations_raw : + inst if can(regex("github", try(inst.access_tokens_url, ""))) + ] + + github_installation_by_owner = { + for inst in local.github_installations : + lower(try(inst.account.login, "")) => inst.id + if try(inst.account.login, "") != "" + } + + github_installation_fallback_id = length(local.github_installations) > 0 ? local.github_installations[0].id : null +} diff --git a/modules/data_lookups/outputs.tf b/modules/data_lookups/outputs.tf new file mode 100644 index 0000000..1a42a16 --- /dev/null +++ b/modules/data_lookups/outputs.tf @@ -0,0 +1,24 @@ +output "lookup_connection_ids" { + description = "Map from literal YAML placeholder (e.g. LOOKUP:My Warehouse) to dbt global connection id from data.dbtcloud_global_connections" + value = local.lookup_connection_ids +} + +output "lookup_connection_keys" { + description = "Set of LOOKUP:… placeholders found under environments.connection and profiles.connection_key" + value = local.lookup_connection_keys +} + +output "lookup_repository_keys" { + description = "Set of LOOKUP:… values when project.repository is a scalar (importer/v2 style); object-shaped repositories are not included" + value = local.lookup_repository_keys +} + +output "github_installation_by_owner" { + description = "Lowercase GitHub org/user login → installation id (empty if dbt_pat unset or API error)" + value = local.github_installation_by_owner +} + +output "github_installation_fallback_id" { + description = "First GitHub installation id when owner cannot be matched (null if none)" + value = local.github_installation_fallback_id +} diff --git a/modules/data_lookups/variables.tf b/modules/data_lookups/variables.tf new file mode 100644 index 0000000..d3f4fe9 --- /dev/null +++ b/modules/data_lookups/variables.tf @@ -0,0 +1,17 @@ +variable "projects" { + description = "Project list from YAML (same shape as root local.projects) — scanned for LOOKUP: connection and repository placeholders" + type = any +} + +variable "dbt_pat" { + description = "Personal access token for GitHub installations API (optional)" + type = string + sensitive = true + default = null +} + +variable "dbt_host_url" { + description = "dbt Cloud host URL (e.g. https://cloud.getdbt.com); used for integrations HTTP calls" + type = string + default = null +} diff --git a/modules/environment_variable_job_overrides/.terraform.lock.hcl b/modules/environment_variable_job_overrides/.terraform.lock.hcl new file mode 100644 index 0000000..8dc8cba --- /dev/null +++ b/modules/environment_variable_job_overrides/.terraform.lock.hcl @@ -0,0 +1,27 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/dbt-labs/dbtcloud" { + version = "1.8.2" + constraints = "~> 1.8" + hashes = [ + "h1:3Hw1in/qcHdAp2BRhEaFRr/ez9lFjBv3V5WIUxHXcSs=", + "h1:ASwPE1I3U7cufOLFvGV+yHmV4/24y2Jrmt8V3Pn9lEY=", + "h1:BIVjUC8FxluKSvwm2jvySD0AvSkFPKwGn/SHez082Oc=", + "h1:IpC3g4srpeI3iJ5GL3GcxSMO/+XHBktS/zumEm/8xbc=", + "zh:01df4a2592c7d66152aa18bae2fb6abcca205f2c0d75486a67e3c8307ea70af2", + "zh:020e8c80eb8973ac0571b8cb2c990d76c82d2032f590ffaec418fc6728a48594", + "zh:0c30dd1d7ac516efe921362143c0372b2dedab59950662c08513c8c1e0430ee4", + "zh:192bc343bfe193d79b4f2e5d5340a44230e169670d190c7034c7db67fd97b3e7", + "zh:24b8175f22a97844000facea8742bd2f0ca5eedb214c900e6b50153308874e86", + "zh:26ef5f40358401f7fadae72b16782d0a8a8761af325aac17b5507234148c19f9", + "zh:7b9cd9637bf607854dfdeceab06dd361686f54864050cfb770d4214a2d0265db", + "zh:8df586c709b83ca62056fd3d9ff5e9d73833acdcb3194d7e5bbc0cde98f87ed9", + "zh:8e561719c50ae83caefa0dd8ebf3cf4907a512d1be90ecb80eb5db6593f9e9b2", + "zh:9071202c4b459dd28fa792a62e229815bae41d637428fa66dca5eb7fa0fc5801", + "zh:926cca7812b272ab976ce49c0858de78bd5c4c6fcffe16cb2d33cdce63522bc2", + "zh:ae927fc0b059c800b0342b6c6f7217dd191349334a6d6ed1cad578f13f64722c", + "zh:b0e484d5c2dc79b1f573aa3bdac4150a2b4094b6fd9c92033b855e7b8f07045c", + "zh:f96b0ce96fe679653e5dd3820f15e7916855ce8deb832f6b92befedc2387f1f6", + ] +} diff --git a/modules/environment_variable_job_overrides/main.tf b/modules/environment_variable_job_overrides/main.tf index 137fced..9889410 100644 --- a/modules/environment_variable_job_overrides/main.tf +++ b/modules/environment_variable_job_overrides/main.tf @@ -9,29 +9,10 @@ terraform { } locals { - # Flatten env var job overrides from environment-nested jobs (legacy layout) - overrides_from_env_jobs = flatten([ - for p in var.projects : [ - for env in try(p.environments, []) : [ - for job in try(env.jobs, []) : [ - for var_name, var_value in try(job.env_var_overrides, {}) : { - project_key = try(p.key, p.name) - project_id = var.project_ids[try(p.key, p.name)] - job_key = "${try(p.key, p.name)}_${env.name}_${job.name}" - var_name = var_name - var_value = var_value - composite_key = "${try(p.key, p.name)}_${env.name}_${job.name}_${var_name}" - } - ] - ] if try(env.jobs, null) != null - ] - ]) - - # Flatten env var job overrides from project-level jobs (new layout) overrides_from_project_jobs = flatten([ for p in var.projects : [ for job in try(p.jobs, []) : [ - for var_name, var_value in try(job.env_var_overrides, {}) : { + for var_name, var_value in try(job.environment_variable_overrides, {}) : { project_key = try(p.key, p.name) project_id = var.project_ids[try(p.key, p.name)] job_key = "${try(p.key, p.name)}_${try(job.key, job.name)}" @@ -44,7 +25,7 @@ locals { ]) all_overrides_map = { - for item in concat(local.overrides_from_env_jobs, local.overrides_from_project_jobs) : + for item in local.overrides_from_project_jobs : item.composite_key => item } } @@ -55,5 +36,19 @@ resource "dbtcloud_environment_variable_job_override" "environment_variable_job_ name = each.value.var_name project_id = each.value.project_id job_definition_id = lookup(var.job_ids, each.value.job_key, null) - raw_value = tostring(each.value.var_value) + raw_value = ( + startswith(tostring(each.value.var_value), "secret_") ? + lookup(var.token_map, trimprefix(tostring(each.value.var_value), "secret_"), tostring(each.value.var_value)) : + tostring(each.value.var_value) + ) + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_environment_variable_job_override (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = null + # source_identity = "VAR_JOB_OVERRIDE:${each.value.project_key}:${each.value.job_key}:${each.value.var_name}" + # source_key = each.value.var_name + # source_project_key = each.value.project_key + # source_name = each.value.var_name + # } } diff --git a/modules/environment_variable_job_overrides/tests/unit.tftest.hcl b/modules/environment_variable_job_overrides/tests/unit.tftest.hcl new file mode 100644 index 0000000..145b065 --- /dev/null +++ b/modules/environment_variable_job_overrides/tests/unit.tftest.hcl @@ -0,0 +1,72 @@ +# Unit tests for modules/environment_variable_job_overrides +# Run from modules/environment_variable_job_overrides/: terraform test + +mock_provider "dbtcloud" {} + +run "v2_environment_variable_overrides_on_project_job" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + jobs = [ + { + name = "Daily" + key = "daily" + environment_key = "dev" + environment_variable_overrides = { + DBT_OVERRIDE = "v2-value" + } + } + ] + } + ] + project_ids = { analytics = "1001" } + job_ids = { + "analytics_daily" = "5001" + } + token_map = {} + } + + assert { + condition = dbtcloud_environment_variable_job_override.environment_variable_job_overrides["analytics_daily_DBT_OVERRIDE"].raw_value == "v2-value" + error_message = "environment_variable_overrides should create job override with expected raw_value" + } +} + +run "secret_prefix_uses_token_map" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + jobs = [ + { + name = "Daily" + key = "daily" + environment_key = "dev" + environment_variable_overrides = { + DBT_SECRET = "secret_my_job_token" + } + } + ] + } + ] + project_ids = { analytics = "1001" } + job_ids = { + "analytics_daily" = "5001" + } + token_map = { + "my_job_token" = "resolved" + } + } + + assert { + condition = dbtcloud_environment_variable_job_override.environment_variable_job_overrides["analytics_daily_DBT_SECRET"].raw_value == "resolved" + error_message = "secret_ prefix should strip and resolve token_map (including underscores in key)" + } +} diff --git a/modules/environment_variable_job_overrides/variables.tf b/modules/environment_variable_job_overrides/variables.tf index abbd408..b0357d3 100644 --- a/modules/environment_variable_job_overrides/variables.tf +++ b/modules/environment_variable_job_overrides/variables.tf @@ -1,5 +1,5 @@ variable "projects" { - description = "List of project configurations. Env var job overrides are read from project.environments[].jobs[].env_var_overrides or project.jobs[].env_var_overrides." + description = "List of project configurations. Job-level environment_variable_overrides on project.jobs[] (see modules/environment_variable_job_overrides)." type = any } @@ -12,3 +12,9 @@ variable "job_ids" { description = "Map of composite key (project_key_job_key) to dbt Cloud job ID (from jobs module)" type = map(string) } + +variable "token_map" { + description = "Secret values for override values prefixed with secret_ (same semantics as modules/environment_variables)." + type = map(string) + default = {} +} diff --git a/modules/environment_variables/.terraform.lock.hcl b/modules/environment_variables/.terraform.lock.hcl new file mode 100644 index 0000000..0c06cc0 --- /dev/null +++ b/modules/environment_variables/.terraform.lock.hcl @@ -0,0 +1,24 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/dbt-labs/dbtcloud" { + version = "1.9.1" + constraints = "~> 1.8" + hashes = [ + "h1:OygRDw8+b5uCeREVO0TJVs71YjB+QOomqfO+zFUSQwg=", + "zh:01b731b2787f7fbbda8b94100059fe9bbfa613948e017712f433784056db9ae2", + "zh:134a842f52383ab3818e9bc75a285947d356248af5a5b38bbb1fca7c8b6e92ec", + "zh:492fde3092595af2af15ca05cf8718b99f5d722d175de8e7dc4fb81d1e1ff42b", + "zh:53dd23b590eda76f704c0f8acd3d1ebc03b42682c25ef8dc73e5366eaa141ff1", + "zh:578ecaa58259bacda28cec78521f7179e93ac07187c11c4cf2fc00a5907faf30", + "zh:5d2a68740b7b02d21a44cb06554afc8f1b458ecfe6b22f3485f1ba62d2e0ba98", + "zh:6f377f5debf908bb01e837bcc83bc5dc38ef23f1c4afb0922df8135e45e34ada", + "zh:7631851aea4cc39ac1625bf70dfd8bffcd1b9494dc9d4d727fd07a2704e2f27f", + "zh:78da15555a98178b07afce709f4386c5ed8ce75e7cbe9bd13434d96e301ab6fd", + "zh:8224a5d6ef02f888c7acf828f617ee2716984d86ea54e9f4f39ed10db777f55f", + "zh:c137fff71396e3a78910fcd83876e941fa420f2a9f3d9e6f960704bec0226a9c", + "zh:c5ad536754aa3488a354772c585d293d25c29eaab8ca634e8b6e5980b99ed2c4", + "zh:e5e897540808d918107f85caeb27baea55bb51b40cc9cb5aa55f70d8ad242241", + "zh:fd1f648cc0a3afdef1b34ff4a691022e5320f4af004c31fcdad4fd8bd13391b6", + ] +} diff --git a/modules/environment_variables/main.tf b/modules/environment_variables/main.tf index 1d75eec..82ba24c 100644 --- a/modules/environment_variables/main.tf +++ b/modules/environment_variables/main.tf @@ -9,7 +9,35 @@ terraform { } locals { - # Flatten all env vars across all projects + # Mirror environments module: same project_key, env_key, composite_key, and display name. + all_environments = flatten([ + for p in var.projects : [ + for env in try(p.environments, []) : { + project_key = try(p.key, p.name) + env_key = try(env.key, env.name) + composite_key = "${try(p.key, p.name)}_${try(env.key, env.name)}" + env_data = env + } + ] + ]) + + # Resolve environment_values[].env (name or YAML key) -> composite_key, then -> dbt Cloud environment name (env_data.name). + env_name_to_resource_key = merge( + { + for item in local.all_environments : + "${item.project_key}_${item.env_data.name}" => item.composite_key + }, + { + for item in local.all_environments : + "${item.project_key}_${item.env_key}" => item.composite_key + } + ) + + env_display_name_by_composite_key = { + for item in local.all_environments : + item.composite_key => item.env_data.name + } + all_env_vars = flatten([ for p in var.projects : [ for ev in try(p.environment_variables, []) : { @@ -22,8 +50,25 @@ locals { ] ]) - env_vars_map = { + protected_env_var_items = [ + for item in local.all_env_vars : + item + if try(item.ev_data.protected, false) == true + ] + + unprotected_env_var_items = [ for item in local.all_env_vars : + item + if try(item.ev_data.protected, false) != true + ] + + env_vars_map = { + for item in local.unprotected_env_var_items : + item.composite_key => item + } + + protected_env_vars_map = { + for item in local.protected_env_var_items : item.composite_key => item } } @@ -31,14 +76,67 @@ locals { resource "dbtcloud_environment_variable" "environment_variables" { for_each = local.env_vars_map - name = each.value.var_name + name = each.value.ev_data.name + project_id = each.value.project_id + + environment_values = { + for item in each.value.ev_data.environment_values : + ( + item.env == "project" ? "project" : ( + lookup(local.env_name_to_resource_key, "${each.value.project_key}_${item.env}", null) != null ? + local.env_display_name_by_composite_key[local.env_name_to_resource_key["${each.value.project_key}_${item.env}"]] : + item.env + ) + ) => ( + startswith(tostring(item.value), "secret_") ? + lookup(var.token_map, trimprefix(tostring(item.value), "secret_"), tostring(item.value)) : + tostring(item.value) + ) + } + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_environment_variable (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.ev_data.id, null) + # source_identity = "VAR:${each.value.project_key}:${each.value.var_name}" + # source_key = each.value.var_name + # source_project_key = each.value.project_key + # source_name = each.value.ev_data.name + # } +} + +resource "dbtcloud_environment_variable" "protected_environment_variables" { + for_each = local.protected_env_vars_map + + name = each.value.ev_data.name project_id = each.value.project_id + environment_values = { for item in each.value.ev_data.environment_values : - item.env => ( - startswith(tostring(item.value), "secret_") - ? lookup(var.token_map, join("_", slice(split("_", tostring(item.value)), 1, length(split("_", tostring(item.value))))), null) - : tostring(item.value) + ( + item.env == "project" ? "project" : ( + lookup(local.env_name_to_resource_key, "${each.value.project_key}_${item.env}", null) != null ? + local.env_display_name_by_composite_key[local.env_name_to_resource_key["${each.value.project_key}_${item.env}"]] : + item.env + ) + ) => ( + startswith(tostring(item.value), "secret_") ? + lookup(var.token_map, trimprefix(tostring(item.value), "secret_"), tostring(item.value)) : + tostring(item.value) ) } + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_environment_variable (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.ev_data.id, null) + # source_identity = "VAR:${each.value.project_key}:${each.value.var_name}" + # source_key = each.value.var_name + # source_project_key = each.value.project_key + # source_name = each.value.ev_data.name + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/environment_variables/tests/unit.tftest.hcl b/modules/environment_variables/tests/unit.tftest.hcl new file mode 100644 index 0000000..2f00a2f --- /dev/null +++ b/modules/environment_variables/tests/unit.tftest.hcl @@ -0,0 +1,113 @@ +# Unit tests for modules/environment_variables +# Run from modules/environment_variables/: terraform test + +mock_provider "dbtcloud" {} + +run "resolves_environment_key_to_display_name" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + environments = [ + { + name = "Production Display" + key = "prod" + type = "deployment" + } + ] + environment_variables = [ + { + name = "DBT_MY_VAR" + environment_values = [ + { env = "prod", value = "from-key" } + ] + } + ] + } + ] + project_ids = { analytics = "1001" } + token_map = {} + } + + assert { + condition = dbtcloud_environment_variable.environment_variables["analytics_DBT_MY_VAR"].environment_values["Production Display"] == "from-key" + error_message = "env YAML key should resolve to environment display name for API map keys" + } +} + +run "protected_and_unprotected_split" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + environments = [ + { + name = "Dev" + key = "dev" + type = "development" + } + ] + environment_variables = [ + { + name = "DBT_A" + environment_values = [{ env = "project", value = "a" }] + protected = false + }, + { + name = "DBT_B" + environment_values = [{ env = "project", value = "b" }] + protected = true + } + ] + } + ] + project_ids = { analytics = "1001" } + token_map = {} + } + + assert { + condition = length(dbtcloud_environment_variable.environment_variables) == 1 + error_message = "Expected one unprotected environment variable" + } + + assert { + condition = length(dbtcloud_environment_variable.protected_environment_variables) == 1 + error_message = "Expected one protected environment variable" + } +} + +run "secret_prefix_uses_token_map" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + environment_variables = [ + { + name = "DBT_SECRET_VAR" + environment_values = [ + { env = "project", value = "secret_my_token_name" } + ] + } + ] + } + ] + project_ids = { analytics = "1001" } + token_map = { + "my_token_name" = "resolved-secret" + } + } + + assert { + condition = dbtcloud_environment_variable.environment_variables["analytics_DBT_SECRET_VAR"].environment_values["project"] == "resolved-secret" + error_message = "secret_ prefix should strip and look up token_map (including keys with underscores)" + } +} diff --git a/modules/environments/main.tf b/modules/environments/main.tf index e05af13..150cba0 100644 --- a/modules/environments/main.tf +++ b/modules/environments/main.tf @@ -27,6 +27,11 @@ locals { item.composite_key => item } + env_connection_ref = { + for k, item in local.envs_map : + k => try(item.env_data.connection, null) + } + protected_envs_map = { for k, item in local.envs_map : k => item @@ -45,27 +50,35 @@ locals { k => lookup(var.credential_ids, k, null) } - # Resolve connection_id: prefer connection key lookup (global connections), - # fall back to direct numeric ID from YAML + # Resolve connection_id: global connection key, LOOKUP:… placeholder, or numeric id. resolve_connection_id = { for k, item in local.envs_map : k => ( - try(item.env_data.connection, null) != null ? - lookup(var.global_connection_ids, tostring(item.env_data.connection), null) != null ? - lookup(var.global_connection_ids, tostring(item.env_data.connection), null) : - try(tonumber(item.env_data.connection), null) : - try(item.env_data.connection_id, null) + local.env_connection_ref[k] != null ? + lookup(var.global_connection_ids, tostring(local.env_connection_ref[k]), null) != null ? + lookup(var.global_connection_ids, tostring(local.env_connection_ref[k]), null) : + try(tonumber(local.env_connection_ref[k]), null) : + null ) } - # Resolve extended_attributes_id via key lookup + # extended_attributes_key lookup, source id remap, or raw extended_attributes_id resolve_extended_attributes_id = { for k, item in local.envs_map : - k => ( - try(item.env_data.extended_attributes_key, null) != null ? - lookup(var.extended_attribute_ids, "${item.project_key}_${item.env_data.extended_attributes_key}", null) : - null - ) + k => try(coalesce( + try(item.env_data.extended_attributes_key, null) != null && try(item.env_data.extended_attributes_key, "") != "" ? + lookup(var.extended_attribute_ids, "${item.project_key}_${item.env_data.extended_attributes_key}", null) : null, + try(item.env_data.extended_attributes_id, null) != null ? + lookup(var.extended_attribute_ids_by_source_id, tostring(item.env_data.extended_attributes_id), null) : null, + try(tonumber(item.env_data.extended_attributes_id), null) + ), null) + } + + # v2/importer: when primary_profile_key is set, connection/credential/extended_attributes on the environment are omitted (profile supplies them). + resolve_primary_profile_id = { + for k, item in local.envs_map : + k => try(item.env_data.primary_profile_key, null) != null && try(item.env_data.primary_profile_key, "") != "" ? + lookup(var.profile_ids, "${item.project_key}_${item.env_data.primary_profile_key}", null) : null } } @@ -79,15 +92,26 @@ resource "dbtcloud_environment" "environments" { project_id = each.value.project_id name = each.value.env_data.name type = each.value.env_data.type - connection_id = local.resolve_connection_id[each.key] - credential_id = local.resolve_credential_id[each.key] + connection_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_connection_id[each.key] + credential_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_credential_id[each.key] dbt_version = try(each.value.env_data.dbt_version, null) enable_model_query_history = try(each.value.env_data.enable_model_query_history, null) custom_branch = try(each.value.env_data.custom_branch, null) deployment_type = try(each.value.env_data.deployment_type, null) use_custom_branch = try(each.value.env_data.custom_branch, null) != null - extended_attributes_id = local.resolve_extended_attributes_id[each.key] + primary_profile_id = local.resolve_primary_profile_id[each.key] + extended_attributes_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_extended_attributes_id[each.key] + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_environment (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.env_data.id, null) + # source_identity = "ENV:${each.value.project_key}:${each.value.env_key}" + # source_key = each.value.env_key + # source_project_key = each.value.project_key + # source_name = each.value.env_data.name + # } } ############################################# @@ -100,15 +124,26 @@ resource "dbtcloud_environment" "protected_environments" { project_id = each.value.project_id name = each.value.env_data.name type = each.value.env_data.type - connection_id = local.resolve_connection_id[each.key] - credential_id = local.resolve_credential_id[each.key] + connection_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_connection_id[each.key] + credential_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_credential_id[each.key] dbt_version = try(each.value.env_data.dbt_version, null) enable_model_query_history = try(each.value.env_data.enable_model_query_history, null) custom_branch = try(each.value.env_data.custom_branch, null) deployment_type = try(each.value.env_data.deployment_type, null) use_custom_branch = try(each.value.env_data.custom_branch, null) != null - extended_attributes_id = local.resolve_extended_attributes_id[each.key] + primary_profile_id = local.resolve_primary_profile_id[each.key] + extended_attributes_id = local.resolve_primary_profile_id[each.key] != null ? null : local.resolve_extended_attributes_id[each.key] + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_environment. + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.env_data.id, null) + # source_identity = "ENV:${each.value.project_key}:${each.value.env_key}" + # source_key = each.value.env_key + # source_project_key = each.value.project_key + # source_name = each.value.env_data.name + # } lifecycle { prevent_destroy = true diff --git a/modules/environments/variables.tf b/modules/environments/variables.tf index ce697b9..dde28e8 100644 --- a/modules/environments/variables.tf +++ b/modules/environments/variables.tf @@ -21,7 +21,19 @@ variable "global_connection_ids" { } variable "extended_attribute_ids" { - description = "Map of composite key (project_key_ea_key) to extended_attributes resource ID (from extended_attributes module)" - type = map(string) + description = "Map of composite key (project_key_ea_key) to dbt Cloud extended_attributes_id (numeric; from extended_attributes module)." + type = map(number) + default = {} +} + +variable "extended_attribute_ids_by_source_id" { + description = "Maps legacy YAML extended_attributes[].id to Terraform-managed extended_attributes_id (from extended_attributes module)." + type = map(number) + default = {} +} + +variable "profile_ids" { + description = "Map of composite key (project_key_profile_key) to dbt Cloud profile_id (from profiles module); used when environments set primary_profile_key." + type = map(number) default = {} } diff --git a/modules/extended_attributes/main.tf b/modules/extended_attributes/main.tf index 68c90a5..56abea3 100644 --- a/modules/extended_attributes/main.tf +++ b/modules/extended_attributes/main.tf @@ -21,15 +21,87 @@ locals { ] ]) - ea_map = { + protected_extended_attributes = [ for item in local.all_extended_attributes : - item.composite_key => item + item + if try(item.ea_data.protected, false) == true + ] + + unprotected_extended_attributes = [ + for item in local.all_extended_attributes : + item + if try(item.ea_data.protected, false) != true + ] + + ea_body = { + for item in local.all_extended_attributes : + item.composite_key => try(item.ea_data.extended_attributes, {}) } + + extended_attribute_ids_by_source_id = merge( + { + for item in local.unprotected_extended_attributes : + tostring(try(item.ea_data.id, null)) => + dbtcloud_extended_attributes.extended_attributes[item.composite_key].extended_attributes_id + if try(item.ea_data.id, null) != null + }, + { + for item in local.protected_extended_attributes : + tostring(try(item.ea_data.id, null)) => + dbtcloud_extended_attributes.protected_extended_attributes[item.composite_key].extended_attributes_id + if try(item.ea_data.id, null) != null + } + ) } +############################################# +# Unprotected extended attributes +############################################# resource "dbtcloud_extended_attributes" "extended_attributes" { - for_each = local.ea_map + for_each = { + for item in local.unprotected_extended_attributes : + item.composite_key => item + } project_id = each.value.project_id - extended_attributes = jsonencode(try(each.value.ea_data.content, each.value.ea_data)) + state = coalesce(try(each.value.ea_data.state, null), 1) + extended_attributes = jsonencode(local.ea_body[each.key]) + + # Deferred: stock dbtcloud provider has no resource_metadata on this resource (terraform providers schema). + # resource_metadata = { + # source_project_id = lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.ea_data.id, null) + # source_identity = "EXTATTR:${each.value.project_key}:${each.value.ea_key}" + # source_key = each.value.ea_key + # source_name = each.value.ea_key + # source_project_key = each.value.project_key + # } +} + +############################################# +# Protected extended attributes — lifecycle.prevent_destroy +############################################# +resource "dbtcloud_extended_attributes" "protected_extended_attributes" { + for_each = { + for item in local.protected_extended_attributes : + item.composite_key => item + } + + project_id = each.value.project_id + state = coalesce(try(each.value.ea_data.state, null), 1) + extended_attributes = jsonencode(local.ea_body[each.key]) + + # Deferred: stock dbtcloud provider has no resource_metadata on this resource. + # resource_metadata = { + # source_project_id = lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.ea_data.id, null) + # source_identity = "EXTATTR:${each.value.project_key}:${each.value.ea_key}" + # source_key = each.value.ea_key + # source_name = each.value.ea_key + # source_project_key = each.value.project_key + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/extended_attributes/outputs.tf b/modules/extended_attributes/outputs.tf index 640faeb..8954b26 100644 --- a/modules/extended_attributes/outputs.tf +++ b/modules/extended_attributes/outputs.tf @@ -1,4 +1,18 @@ output "extended_attribute_ids" { - description = "Map of composite key (project_key_ea_key) to extended_attributes resource ID" - value = { for k, ea in dbtcloud_extended_attributes.extended_attributes : k => ea.id } + description = "Map of composite key (project_key_ea_key) to dbt Cloud extended_attributes_id (numeric API id for environments/profiles)." + value = merge( + { + for k, ea in dbtcloud_extended_attributes.extended_attributes : + k => ea.extended_attributes_id + }, + { + for k, ea in dbtcloud_extended_attributes.protected_extended_attributes : + k => ea.extended_attributes_id + } + ) +} + +output "extended_attribute_ids_by_source_id" { + description = "Maps YAML extended_attributes[].id (legacy dbt Cloud id) to Terraform-managed extended_attributes_id after apply." + value = local.extended_attribute_ids_by_source_id } diff --git a/modules/global_connections/main.tf b/modules/global_connections/main.tf index bc4e1fc..6295262 100644 --- a/modules/global_connections/main.tf +++ b/modules/global_connections/main.tf @@ -9,9 +9,22 @@ terraform { } locals { + # BigQuery: deterministic placeholder when no real key material is supplied (matches v2 / API expectations for create). + bigquery_dummy_private_key_id = "0000000000000000000000000000000000000000" + bigquery_dummy_private_key_seed = "terraform-dbtcloud-yaml:dummy:bigquery:private-key" + bigquery_dummy_private_key_body = join("", [ + for i in range(0, 8) : base64encode(sha512(format("%s:%d", local.bigquery_dummy_private_key_seed, i))) + ]) + bigquery_dummy_private_key_lines = regexall(".{1,64}", local.bigquery_dummy_private_key_body) + bigquery_dummy_private_key = join("\n", concat( + [format("-----BEGIN %s-----", "PRIVATE KEY")], + local.bigquery_dummy_private_key_lines, + [format("-----END %s-----", "PRIVATE KEY")] + )) + connections_map = { for conn in var.connections_data : - try(conn.key, conn.name) => conn + conn.key => conn } protected_connections_map = { @@ -25,6 +38,38 @@ locals { k => conn if try(conn.protected, false) != true } + + privatelink_endpoints_map = { + for ple in var.privatelink_endpoints : + ple.key => ple + } + + needs_privatelink_data = length([ + for k, conn in local.connections_map : k + if try(conn.private_link_endpoint_key, null) != null && try(conn.private_link_endpoint_id, null) == null + ]) > 0 + + # Resolve PrivateLink via globals.privatelink_endpoints + account data source when only private_link_endpoint_key is set. + private_link_endpoint_id_by_key = { + for k, conn in local.connections_map : k => ( + try(conn.private_link_endpoint_id, null) != null ? + conn.private_link_endpoint_id : + ( + length(data.dbtcloud_privatelink_endpoints.all) > 0 && + try(conn.private_link_endpoint_key, null) != null && + lookup(local.privatelink_endpoints_map, conn.private_link_endpoint_key, null) != null + ) ? data.dbtcloud_privatelink_endpoints.all[0].endpoints[ + index( + [for ep in data.dbtcloud_privatelink_endpoints.all[0].endpoints : ep.id], + lookup(local.privatelink_endpoints_map, conn.private_link_endpoint_key, { endpoint_id = null }).endpoint_id + ) + ].id : null + ) + } +} + +data "dbtcloud_privatelink_endpoints" "all" { + count = local.needs_privatelink_data ? 1 : 0 } ############################################# @@ -36,51 +81,153 @@ resource "dbtcloud_global_connection" "connections" { name = each.value.name - private_link_endpoint_id = try(each.value.private_link_endpoint_id, null) + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "CON:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + private_link_endpoint_id = local.private_link_endpoint_id_by_key[each.key] + + # Prefer provider_config, then details (globals.connections[]); else top-level fields on the connection object. databricks = try(each.value.type, "") == "databricks" ? { - host = try(each.value.host, "") - http_path = try(each.value.http_path, "") - catalog = try(each.value.catalog, null) - client_id = try(var.connection_credentials[each.key].client_id, null) - client_secret = try(var.connection_credentials[each.key].client_secret, null) + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + http_path = try(each.value.provider_config.http_path, try(each.value.details.http_path, try(each.value.http_path, ""))) + catalog = try(each.value.provider_config.catalog, try(each.value.details.catalog, try(each.value.catalog, null))) + client_id = try(var.connection_credentials[each.key].client_id, try(each.value.provider_config.client_id, try(each.value.details.client_id, null))) + client_secret = try(var.connection_credentials[each.key].client_secret, try(each.value.provider_config.client_secret, try(each.value.details.client_secret, null))) } : null snowflake = try(each.value.type, "") == "snowflake" ? { - account = try(each.value.account, "") - database = try(each.value.database, "") - warehouse = try(each.value.warehouse, "") - role = try(each.value.role, null) - client_session_keep_alive = try(each.value.client_session_keep_alive, false) - allow_sso = try(each.value.allow_sso, false) - oauth_client_id = try(var.connection_credentials[each.key].oauth_client_id, null) - oauth_client_secret = try(var.connection_credentials[each.key].oauth_client_secret, null) + account = try(each.value.provider_config.account, try(each.value.details.account, try(each.value.account, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + warehouse = try(each.value.provider_config.warehouse, try(each.value.details.warehouse, try(each.value.warehouse, ""))) + role = try(each.value.provider_config.role, try(each.value.details.role, try(each.value.role, null))) + client_session_keep_alive = try(each.value.provider_config.client_session_keep_alive, try(each.value.details.client_session_keep_alive, try(each.value.client_session_keep_alive, false))) + allow_sso = try(each.value.provider_config.allow_sso, try(each.value.details.allow_sso, try(each.value.allow_sso, false))) + oauth_client_id = try(var.connection_credentials[each.key].oauth_client_id, try(each.value.provider_config.oauth_client_id, try(each.value.details.oauth_client_id, null))) + oauth_client_secret = try(var.connection_credentials[each.key].oauth_client_secret, try(each.value.provider_config.oauth_client_secret, try(each.value.details.oauth_client_secret, null))) } : null bigquery = try(each.value.type, "") == "bigquery" ? { - gcp_project_id = try(each.value.gcp_project_id, "") - private_key_id = try(var.connection_credentials[each.key].private_key_id, null) - private_key = try(var.connection_credentials[each.key].private_key, null) - client_email = try(each.value.client_email, null) - client_id = try(each.value.client_id, null) - auth_uri = try(each.value.auth_uri, null) - token_uri = try(each.value.token_uri, null) - auth_provider_x509_cert_url = try(each.value.auth_provider_x509_cert_url, null) - client_x509_cert_url = try(each.value.client_x509_cert_url, null) - timeout_seconds = try(each.value.timeout_seconds, null) - location = try(each.value.location, null) + gcp_project_id = try(each.value.provider_config.gcp_project_id, try(each.value.details.gcp_project_id, try(each.value.gcp_project_id, ""))) + deployment_env_auth_type = try(each.value.provider_config.deployment_env_auth_type, try(each.value.details.deployment_env_auth_type, null)) + private_key_id = coalesce(nonsensitive(try(var.connection_credentials[each.key].private_key_id, null)), try(each.value.provider_config.private_key_id, try(each.value.details.private_key_id, null)), local.bigquery_dummy_private_key_id) + private_key = coalesce(try(var.connection_credentials[each.key].private_key, try(each.value.provider_config.private_key, try(each.value.details.private_key, null))), local.bigquery_dummy_private_key) + client_email = try(each.value.provider_config.client_email, try(each.value.details.client_email, try(each.value.client_email, null))) + client_id = try(each.value.provider_config.client_id, try(each.value.details.client_id, try(each.value.client_id, null))) + auth_uri = try(each.value.provider_config.auth_uri, try(each.value.details.auth_uri, try(each.value.auth_uri, null))) + token_uri = try(each.value.provider_config.token_uri, try(each.value.details.token_uri, try(each.value.token_uri, null))) + auth_provider_x509_cert_url = try(each.value.provider_config.auth_provider_x509_cert_url, try(each.value.details.auth_provider_x509_cert_url, try(each.value.auth_provider_x509_cert_url, null))) + client_x509_cert_url = try(each.value.provider_config.client_x509_cert_url, try(each.value.details.client_x509_cert_url, try(each.value.client_x509_cert_url, null))) + application_id = try(var.connection_credentials[each.key].application_id, try(each.value.provider_config.application_id, try(each.value.details.application_id, null))) + application_secret = try(var.connection_credentials[each.key].application_secret, try(each.value.provider_config.application_secret, try(each.value.details.application_secret, null))) + scopes = try(each.value.provider_config.scopes, try(each.value.details.scopes, null)) + timeout_seconds = try(each.value.provider_config.timeout_seconds, try(each.value.details.timeout_seconds, try(each.value.timeout_seconds, null))) + location = try(each.value.provider_config.location, try(each.value.details.location, try(each.value.location, null))) + maximum_bytes_billed = try(each.value.provider_config.maximum_bytes_billed, try(each.value.details.maximum_bytes_billed, null)) + priority = try(each.value.provider_config.priority, try(each.value.details.priority, null)) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, null)) + job_creation_timeout_seconds = try(each.value.provider_config.job_creation_timeout_seconds, null) + job_execution_timeout_seconds = try(each.value.provider_config.job_execution_timeout_seconds, null) + job_retry_deadline_seconds = try(each.value.provider_config.job_retry_deadline_seconds, null) + execution_project = try(each.value.provider_config.execution_project, try(each.value.details.execution_project, null)) + impersonate_service_account = try(each.value.provider_config.impersonate_service_account, null) + dataproc_region = try(each.value.provider_config.dataproc_region, try(each.value.details.dataproc_region, null)) + dataproc_cluster_name = try(each.value.provider_config.dataproc_cluster_name, try(each.value.details.dataproc_cluster_name, null)) + gcs_bucket = try(each.value.provider_config.gcs_bucket, try(each.value.details.gcs_bucket, null)) + use_latest_adapter = try(each.value.provider_config.use_latest_adapter, null) } : null postgres = try(each.value.type, "") == "postgres" ? { - hostname = try(each.value.hostname, "") - dbname = try(each.value.dbname, "") - port = try(each.value.port, 5432) + hostname = try(each.value.provider_config.hostname, try(each.value.details.hostname, try(each.value.hostname, ""))) + dbname = try(each.value.provider_config.dbname, try(each.value.details.dbname, try(each.value.dbname, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, 5432))) + ssh_tunnel = try(each.value.provider_config.ssh_tunnel_hostname, try(each.value.ssh_tunnel_hostname, null)) != null ? { + hostname = try(each.value.provider_config.ssh_tunnel_hostname, each.value.ssh_tunnel_hostname) + port = try(each.value.provider_config.ssh_tunnel_port, try(each.value.ssh_tunnel_port, 22)) + username = try(each.value.provider_config.ssh_tunnel_username, try(each.value.ssh_tunnel_username, "dbt")) + } : null } : null redshift = try(each.value.type, "") == "redshift" ? { - hostname = try(each.value.hostname, "") - dbname = try(each.value.dbname, "") - port = try(each.value.port, 5439) + hostname = try(each.value.provider_config.hostname, try(each.value.details.hostname, try(each.value.hostname, ""))) + dbname = try(each.value.provider_config.dbname, try(each.value.details.dbname, try(each.value.dbname, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, 5439))) + ssh_tunnel = try(each.value.provider_config.ssh_tunnel_hostname, try(each.value.ssh_tunnel_hostname, null)) != null ? { + hostname = try(each.value.provider_config.ssh_tunnel_hostname, each.value.ssh_tunnel_hostname) + port = try(each.value.provider_config.ssh_tunnel_port, try(each.value.ssh_tunnel_port, 22)) + username = try(each.value.provider_config.ssh_tunnel_username, try(each.value.ssh_tunnel_username, "dbt")) + } : null + } : null + + athena = try(each.value.type, "") == "athena" ? { + region_name = try(each.value.provider_config.region_name, try(each.value.details.region_name, try(each.value.region_name, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + s3_staging_dir = try(each.value.provider_config.s3_staging_dir, try(each.value.details.s3_staging_dir, try(each.value.s3_staging_dir, ""))) + work_group = try(each.value.provider_config.work_group, try(each.value.details.work_group, try(each.value.work_group, null))) + s3_data_dir = try(each.value.provider_config.s3_data_dir, try(each.value.details.s3_data_dir, try(each.value.s3_data_dir, null))) + s3_tmp_table_dir = try(each.value.provider_config.s3_tmp_table_dir, try(each.value.details.s3_tmp_table_dir, try(each.value.s3_tmp_table_dir, null))) + s3_data_naming = try(each.value.provider_config.s3_data_naming, try(each.value.details.s3_data_naming, try(each.value.s3_data_naming, null))) + num_retries = try(each.value.provider_config.num_retries, try(each.value.details.num_retries, try(each.value.num_retries, null))) + num_boto3_retries = try(each.value.provider_config.num_boto3_retries, try(each.value.details.num_boto3_retries, try(each.value.num_boto3_retries, null))) + num_iceberg_retries = try(each.value.provider_config.num_iceberg_retries, try(each.value.details.num_iceberg_retries, try(each.value.num_iceberg_retries, null))) + poll_interval = try(each.value.provider_config.poll_interval, try(each.value.details.poll_interval, try(each.value.poll_interval, null))) + spark_work_group = try(each.value.provider_config.spark_work_group, try(each.value.details.spark_work_group, try(each.value.spark_work_group, null))) + } : null + + fabric = try(each.value.type, "") == "fabric" ? { + server = try(each.value.provider_config.server, try(each.value.details.server, try(each.value.server, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + login_timeout = try(each.value.provider_config.login_timeout, try(each.value.details.login_timeout, try(each.value.login_timeout, null))) + query_timeout = try(each.value.provider_config.query_timeout, try(each.value.details.query_timeout, try(each.value.query_timeout, null))) + } : null + + synapse = try(each.value.type, "") == "synapse" ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + login_timeout = try(each.value.provider_config.login_timeout, try(each.value.details.login_timeout, try(each.value.login_timeout, null))) + query_timeout = try(each.value.provider_config.query_timeout, try(each.value.details.query_timeout, try(each.value.query_timeout, null))) + } : null + + # YAML type starburst_trino maps to provider starburst block. + starburst = contains(["starburst", "starburst_trino"], try(each.value.type, "")) ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + method = try(each.value.provider_config.method, try(each.value.details.method, try(each.value.method, null))) + } : null + + teradata = try(each.value.type, "") == "teradata" ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + tmode = try(each.value.provider_config.tmode, try(each.value.details.tmode, try(each.value.tmode, "ANSI"))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(tostring(each.value.port), null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + request_timeout = try(each.value.provider_config.request_timeout, try(each.value.details.request_timeout, try(each.value.request_timeout, null))) + } : null + + salesforce = try(each.value.type, "") == "salesforce" ? { + login_url = try(each.value.provider_config.login_url, try(each.value.details.login_url, try(each.value.login_url, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, null))) + data_transform_run_timeout = try(each.value.provider_config.data_transform_run_timeout, try(each.value.details.data_transform_run_timeout, try(each.value.data_transform_run_timeout, null))) + } : null + + # YAML type spark maps to provider apache_spark (same as credentials module). + apache_spark = contains(["apache_spark", "spark"], try(each.value.type, "")) ? { + method = try(each.value.provider_config.method, try(each.value.details.method, try(each.value.method, "http"))) + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + cluster = try(each.value.provider_config.cluster, try(each.value.details.cluster, try(each.value.cluster, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + organization = try(each.value.provider_config.organization, try(each.value.details.organization, try(each.value.organization, null))) + user = try(each.value.provider_config.user, try(each.value.details.user, try(each.value.user, null))) + auth = try(each.value.provider_config.auth, try(each.value.details.auth, try(each.value.auth, null))) + connect_timeout = try(each.value.provider_config.connect_timeout, try(each.value.details.connect_timeout, try(each.value.connect_timeout, null))) + connect_retries = try(each.value.provider_config.connect_retries, try(each.value.details.connect_retries, try(each.value.connect_retries, null))) } : null } @@ -93,51 +240,150 @@ resource "dbtcloud_global_connection" "protected_connections" { name = each.value.name - private_link_endpoint_id = try(each.value.private_link_endpoint_id, null) + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "CON:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + + private_link_endpoint_id = local.private_link_endpoint_id_by_key[each.key] databricks = try(each.value.type, "") == "databricks" ? { - host = try(each.value.host, "") - http_path = try(each.value.http_path, "") - catalog = try(each.value.catalog, null) - client_id = try(var.connection_credentials[each.key].client_id, null) - client_secret = try(var.connection_credentials[each.key].client_secret, null) + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + http_path = try(each.value.provider_config.http_path, try(each.value.details.http_path, try(each.value.http_path, ""))) + catalog = try(each.value.provider_config.catalog, try(each.value.details.catalog, try(each.value.catalog, null))) + client_id = try(var.connection_credentials[each.key].client_id, try(each.value.provider_config.client_id, try(each.value.details.client_id, null))) + client_secret = try(var.connection_credentials[each.key].client_secret, try(each.value.provider_config.client_secret, try(each.value.details.client_secret, null))) } : null snowflake = try(each.value.type, "") == "snowflake" ? { - account = try(each.value.account, "") - database = try(each.value.database, "") - warehouse = try(each.value.warehouse, "") - role = try(each.value.role, null) - client_session_keep_alive = try(each.value.client_session_keep_alive, false) - allow_sso = try(each.value.allow_sso, false) - oauth_client_id = try(var.connection_credentials[each.key].oauth_client_id, null) - oauth_client_secret = try(var.connection_credentials[each.key].oauth_client_secret, null) + account = try(each.value.provider_config.account, try(each.value.details.account, try(each.value.account, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + warehouse = try(each.value.provider_config.warehouse, try(each.value.details.warehouse, try(each.value.warehouse, ""))) + role = try(each.value.provider_config.role, try(each.value.details.role, try(each.value.role, null))) + client_session_keep_alive = try(each.value.provider_config.client_session_keep_alive, try(each.value.details.client_session_keep_alive, try(each.value.client_session_keep_alive, false))) + allow_sso = try(each.value.provider_config.allow_sso, try(each.value.details.allow_sso, try(each.value.allow_sso, false))) + oauth_client_id = try(var.connection_credentials[each.key].oauth_client_id, try(each.value.provider_config.oauth_client_id, try(each.value.details.oauth_client_id, null))) + oauth_client_secret = try(var.connection_credentials[each.key].oauth_client_secret, try(each.value.provider_config.oauth_client_secret, try(each.value.details.oauth_client_secret, null))) } : null bigquery = try(each.value.type, "") == "bigquery" ? { - gcp_project_id = try(each.value.gcp_project_id, "") - private_key_id = try(var.connection_credentials[each.key].private_key_id, null) - private_key = try(var.connection_credentials[each.key].private_key, null) - client_email = try(each.value.client_email, null) - client_id = try(each.value.client_id, null) - auth_uri = try(each.value.auth_uri, null) - token_uri = try(each.value.token_uri, null) - auth_provider_x509_cert_url = try(each.value.auth_provider_x509_cert_url, null) - client_x509_cert_url = try(each.value.client_x509_cert_url, null) - timeout_seconds = try(each.value.timeout_seconds, null) - location = try(each.value.location, null) + gcp_project_id = try(each.value.provider_config.gcp_project_id, try(each.value.details.gcp_project_id, try(each.value.gcp_project_id, ""))) + deployment_env_auth_type = try(each.value.provider_config.deployment_env_auth_type, try(each.value.details.deployment_env_auth_type, null)) + private_key_id = coalesce(nonsensitive(try(var.connection_credentials[each.key].private_key_id, null)), try(each.value.provider_config.private_key_id, try(each.value.details.private_key_id, null)), local.bigquery_dummy_private_key_id) + private_key = coalesce(try(var.connection_credentials[each.key].private_key, try(each.value.provider_config.private_key, try(each.value.details.private_key, null))), local.bigquery_dummy_private_key) + client_email = try(each.value.provider_config.client_email, try(each.value.details.client_email, try(each.value.client_email, null))) + client_id = try(each.value.provider_config.client_id, try(each.value.details.client_id, try(each.value.client_id, null))) + auth_uri = try(each.value.provider_config.auth_uri, try(each.value.details.auth_uri, try(each.value.auth_uri, null))) + token_uri = try(each.value.provider_config.token_uri, try(each.value.details.token_uri, try(each.value.token_uri, null))) + auth_provider_x509_cert_url = try(each.value.provider_config.auth_provider_x509_cert_url, try(each.value.details.auth_provider_x509_cert_url, try(each.value.auth_provider_x509_cert_url, null))) + client_x509_cert_url = try(each.value.provider_config.client_x509_cert_url, try(each.value.details.client_x509_cert_url, try(each.value.client_x509_cert_url, null))) + application_id = try(var.connection_credentials[each.key].application_id, try(each.value.provider_config.application_id, try(each.value.details.application_id, null))) + application_secret = try(var.connection_credentials[each.key].application_secret, try(each.value.provider_config.application_secret, try(each.value.details.application_secret, null))) + scopes = try(each.value.provider_config.scopes, try(each.value.details.scopes, null)) + timeout_seconds = try(each.value.provider_config.timeout_seconds, try(each.value.details.timeout_seconds, try(each.value.timeout_seconds, null))) + location = try(each.value.provider_config.location, try(each.value.details.location, try(each.value.location, null))) + maximum_bytes_billed = try(each.value.provider_config.maximum_bytes_billed, try(each.value.details.maximum_bytes_billed, null)) + priority = try(each.value.provider_config.priority, try(each.value.details.priority, null)) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, null)) + job_creation_timeout_seconds = try(each.value.provider_config.job_creation_timeout_seconds, null) + job_execution_timeout_seconds = try(each.value.provider_config.job_execution_timeout_seconds, null) + job_retry_deadline_seconds = try(each.value.provider_config.job_retry_deadline_seconds, null) + execution_project = try(each.value.provider_config.execution_project, try(each.value.details.execution_project, null)) + impersonate_service_account = try(each.value.provider_config.impersonate_service_account, null) + dataproc_region = try(each.value.provider_config.dataproc_region, try(each.value.details.dataproc_region, null)) + dataproc_cluster_name = try(each.value.provider_config.dataproc_cluster_name, try(each.value.details.dataproc_cluster_name, null)) + gcs_bucket = try(each.value.provider_config.gcs_bucket, try(each.value.details.gcs_bucket, null)) + use_latest_adapter = try(each.value.provider_config.use_latest_adapter, null) } : null postgres = try(each.value.type, "") == "postgres" ? { - hostname = try(each.value.hostname, "") - dbname = try(each.value.dbname, "") - port = try(each.value.port, 5432) + hostname = try(each.value.provider_config.hostname, try(each.value.details.hostname, try(each.value.hostname, ""))) + dbname = try(each.value.provider_config.dbname, try(each.value.details.dbname, try(each.value.dbname, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, 5432))) + ssh_tunnel = try(each.value.provider_config.ssh_tunnel_hostname, try(each.value.ssh_tunnel_hostname, null)) != null ? { + hostname = try(each.value.provider_config.ssh_tunnel_hostname, each.value.ssh_tunnel_hostname) + port = try(each.value.provider_config.ssh_tunnel_port, try(each.value.ssh_tunnel_port, 22)) + username = try(each.value.provider_config.ssh_tunnel_username, try(each.value.ssh_tunnel_username, "dbt")) + } : null } : null redshift = try(each.value.type, "") == "redshift" ? { - hostname = try(each.value.hostname, "") - dbname = try(each.value.dbname, "") - port = try(each.value.port, 5439) + hostname = try(each.value.provider_config.hostname, try(each.value.details.hostname, try(each.value.hostname, ""))) + dbname = try(each.value.provider_config.dbname, try(each.value.details.dbname, try(each.value.dbname, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, 5439))) + ssh_tunnel = try(each.value.provider_config.ssh_tunnel_hostname, try(each.value.ssh_tunnel_hostname, null)) != null ? { + hostname = try(each.value.provider_config.ssh_tunnel_hostname, each.value.ssh_tunnel_hostname) + port = try(each.value.provider_config.ssh_tunnel_port, try(each.value.ssh_tunnel_port, 22)) + username = try(each.value.provider_config.ssh_tunnel_username, try(each.value.ssh_tunnel_username, "dbt")) + } : null + } : null + + athena = try(each.value.type, "") == "athena" ? { + region_name = try(each.value.provider_config.region_name, try(each.value.details.region_name, try(each.value.region_name, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + s3_staging_dir = try(each.value.provider_config.s3_staging_dir, try(each.value.details.s3_staging_dir, try(each.value.s3_staging_dir, ""))) + work_group = try(each.value.provider_config.work_group, try(each.value.details.work_group, try(each.value.work_group, null))) + s3_data_dir = try(each.value.provider_config.s3_data_dir, try(each.value.details.s3_data_dir, try(each.value.s3_data_dir, null))) + s3_tmp_table_dir = try(each.value.provider_config.s3_tmp_table_dir, try(each.value.details.s3_tmp_table_dir, try(each.value.s3_tmp_table_dir, null))) + s3_data_naming = try(each.value.provider_config.s3_data_naming, try(each.value.details.s3_data_naming, try(each.value.s3_data_naming, null))) + num_retries = try(each.value.provider_config.num_retries, try(each.value.details.num_retries, try(each.value.num_retries, null))) + num_boto3_retries = try(each.value.provider_config.num_boto3_retries, try(each.value.details.num_boto3_retries, try(each.value.num_boto3_retries, null))) + num_iceberg_retries = try(each.value.provider_config.num_iceberg_retries, try(each.value.details.num_iceberg_retries, try(each.value.num_iceberg_retries, null))) + poll_interval = try(each.value.provider_config.poll_interval, try(each.value.details.poll_interval, try(each.value.poll_interval, null))) + spark_work_group = try(each.value.provider_config.spark_work_group, try(each.value.details.spark_work_group, try(each.value.spark_work_group, null))) + } : null + + fabric = try(each.value.type, "") == "fabric" ? { + server = try(each.value.provider_config.server, try(each.value.details.server, try(each.value.server, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + login_timeout = try(each.value.provider_config.login_timeout, try(each.value.details.login_timeout, try(each.value.login_timeout, null))) + query_timeout = try(each.value.provider_config.query_timeout, try(each.value.details.query_timeout, try(each.value.query_timeout, null))) + } : null + + synapse = try(each.value.type, "") == "synapse" ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + login_timeout = try(each.value.provider_config.login_timeout, try(each.value.details.login_timeout, try(each.value.login_timeout, null))) + query_timeout = try(each.value.provider_config.query_timeout, try(each.value.details.query_timeout, try(each.value.query_timeout, null))) + } : null + + starburst = contains(["starburst", "starburst_trino"], try(each.value.type, "")) ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + method = try(each.value.provider_config.method, try(each.value.details.method, try(each.value.method, null))) + } : null + + teradata = try(each.value.type, "") == "teradata" ? { + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + tmode = try(each.value.provider_config.tmode, try(each.value.details.tmode, try(each.value.tmode, "ANSI"))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(tostring(each.value.port), null))) + retries = try(each.value.provider_config.retries, try(each.value.details.retries, try(each.value.retries, null))) + request_timeout = try(each.value.provider_config.request_timeout, try(each.value.details.request_timeout, try(each.value.request_timeout, null))) + } : null + + salesforce = try(each.value.type, "") == "salesforce" ? { + login_url = try(each.value.provider_config.login_url, try(each.value.details.login_url, try(each.value.login_url, ""))) + database = try(each.value.provider_config.database, try(each.value.details.database, try(each.value.database, null))) + data_transform_run_timeout = try(each.value.provider_config.data_transform_run_timeout, try(each.value.details.data_transform_run_timeout, try(each.value.data_transform_run_timeout, null))) + } : null + + apache_spark = contains(["apache_spark", "spark"], try(each.value.type, "")) ? { + method = try(each.value.provider_config.method, try(each.value.details.method, try(each.value.method, "http"))) + host = try(each.value.provider_config.host, try(each.value.details.host, try(each.value.host, ""))) + cluster = try(each.value.provider_config.cluster, try(each.value.details.cluster, try(each.value.cluster, ""))) + port = try(each.value.provider_config.port, try(each.value.details.port, try(each.value.port, null))) + organization = try(each.value.provider_config.organization, try(each.value.details.organization, try(each.value.organization, null))) + user = try(each.value.provider_config.user, try(each.value.details.user, try(each.value.user, null))) + auth = try(each.value.provider_config.auth, try(each.value.details.auth, try(each.value.auth, null))) + connect_timeout = try(each.value.provider_config.connect_timeout, try(each.value.details.connect_timeout, try(each.value.connect_timeout, null))) + connect_retries = try(each.value.provider_config.connect_retries, try(each.value.details.connect_retries, try(each.value.connect_retries, null))) } : null lifecycle { diff --git a/modules/global_connections/variables.tf b/modules/global_connections/variables.tf index 66f4e22..d37d352 100644 --- a/modules/global_connections/variables.tf +++ b/modules/global_connections/variables.tf @@ -10,3 +10,9 @@ variable "connection_credentials" { default = {} sensitive = true } + +variable "privatelink_endpoints" { + description = "Optional account-level PrivateLink endpoint registry (key + endpoint_id) for resolving global_connections[].private_link_endpoint_key" + type = any + default = [] +} diff --git a/modules/groups/main.tf b/modules/groups/main.tf index a2e190f..2951b43 100644 --- a/modules/groups/main.tf +++ b/modules/groups/main.tf @@ -11,7 +11,12 @@ terraform { locals { groups_map = { for g in var.groups_data : - try(g.key, g.name) => g + g.key => g + } + + groups_permissions_by_key = { + for k, g in local.groups_map : + k => try(g.group_permissions, []) } protected_groups_map = { @@ -32,14 +37,32 @@ resource "dbtcloud_group" "groups" { name = each.value.name assign_by_default = try(each.value.assign_by_default, false) - sso_mapping_groups = try(each.value.sso_mapping_groups, null) + sso_mapping_groups = try(each.value.sso_mapping_groups, []) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "GRP:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } dynamic "group_permissions" { - for_each = try(each.value.permissions, []) + for_each = var.skip_global_project_permissions ? [] : tolist(try(local.groups_permissions_by_key[each.key], [])) content { - permission_set = group_permissions.value.permission_set - project_id = try(group_permissions.value.project_id, null) - all_projects = try(group_permissions.value.all_projects, group_permissions.value.project_id == null) + permission_set = group_permissions.value.permission_set + all_projects = try(group_permissions.value.all_projects, group_permissions.value.project_id == null) + project_id = try(group_permissions.value.project_id, null) + writable_environment_categories = try(group_permissions.value.writable_environment_categories, []) + } + } + dynamic "group_permissions" { + for_each = var.skip_global_project_permissions ? tolist(try(local.groups_permissions_by_key[each.key], [])) : [] + content { + permission_set = group_permissions.value.permission_set + all_projects = true + project_id = null + writable_environment_categories = try(group_permissions.value.writable_environment_categories, []) } } } @@ -49,14 +72,32 @@ resource "dbtcloud_group" "protected_groups" { name = each.value.name assign_by_default = try(each.value.assign_by_default, false) - sso_mapping_groups = try(each.value.sso_mapping_groups, null) + sso_mapping_groups = try(each.value.sso_mapping_groups, []) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "GRP:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } dynamic "group_permissions" { - for_each = try(each.value.permissions, []) + for_each = var.skip_global_project_permissions ? [] : tolist(try(local.groups_permissions_by_key[each.key], [])) + content { + permission_set = group_permissions.value.permission_set + all_projects = try(group_permissions.value.all_projects, group_permissions.value.project_id == null) + project_id = try(group_permissions.value.project_id, null) + writable_environment_categories = try(group_permissions.value.writable_environment_categories, []) + } + } + dynamic "group_permissions" { + for_each = var.skip_global_project_permissions ? tolist(try(local.groups_permissions_by_key[each.key], [])) : [] content { - permission_set = group_permissions.value.permission_set - project_id = try(group_permissions.value.project_id, null) - all_projects = try(group_permissions.value.all_projects, group_permissions.value.project_id == null) + permission_set = group_permissions.value.permission_set + all_projects = true + project_id = null + writable_environment_categories = try(group_permissions.value.writable_environment_categories, []) } } diff --git a/modules/groups/variables.tf b/modules/groups/variables.tf index e655ba3..42be196 100644 --- a/modules/groups/variables.tf +++ b/modules/groups/variables.tf @@ -3,3 +3,9 @@ variable "groups_data" { type = any default = [] } + +variable "skip_global_project_permissions" { + description = "When true, project-scoped permission entries are collapsed to all_projects-only blocks so global groups do not expand the project dependency graph (scoped global-object adoption)." + type = bool + default = false +} diff --git a/modules/ip_restrictions/main.tf b/modules/ip_restrictions/main.tf index 687eae0..60eef43 100644 --- a/modules/ip_restrictions/main.tf +++ b/modules/ip_restrictions/main.tf @@ -8,21 +8,73 @@ terraform { } } +############################################# +# IP Restrictions Rules (account-level collection) +############################################# + locals { ip_rules_map = { for rule in var.ip_rules_data : - try(rule.key, rule.name) => rule + rule.key => rule + } + + unprotected_ip_restrictions_map = { + for key, rule in local.ip_rules_map : + key => rule if !try(rule.protected, false) + } + + protected_ip_restrictions_map = { + for key, rule in local.ip_rules_map : + key => rule if try(rule.protected, false) } } resource "dbtcloud_ip_restrictions_rule" "ip_rules" { - for_each = local.ip_rules_map + for_each = local.unprotected_ip_restrictions_map name = each.value.name type = try(each.value.type, "allow") description = try(each.value.description, null) - rule_set_enabled = try(each.value.rule_set_enabled, true) + rule_set_enabled = try(each.value.rule_set_enabled, false) + cidrs = [ - for c in try(each.value.cidrs, []) : { cidr = c.cidr } + for c in try(each.value.cidrs, []) : { + cidr = c.cidr + } ] + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/ip_restrictions.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "IPRST:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } +} + +resource "dbtcloud_ip_restrictions_rule" "protected_ip_rules" { + for_each = local.protected_ip_restrictions_map + + name = each.value.name + type = try(each.value.type, "allow") + description = try(each.value.description, null) + rule_set_enabled = try(each.value.rule_set_enabled, false) + + cidrs = [ + for c in try(each.value.cidrs, []) : { + cidr = c.cidr + } + ] + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/ip_restrictions.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "IPRST:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/ip_restrictions/outputs.tf b/modules/ip_restrictions/outputs.tf index 7b1f3e2..48999ed 100644 --- a/modules/ip_restrictions/outputs.tf +++ b/modules/ip_restrictions/outputs.tf @@ -1,4 +1,7 @@ output "ip_rule_ids" { description = "Map of IP rule key to dbt Cloud IP restriction rule ID" - value = { for k, r in dbtcloud_ip_restrictions_rule.ip_rules : k => r.id } + value = merge( + { for k, r in dbtcloud_ip_restrictions_rule.ip_rules : k => r.id }, + { for k, r in dbtcloud_ip_restrictions_rule.protected_ip_rules : k => r.id }, + ) } diff --git a/modules/ip_restrictions/variables.tf b/modules/ip_restrictions/variables.tf index 668c845..b893a76 100644 --- a/modules/ip_restrictions/variables.tf +++ b/modules/ip_restrictions/variables.tf @@ -1,5 +1,5 @@ variable "ip_rules_data" { - description = "List of IP restriction rule configurations from YAML ip_restrictions[]" + description = "List of IP restriction rule configurations from YAML ip_restrictions[]. Optional protected: true applies lifecycle.prevent_destroy; optional id for resource_metadata.source_id when provider supports it." type = any default = [] } diff --git a/modules/jobs/main.tf b/modules/jobs/main.tf index 70cb5de..3014098 100644 --- a/modules/jobs/main.tf +++ b/modules/jobs/main.tf @@ -9,37 +9,19 @@ terraform { } locals { - # Flatten all jobs across all projects. - # Jobs can be at project level (job.environment_key) or nested under environments. - # We support both layouts for backward compatibility. - all_jobs_flat = flatten(concat( - # Project-level jobs (new layout: project.jobs[].environment_key) - [ - for p in var.projects : [ - for job in try(p.jobs, []) : { - project_key = try(p.key, p.name) - project_id = var.project_ids[try(p.key, p.name)] - env_key = try(job.environment_key, job.environment) - composite_key = "${try(p.key, p.name)}_${try(job.key, job.name)}" - job_data = job - } - ] - ], - # Environment-nested jobs (legacy v1 layout: project.environments[].jobs[]) - [ - for p in var.projects : [ - for env in try(p.environments, []) : [ - for job in try(env.jobs, []) : { - project_key = try(p.key, p.name) - project_id = var.project_ids[try(p.key, p.name)] - env_key = try(env.key, env.name) - composite_key = "${try(p.key, p.name)}_${try(env.name, env.key)}_${job.name}" - job_data = job - } - ] if try(env.jobs, null) != null - ] + # Project-level jobs only (project.jobs[].environment_key). + all_jobs_flat = flatten([ + for p in var.projects : [ + for job in try(p.jobs, []) : { + project_key = try(p.key, p.name) + project_id = var.project_ids[try(p.key, p.name)] + env_key = job.environment_key + composite_key = "${try(p.key, p.name)}_${try(job.key, job.name)}" + job_key = try(job.key, job.name) + job_data = job + } ] - )) + ]) jobs_map = { for item in local.all_jobs_flat : @@ -69,6 +51,31 @@ locals { ) } + # deployment_type from environments module (SAO / run_compare_changes validation) + env_deployment_type_by_job = { + for k, item in local.jobs_map : + k => try(var.deployment_types["${item.project_key}_${item.env_key}"], null) + } + + # State-aware orchestration: run_compare_changes only when env is staging/production + # and job defers to a different environment (v2/importer parity). + validate_run_compare_changes = { + for k, item in local.jobs_map : + k => ( + try(item.job_data.run_compare_changes, false) == true ? + ( + contains( + ["staging", "production"], + local.env_deployment_type_by_job[k] != null ? local.env_deployment_type_by_job[k] : "" + ) && + ( + try(item.job_data.deferring_environment_key, null) != null && + item.job_data.deferring_environment_key != item.env_key + ) + ) : false + ) + } + # Detect CI/Merge jobs — force_node_selection must be null for these is_ci_or_merge_job = { for k, item in local.jobs_map : @@ -80,10 +87,53 @@ locals { ) } - # force_node_selection: null for CI/Merge, otherwise from YAML + # COMPAT(v1-schema): cost_optimization_features ["state_aware_orchestration"] => force_node_selection false (v2/importer) force_node_selection_effective = { for k, item in local.jobs_map : - k => local.is_ci_or_merge_job[k] ? null : try(item.job_data.force_node_selection, null) + k => ( + local.is_ci_or_merge_job[k] + ? null + : ( + length(coalesce(try(item.job_data.cost_optimization_features, null), [])) > 0 && contains(coalesce(try(item.job_data.cost_optimization_features, null), []), "state_aware_orchestration") + ? false + : try(item.job_data.force_node_selection, null) + ) + ) + } + + # API canonicalizes compare_changes_flags for CI/Merge; normalize to avoid inconsistent results after apply. + compare_changes_flags_effective = { + for k, item in local.jobs_map : + k => ( + local.is_ci_or_merge_job[k] + ? "--select state:modified" + : ( + try(item.job_data.compare_changes_flags, null) == null || + try(item.job_data.compare_changes_flags, null) == false || + lower(trimspace(try(tostring(item.job_data.compare_changes_flags), ""))) == "false" || + trimspace(try(tostring(item.job_data.compare_changes_flags), "")) == "" + ? "--select state:modified" + : trimspace(try(tostring(item.job_data.compare_changes_flags), "--select state:modified")) + ) + ) + } + + # API only allows linting on CI jobs (git_provider_webhook or job_type ci). + run_lint_effective = { + for k, item in local.jobs_map : + k => ( + ( + try(item.job_data.triggers.git_provider_webhook, false) == true || + lower(trimspace(try(tostring(item.job_data.job_type), ""))) == "ci" + ) + ? try(item.job_data.run_lint, false) + : false + ) + } + + errors_on_lint_failure_effective = { + for k, item in local.jobs_map : + k => local.run_lint_effective[k] ? try(item.job_data.errors_on_lint_failure, false) : false } # Schedule mutual exclusivity: cron takes precedence, then interval, then hours @@ -113,7 +163,6 @@ locals { ) } - # Protected/unprotected split protected_jobs_map = { for k, item in local.jobs_map : k => item @@ -140,16 +189,19 @@ resource "dbtcloud_job" "jobs" { execute_steps = each.value.job_data.execute_steps triggers = each.value.job_data.triggers - dbt_version = try(each.value.job_data.dbt_version, null) - description = try(each.value.job_data.description, null) - errors_on_lint_failure = try(each.value.job_data.errors_on_lint_failure, true) - generate_docs = try(each.value.job_data.generate_docs, false) - is_active = try(each.value.job_data.is_active, true) - num_threads = try(each.value.job_data.num_threads, 4) - run_compare_changes = try(each.value.job_data.run_compare_changes, false) - run_generate_sources = try(each.value.job_data.run_generate_sources, false) - run_lint = try(each.value.job_data.run_lint, false) - self_deferring = try(each.value.job_data.self_deferring, null) + dbt_version = try(each.value.job_data.dbt_version, null) + description = try(each.value.job_data.description, null) + errors_on_lint_failure = local.errors_on_lint_failure_effective[each.key] + generate_docs = try(each.value.job_data.generate_docs, false) + is_active = try(each.value.job_data.is_active, true) + num_threads = coalesce(try(each.value.job_data.num_threads, null), 4) + run_compare_changes = local.validate_run_compare_changes[each.key] + compare_changes_flags = local.compare_changes_flags_effective[each.key] + run_generate_sources = try(each.value.job_data.run_generate_sources, false) + run_lint = local.run_lint_effective[each.key] + self_deferring = ( + try(each.value.job_data.deferring_environment_key, null) == null + ) ? try(each.value.job_data.self_deferring, null) : null target_name = try(each.value.job_data.target_name, null) timeout_seconds = try(each.value.job_data.timeout_seconds, 0) triggers_on_draft_pr = try(each.value.job_data.triggers_on_draft_pr, false) @@ -161,6 +213,23 @@ resource "dbtcloud_job" "jobs" { schedule_hours = local.schedule_hours_effective[each.key] schedule_interval = local.schedule_interval_effective[each.key] schedule_type = try(each.value.job_data.schedule_type, null) + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_job (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.job_data.id, null) + # source_identity = "JOB:${each.value.project_key}:${each.value.job_key}" + # source_key = each.value.job_key + # source_project_key = each.value.project_key + # source_name = each.value.job_data.name + # } + + lifecycle { + ignore_changes = [ + job_completion_trigger_condition, + job_type, + ] + } } ############################################# @@ -176,16 +245,19 @@ resource "dbtcloud_job" "protected_jobs" { execute_steps = each.value.job_data.execute_steps triggers = each.value.job_data.triggers - dbt_version = try(each.value.job_data.dbt_version, null) - description = try(each.value.job_data.description, null) - errors_on_lint_failure = try(each.value.job_data.errors_on_lint_failure, true) - generate_docs = try(each.value.job_data.generate_docs, false) - is_active = try(each.value.job_data.is_active, true) - num_threads = try(each.value.job_data.num_threads, 4) - run_compare_changes = try(each.value.job_data.run_compare_changes, false) - run_generate_sources = try(each.value.job_data.run_generate_sources, false) - run_lint = try(each.value.job_data.run_lint, false) - self_deferring = try(each.value.job_data.self_deferring, null) + dbt_version = try(each.value.job_data.dbt_version, null) + description = try(each.value.job_data.description, null) + errors_on_lint_failure = local.errors_on_lint_failure_effective[each.key] + generate_docs = try(each.value.job_data.generate_docs, false) + is_active = try(each.value.job_data.is_active, true) + num_threads = coalesce(try(each.value.job_data.num_threads, null), 4) + run_compare_changes = local.validate_run_compare_changes[each.key] + compare_changes_flags = local.compare_changes_flags_effective[each.key] + run_generate_sources = try(each.value.job_data.run_generate_sources, false) + run_lint = local.run_lint_effective[each.key] + self_deferring = ( + try(each.value.job_data.deferring_environment_key, null) == null + ) ? try(each.value.job_data.self_deferring, null) : null target_name = try(each.value.job_data.target_name, null) timeout_seconds = try(each.value.job_data.timeout_seconds, 0) triggers_on_draft_pr = try(each.value.job_data.triggers_on_draft_pr, false) @@ -198,7 +270,21 @@ resource "dbtcloud_job" "protected_jobs" { schedule_interval = local.schedule_interval_effective[each.key] schedule_type = try(each.value.job_data.schedule_type, null) + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_job (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.job_data.id, null) + # source_identity = "JOB:${each.value.project_key}:${each.value.job_key}" + # source_key = each.value.job_key + # source_project_key = each.value.project_key + # source_name = each.value.job_data.name + # } + lifecycle { prevent_destroy = true + ignore_changes = [ + job_completion_trigger_condition, + job_type, + ] } } diff --git a/modules/jobs/tests/unit.tftest.hcl b/modules/jobs/tests/unit.tftest.hcl index ecdf349..ebc7a27 100644 --- a/modules/jobs/tests/unit.tftest.hcl +++ b/modules/jobs/tests/unit.tftest.hcl @@ -1,5 +1,5 @@ # Unit tests for modules/jobs -# Validates dual layout support, composite key construction, environment ID +# Validates project.jobs[] layout, composite key construction, environment ID # resolution, CI job detection, schedule mutual exclusivity, and protected jobs. # Run from modules/jobs/: terraform test @@ -85,52 +85,6 @@ run "project_level_job_environment_id_resolved" { } } -# ── Legacy nested job layout ────────────────────────────────────────────────── - -run "legacy_nested_job_created" { - command = plan - - variables { - projects = [ - { - key = "analytics" - name = "Analytics" - environments = [ - { - name = "Production" - key = "prod" - jobs = [ - { - name = "Legacy Job" - execute_steps = ["dbt run"] - triggers = { - schedule = false - github_webhook = false - git_provider_webhook = false - on_merge = false - } - } - ] - } - ] - } - ] - project_ids = { analytics = "1001" } - environment_ids = { analytics_prod = "2001" } - } - - assert { - condition = length(dbtcloud_job.jobs) == 1 - error_message = "Expected one job from legacy nested layout" - } - - assert { - # Legacy layout uses env.name (not env.key): "${project_key}_${env.name}_${job.name}" - condition = contains(keys(dbtcloud_job.jobs), "analytics_Production_Legacy Job") - error_message = "Legacy layout composite key should be project_envname_jobname" - } -} - # ── CI job detection and force_node_selection ───────────────────────────────── run "ci_job_github_webhook_clears_force_node_selection" { @@ -292,6 +246,116 @@ run "protected_job_routed_to_protected_resource" { # ── Deferring environment resolution ───────────────────────────────────────── +run "cost_optimization_state_aware_sets_force_node_false" { + command = apply + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + jobs = [ + { + name = "SAO Run" + key = "sao_run" + environment_key = "prod" + execute_steps = ["dbt build"] + force_node_selection = true + cost_optimization_features = ["state_aware_orchestration"] + triggers = { + schedule = false + github_webhook = false + git_provider_webhook = false + on_merge = false + } + } + ] + } + ] + project_ids = { analytics = "1001" } + environment_ids = { analytics_prod = "2001" } + } + + assert { + condition = dbtcloud_job.jobs["analytics_sao_run"].force_node_selection == false + error_message = "state_aware_orchestration in cost_optimization_features should set force_node_selection to false" + } +} + +run "run_compare_changes_true_when_staging_and_cross_env_defer" { + command = apply + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + jobs = [ + { + name = "Compare Run" + key = "compare_run" + environment_key = "prod" + deferring_environment_key = "staging" + run_compare_changes = true + execute_steps = ["dbt build"] + triggers = { + schedule = false + github_webhook = false + git_provider_webhook = false + on_merge = false + } + } + ] + } + ] + project_ids = { analytics = "1001" } + environment_ids = { analytics_prod = "2001", analytics_staging = "2002" } + deployment_types = { analytics_prod = "staging" } + } + + assert { + condition = dbtcloud_job.jobs["analytics_compare_run"].run_compare_changes == true + error_message = "run_compare_changes should be true when job env is staging/production and deferral is a different environment" + } +} + +run "run_compare_changes_false_when_not_eligible" { + command = apply + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + jobs = [ + { + name = "No Compare" + key = "no_compare" + environment_key = "prod" + deferring_environment_key = "staging" + run_compare_changes = true + execute_steps = ["dbt build"] + triggers = { + schedule = false + github_webhook = false + git_provider_webhook = false + on_merge = false + } + } + ] + } + ] + project_ids = { analytics = "1001" } + environment_ids = { analytics_prod = "2001", analytics_staging = "2002" } + deployment_types = { analytics_prod = "other" } + } + + assert { + condition = dbtcloud_job.jobs["analytics_no_compare"].run_compare_changes == false + error_message = "run_compare_changes should be false when deployment_type is not staging or production" + } +} + run "deferring_environment_id_resolved" { command = apply diff --git a/modules/jobs/variables.tf b/modules/jobs/variables.tf index f17774c..3f31815 100644 --- a/modules/jobs/variables.tf +++ b/modules/jobs/variables.tf @@ -1,5 +1,5 @@ variable "projects" { - description = "List of project configurations. Jobs may be at project.jobs[] (with environment_key) or project.environments[].jobs[] (legacy)." + description = "List of project configurations. Jobs are defined only on project.jobs[] with environment_key." type = any } @@ -12,3 +12,9 @@ variable "environment_ids" { description = "Map of composite key (project_key_env_key) to dbt Cloud environment ID (from environments module)" type = map(string) } + +variable "deployment_types" { + description = "Map of project_key_env_key to environment deployment_type (from module.environments.deployment_types). Used to gate run_compare_changes (staging/production + cross-env deferral only)." + type = map(any) + default = {} +} diff --git a/modules/lineage_integrations/.terraform.lock.hcl b/modules/lineage_integrations/.terraform.lock.hcl new file mode 100644 index 0000000..8dc8cba --- /dev/null +++ b/modules/lineage_integrations/.terraform.lock.hcl @@ -0,0 +1,27 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/dbt-labs/dbtcloud" { + version = "1.8.2" + constraints = "~> 1.8" + hashes = [ + "h1:3Hw1in/qcHdAp2BRhEaFRr/ez9lFjBv3V5WIUxHXcSs=", + "h1:ASwPE1I3U7cufOLFvGV+yHmV4/24y2Jrmt8V3Pn9lEY=", + "h1:BIVjUC8FxluKSvwm2jvySD0AvSkFPKwGn/SHez082Oc=", + "h1:IpC3g4srpeI3iJ5GL3GcxSMO/+XHBktS/zumEm/8xbc=", + "zh:01df4a2592c7d66152aa18bae2fb6abcca205f2c0d75486a67e3c8307ea70af2", + "zh:020e8c80eb8973ac0571b8cb2c990d76c82d2032f590ffaec418fc6728a48594", + "zh:0c30dd1d7ac516efe921362143c0372b2dedab59950662c08513c8c1e0430ee4", + "zh:192bc343bfe193d79b4f2e5d5340a44230e169670d190c7034c7db67fd97b3e7", + "zh:24b8175f22a97844000facea8742bd2f0ca5eedb214c900e6b50153308874e86", + "zh:26ef5f40358401f7fadae72b16782d0a8a8761af325aac17b5507234148c19f9", + "zh:7b9cd9637bf607854dfdeceab06dd361686f54864050cfb770d4214a2d0265db", + "zh:8df586c709b83ca62056fd3d9ff5e9d73833acdcb3194d7e5bbc0cde98f87ed9", + "zh:8e561719c50ae83caefa0dd8ebf3cf4907a512d1be90ecb80eb5db6593f9e9b2", + "zh:9071202c4b459dd28fa792a62e229815bae41d637428fa66dca5eb7fa0fc5801", + "zh:926cca7812b272ab976ce49c0858de78bd5c4c6fcffe16cb2d33cdce63522bc2", + "zh:ae927fc0b059c800b0342b6c6f7217dd191349334a6d6ed1cad578f13f64722c", + "zh:b0e484d5c2dc79b1f573aa3bdac4150a2b4094b6fd9c92033b855e7b8f07045c", + "zh:f96b0ce96fe679653e5dd3820f15e7916855ce8deb832f6b92befedc2387f1f6", + ] +} diff --git a/modules/lineage_integrations/main.tf b/modules/lineage_integrations/main.tf index 3a76c24..80964af 100644 --- a/modules/lineage_integrations/main.tf +++ b/modules/lineage_integrations/main.tf @@ -32,10 +32,15 @@ resource "dbtcloud_lineage_integration" "integrations" { project_id = each.value.project_id host = each.value.li_data.host - site_id = each.value.li_data.site_id - token_name = each.value.li_data.token_name - token = try( - lookup(var.lineage_tokens, each.key, null), - each.value.li_data.token + # Provider marks site_id / token_name required; use empty string when omitted (e.g. non-Tableau integrations). + site_id = coalesce(try(each.value.li_data.site_id, null), "") + token_name = coalesce(try(each.value.li_data.token_name, null), "") + token = coalesce( + try(lookup(var.lineage_tokens, each.key, null), null), + try(each.value.li_data.token, null), + "" ) } + +# Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_lineage_integration (terraform providers schema). +# v2/importer: resource_metadata.source_identity LNGI::; source_id from lineage_integrations[].id. diff --git a/modules/lineage_integrations/tests/unit.tftest.hcl b/modules/lineage_integrations/tests/unit.tftest.hcl new file mode 100644 index 0000000..d6c9e51 --- /dev/null +++ b/modules/lineage_integrations/tests/unit.tftest.hcl @@ -0,0 +1,64 @@ +# Unit tests for modules/lineage_integrations — run from modules/lineage_integrations/: terraform test + +mock_provider "dbtcloud" {} + +run "lineage_token_from_var_map" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + lineage_integrations = [ + { + name = "Tableau" + key = "tableau_prod" + host = "https://tableau.example.com" + site_id = "site" + token_name = "pat" + } + ] + } + ] + project_ids = { analytics = "1001" } + lineage_tokens = { + "analytics_tableau_prod" = "secret-token-value" + } + } + + assert { + condition = dbtcloud_lineage_integration.integrations["analytics_tableau_prod"].host == "https://tableau.example.com" + error_message = "lineage integration should be created for composite key analytics_tableau_prod" + } +} + +run "lineage_inline_token_fallback" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + lineage_integrations = [ + { + name = "Tableau" + key = "tableau_prod" + host = "https://tableau.example.com" + site_id = "site" + token_name = "pat" + token = "inline-fallback" + } + ] + } + ] + project_ids = { analytics = "1001" } + lineage_tokens = {} + } + + assert { + condition = dbtcloud_lineage_integration.integrations["analytics_tableau_prod"].token == "inline-fallback" + error_message = "inline token on integration should be used when lineage_tokens omits the key" + } +} diff --git a/modules/notifications/main.tf b/modules/notifications/main.tf index 6ea8e5f..3defce4 100644 --- a/modules/notifications/main.tf +++ b/modules/notifications/main.tf @@ -11,12 +11,45 @@ terraform { locals { notifications_map = { for n in var.notifications_data : - try(n.key, n.name) => n + n.key => n + } + + unprotected_notifications_map = { + for k, n in local.notifications_map : + k => n if try(n.protected, false) != true + } + + protected_notifications_map = { + for k, n in local.notifications_map : + k => n if try(n.protected, false) == true } } resource "dbtcloud_notification" "notifications" { - for_each = local.notifications_map + for_each = local.unprotected_notifications_map + + user_id = try(each.value.user_id, null) + on_cancel = try(each.value.on_cancel, []) + on_failure = try(each.value.on_failure, []) + on_success = try(each.value.on_success, []) + on_warning = try(each.value.on_warning, []) + notification_type = try(each.value.notification_type, 1) + slack_channel_id = try(each.value.slack_channel_id, null) + slack_channel_name = try(each.value.slack_channel_name, null) + external_email = try(each.value.external_email, null) + state = try(each.value.state, 1) + + # resource_metadata: pending official dbtcloud provider support (importer pattern: NTO:${key}). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "NTO:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } +} + +resource "dbtcloud_notification" "protected_notifications" { + for_each = local.protected_notifications_map user_id = try(each.value.user_id, null) on_cancel = try(each.value.on_cancel, []) @@ -27,4 +60,17 @@ resource "dbtcloud_notification" "notifications" { slack_channel_id = try(each.value.slack_channel_id, null) slack_channel_name = try(each.value.slack_channel_name, null) external_email = try(each.value.external_email, null) + state = try(each.value.state, 1) + + # resource_metadata: pending official dbtcloud provider support (importer pattern: NTO:${key}). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "NTO:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/notifications/outputs.tf b/modules/notifications/outputs.tf index 2077bef..ec936b0 100644 --- a/modules/notifications/outputs.tf +++ b/modules/notifications/outputs.tf @@ -1,4 +1,7 @@ output "notification_ids" { description = "Map of notification key to dbt Cloud notification ID" - value = { for k, n in dbtcloud_notification.notifications : k => n.id } + value = merge( + { for k, n in dbtcloud_notification.notifications : k => n.id }, + { for k, n in dbtcloud_notification.protected_notifications : k => n.id }, + ) } diff --git a/modules/oauth_configurations/main.tf b/modules/oauth_configurations/main.tf index a33403e..5e9ac12 100644 --- a/modules/oauth_configurations/main.tf +++ b/modules/oauth_configurations/main.tf @@ -8,24 +8,72 @@ terraform { } } +############################################# +# OAuth configurations (account-level) +# client_secret is sensitive — sourced from var.oauth_client_secrets (or optional YAML). +############################################# + locals { - oauth_map = { + oauth_configurations_map = { for o in var.oauth_data : - try(o.key, o.name) => o + o.key => o + } + unprotected_oauth_map = { + for key, oauth in local.oauth_configurations_map : + key => oauth if !try(oauth.protected, false) + } + protected_oauth_map = { + for key, oauth in local.oauth_configurations_map : + key => oauth if try(oauth.protected, false) } } resource "dbtcloud_oauth_configuration" "oauth_configurations" { - for_each = local.oauth_map - - name = each.value.name - type = each.value.type - authorize_url = each.value.authorize_url - token_url = each.value.token_url - redirect_uri = each.value.redirect_uri - client_id = each.value.client_id - client_secret = try( - lookup(var.oauth_client_secrets, each.key, null), - each.value.client_secret - ) + for_each = local.unprotected_oauth_map + + name = each.value.name + type = try(each.value.type, null) + client_id = try(each.value.client_id, null) + # COMPAT(v1-schema): secrets from root map (preferred) or inline YAML — v2 schema may narrow to one path. + client_secret = lookup(var.oauth_client_secrets, each.key, try(each.value.client_secret, "")) + authorize_url = try(each.value.authorize_url, null) + token_url = try(each.value.token_url, null) + redirect_uri = try(each.value.redirect_uri, null) + + application_id_uri = try(each.value.application_id_uri, null) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/oauth_configurations.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "OAUTH:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } +} + +resource "dbtcloud_oauth_configuration" "protected_oauth_configurations" { + for_each = local.protected_oauth_map + + name = each.value.name + type = try(each.value.type, null) + client_id = try(each.value.client_id, null) + # COMPAT(v1-schema): secrets from root map (preferred) or inline YAML — v2 schema may narrow to one path. + client_secret = lookup(var.oauth_client_secrets, each.key, try(each.value.client_secret, "")) + authorize_url = try(each.value.authorize_url, null) + token_url = try(each.value.token_url, null) + redirect_uri = try(each.value.redirect_uri, null) + + application_id_uri = try(each.value.application_id_uri, null) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/oauth_configurations.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "OAUTH:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/oauth_configurations/outputs.tf b/modules/oauth_configurations/outputs.tf index 8f9122b..1f34d88 100644 --- a/modules/oauth_configurations/outputs.tf +++ b/modules/oauth_configurations/outputs.tf @@ -1,4 +1,7 @@ output "oauth_configuration_ids" { description = "Map of OAuth configuration key to dbt Cloud OAuth configuration ID" - value = { for k, o in dbtcloud_oauth_configuration.oauth_configurations : k => o.id } + value = merge( + { for k, o in dbtcloud_oauth_configuration.oauth_configurations : k => o.id }, + { for k, o in dbtcloud_oauth_configuration.protected_oauth_configurations : k => o.id }, + ) } diff --git a/modules/oauth_configurations/variables.tf b/modules/oauth_configurations/variables.tf index a590940..f2dd602 100644 --- a/modules/oauth_configurations/variables.tf +++ b/modules/oauth_configurations/variables.tf @@ -1,5 +1,5 @@ variable "oauth_data" { - description = "List of OAuth configuration entries from YAML oauth_configurations[]" + description = "List of OAuth configuration entries from YAML oauth_configurations[] (optional: id for resource_metadata.source_id when provider supports it, protected, application_id_uri for Entra)" type = any default = [] } diff --git a/modules/profiles/main.tf b/modules/profiles/main.tf index 63490ef..80e9ad2 100644 --- a/modules/profiles/main.tf +++ b/modules/profiles/main.tf @@ -21,27 +21,94 @@ locals { ] ]) - profiles_map = { + protected_profiles_map = { for item in local.all_profiles : item.composite_key => item + if try(item.profile_data.protected, false) == true + } + + unprotected_profiles_map = { + for item in local.all_profiles : + item.composite_key => item + if try(item.profile_data.protected, false) != true + } + + # Global connection key, LOOKUP:… (via root global_connection_ids_effective), or numeric id. + resolve_profile_connection_id = { + for item in local.all_profiles : + item.composite_key => ( + try(item.profile_data.connection_key, null) != null ? + lookup(var.global_connection_ids, tostring(item.profile_data.connection_key), null) != null ? + lookup(var.global_connection_ids, tostring(item.profile_data.connection_key), null) : + try(tonumber(item.profile_data.connection_key), null) : + null + ) + } + + resolve_profile_credential_id = { + for item in local.all_profiles : + item.composite_key => try(coalesce( + lookup(var.credential_ids, item.composite_key, null), + try(item.profile_data.credentials_key, null) != null && try(item.profile_data.credentials_key, "") != "" ? + lookup(var.credential_ids, "${item.project_key}_${item.profile_data.credentials_key}", null) : null, + try(item.profile_data.credentials_id, null) != null ? + lookup(var.credential_ids_by_source_id, tostring(item.profile_data.credentials_id), null) : null, + try(tonumber(item.profile_data.credentials_id), null), + ), null) + } + + resolve_profile_extended_attributes_id = { + for item in local.all_profiles : + item.composite_key => try(coalesce( + try(item.profile_data.extended_attributes_key, null) != null && try(item.profile_data.extended_attributes_key, "") != "" ? + lookup(var.extended_attribute_ids, "${item.project_key}_${item.profile_data.extended_attributes_key}", null) : null, + try(item.profile_data.extended_attributes_id, null) != null ? + lookup(var.extended_attribute_ids_by_source_id, tostring(item.profile_data.extended_attributes_id), null) : null, + try(tonumber(item.profile_data.extended_attributes_id), null) + ), null) } } resource "dbtcloud_profile" "profiles" { - for_each = local.profiles_map - - project_id = each.value.project_id - key = each.value.profile_key - connection_id = try( - lookup(var.global_connection_ids, tostring(each.value.profile_data.connection_key), null), - try(tonumber(each.value.profile_data.connection_id), null) - ) - credentials_id = try( - lookup(var.credential_ids, "${each.value.project_key}_${each.value.profile_data.credential_key}", null), - try(each.value.profile_data.credentials_id, null) - ) - extended_attributes_id = try( - lookup(var.extended_attribute_ids, "${each.value.project_key}_${each.value.profile_data.extended_attributes_key}", null), - null - ) + for_each = local.unprotected_profiles_map + + project_id = each.value.project_id + key = each.value.profile_key + connection_id = local.resolve_profile_connection_id[each.key] + credentials_id = local.resolve_profile_credential_id[each.key] + extended_attributes_id = local.resolve_profile_extended_attributes_id[each.key] + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_profile (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.profile_data.id, null) + # source_identity = "PRF:${each.value.project_key}:${each.value.profile_key}" + # source_key = each.value.profile_key + # source_name = each.value.profile_key + # source_project_key = each.value.project_key + # } +} + +resource "dbtcloud_profile" "protected_profiles" { + for_each = local.protected_profiles_map + + project_id = each.value.project_id + key = each.value.profile_key + connection_id = local.resolve_profile_connection_id[each.key] + credentials_id = local.resolve_profile_credential_id[each.key] + extended_attributes_id = local.resolve_profile_extended_attributes_id[each.key] + + # Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_profile (terraform providers schema). + # resource_metadata = { + # source_project_id = null # v2 importer: lookup(local.source_project_ids_by_key, each.value.project_key, null) + # source_id = try(each.value.profile_data.id, null) + # source_identity = "PRF:${each.value.project_key}:${each.value.profile_key}" + # source_key = each.value.profile_key + # source_name = each.value.profile_key + # source_project_key = each.value.project_key + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/profiles/outputs.tf b/modules/profiles/outputs.tf index d12b063..321e891 100644 --- a/modules/profiles/outputs.tf +++ b/modules/profiles/outputs.tf @@ -1,4 +1,7 @@ output "profile_ids" { - description = "Map of composite key (project_key_profile_key) to dbt Cloud profile ID" - value = { for k, p in dbtcloud_profile.profiles : k => p.id } + description = "Map of composite key (project_key_profile_key) to dbt Cloud profile_id (numeric API id; use for environment primary_profile_id)" + value = merge( + { for k, p in dbtcloud_profile.profiles : k => p.profile_id }, + { for k, p in dbtcloud_profile.protected_profiles : k => p.profile_id }, + ) } diff --git a/modules/profiles/variables.tf b/modules/profiles/variables.tf index 0ff65c2..3118940 100644 --- a/modules/profiles/variables.tf +++ b/modules/profiles/variables.tf @@ -15,13 +15,25 @@ variable "global_connection_ids" { } variable "credential_ids" { - description = "Map of composite key (project_key_env_key) to credential ID (from credentials module)" + description = "Map of composite key (project_key_env_key or project_key_profile_key) to credential ID (from credentials module)" type = map(string) default = {} } -variable "extended_attribute_ids" { - description = "Map of composite key (project_key_ea_key) to extended_attributes ID (from extended_attributes module)" +variable "credential_ids_by_source_id" { + description = "Maps legacy YAML credential.id to Terraform-managed credential_id (from credentials module)." type = map(string) default = {} } + +variable "extended_attribute_ids" { + description = "Map of composite key (project_key_ea_key) to dbt Cloud extended_attributes_id (numeric; from extended_attributes module)." + type = map(number) + default = {} +} + +variable "extended_attribute_ids_by_source_id" { + description = "Maps legacy YAML extended_attributes[].id to Terraform-managed extended_attributes_id (from extended_attributes module)." + type = map(number) + default = {} +} diff --git a/modules/project/main.tf b/modules/project/main.tf index 22d89fd..c65def0 100644 --- a/modules/project/main.tf +++ b/modules/project/main.tf @@ -37,6 +37,15 @@ resource "dbtcloud_project" "projects" { for_each = local.unprotected_projects_map name = "${var.target_name}${each.value.name}" + + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_project (v2 parity). + # resource_metadata = { + # source_project_id = try(each.value.id, null) + # source_id = try(each.value.id, null) + # source_identity = "PRJ:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } } ############################################# @@ -48,6 +57,15 @@ resource "dbtcloud_project" "protected_projects" { name = "${var.target_name}${each.value.name}" + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_project (v2 parity). + # resource_metadata = { + # source_project_id = try(each.value.id, null) + # source_id = try(each.value.id, null) + # source_identity = "PRJ:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } + lifecycle { prevent_destroy = true } diff --git a/modules/project_artefacts/.terraform.lock.hcl b/modules/project_artefacts/.terraform.lock.hcl new file mode 100644 index 0000000..8dc8cba --- /dev/null +++ b/modules/project_artefacts/.terraform.lock.hcl @@ -0,0 +1,27 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/dbt-labs/dbtcloud" { + version = "1.8.2" + constraints = "~> 1.8" + hashes = [ + "h1:3Hw1in/qcHdAp2BRhEaFRr/ez9lFjBv3V5WIUxHXcSs=", + "h1:ASwPE1I3U7cufOLFvGV+yHmV4/24y2Jrmt8V3Pn9lEY=", + "h1:BIVjUC8FxluKSvwm2jvySD0AvSkFPKwGn/SHez082Oc=", + "h1:IpC3g4srpeI3iJ5GL3GcxSMO/+XHBktS/zumEm/8xbc=", + "zh:01df4a2592c7d66152aa18bae2fb6abcca205f2c0d75486a67e3c8307ea70af2", + "zh:020e8c80eb8973ac0571b8cb2c990d76c82d2032f590ffaec418fc6728a48594", + "zh:0c30dd1d7ac516efe921362143c0372b2dedab59950662c08513c8c1e0430ee4", + "zh:192bc343bfe193d79b4f2e5d5340a44230e169670d190c7034c7db67fd97b3e7", + "zh:24b8175f22a97844000facea8742bd2f0ca5eedb214c900e6b50153308874e86", + "zh:26ef5f40358401f7fadae72b16782d0a8a8761af325aac17b5507234148c19f9", + "zh:7b9cd9637bf607854dfdeceab06dd361686f54864050cfb770d4214a2d0265db", + "zh:8df586c709b83ca62056fd3d9ff5e9d73833acdcb3194d7e5bbc0cde98f87ed9", + "zh:8e561719c50ae83caefa0dd8ebf3cf4907a512d1be90ecb80eb5db6593f9e9b2", + "zh:9071202c4b459dd28fa792a62e229815bae41d637428fa66dca5eb7fa0fc5801", + "zh:926cca7812b272ab976ce49c0858de78bd5c4c6fcffe16cb2d33cdce63522bc2", + "zh:ae927fc0b059c800b0342b6c6f7217dd191349334a6d6ed1cad578f13f64722c", + "zh:b0e484d5c2dc79b1f573aa3bdac4150a2b4094b6fd9c92033b855e7b8f07045c", + "zh:f96b0ce96fe679653e5dd3820f15e7916855ce8deb832f6b92befedc2387f1f6", + ] +} diff --git a/modules/project_artefacts/main.tf b/modules/project_artefacts/main.tf index bf38692..4c4c330 100644 --- a/modules/project_artefacts/main.tf +++ b/modules/project_artefacts/main.tf @@ -9,11 +9,22 @@ terraform { } locals { - # Only create artefacts for projects that have an artefacts block + artefacts_rows = [ + for p in var.projects : { + project_key = try(p.key, p.name) + docs_job_key = try(p.project_artefacts.docs_job_key, null) + freshness_job_key = try(p.project_artefacts.freshness_job_key, null) + has_block = try(p.project_artefacts, null) != null + } + ] + artefacts_map = { - for p in var.projects : - try(p.key, p.name) => p - if try(p.artefacts, null) != null + for row in local.artefacts_rows : + row.project_key => row + if row.has_block && ( + (try(row.docs_job_key, null) != null && try(tostring(row.docs_job_key), "") != "") || + (try(row.freshness_job_key, null) != null && try(tostring(row.freshness_job_key), "") != "") + ) } } @@ -22,13 +33,17 @@ resource "dbtcloud_project_artefacts" "artefacts" { project_id = var.project_ids[each.key] - docs_job_id = try( - lookup(var.job_ids, "${each.key}_${each.value.artefacts.docs_job}", null), - null + docs_job_id = ( + each.value.docs_job_key != null + ? try(lookup(var.job_ids, "${each.key}_${each.value.docs_job_key}", null), null) + : null ) - freshness_job_id = try( - lookup(var.job_ids, "${each.key}_${each.value.artefacts.freshness_job}", null), - null + freshness_job_id = ( + each.value.freshness_job_key != null + ? try(lookup(var.job_ids, "${each.key}_${each.value.freshness_job_key}", null), null) + : null ) } + +# Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_project_artefacts (terraform providers schema). diff --git a/modules/project_artefacts/tests/unit.tftest.hcl b/modules/project_artefacts/tests/unit.tftest.hcl new file mode 100644 index 0000000..48361d1 --- /dev/null +++ b/modules/project_artefacts/tests/unit.tftest.hcl @@ -0,0 +1,28 @@ +# Unit tests for modules/project_artefacts — run from modules/project_artefacts/: terraform test + +mock_provider "dbtcloud" {} + +run "v2_project_artefacts_docs_job_key" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + project_artefacts = { + docs_job_key = "daily" + } + } + ] + project_ids = { analytics = "1001" } + job_ids = { + "analytics_daily" = "5001" + } + } + + assert { + condition = tostring(dbtcloud_project_artefacts.artefacts["analytics"].docs_job_id) == "5001" + error_message = "project_artefacts.docs_job_key should resolve job id via project_key_job_key" + } +} diff --git a/modules/project_artefacts/variables.tf b/modules/project_artefacts/variables.tf index 268eeb6..1fba367 100644 --- a/modules/project_artefacts/variables.tf +++ b/modules/project_artefacts/variables.tf @@ -1,5 +1,5 @@ variable "projects" { - description = "List of project configurations. Each project may have an 'artefacts' block with docs_job and freshness_job keys." + description = "Project configs. Optional project_artefacts block (docs_job_key, freshness_job_key); see modules/project_artefacts." type = any } diff --git a/modules/project_repository/main.tf b/modules/project_repository/main.tf index d8d58cd..a0b47b4 100644 --- a/modules/project_repository/main.tf +++ b/modules/project_repository/main.tf @@ -8,9 +8,56 @@ terraform { } } +locals { + protected_repository_key_set = toset(var.protected_repository_keys) + + unprotected_project_repository_ids = { + for k, rid in var.repository_ids : k => rid + if !contains(local.protected_repository_key_set, k) + } + + protected_project_repository_ids = { + for k, rid in var.repository_ids : k => rid + if contains(local.protected_repository_key_set, k) + } +} + resource "dbtcloud_project_repository" "project_repositories" { - for_each = var.repository_ids + for_each = local.unprotected_project_repository_ids project_id = var.project_ids[each.key] repository_id = each.value + + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_project_repository (v2 parity). + # v2 used try(project.id) / try(repo.id) from YAML for source_*_id; wire when uncommenting. + # resource_metadata = { + # source_project_id = null + # source_id = null + # source_identity = "PREP:${each.key}" + # source_key = each.key + # source_project_key = each.key + # source_name = each.key + # } +} + +resource "dbtcloud_project_repository" "protected_project_repositories" { + for_each = local.protected_project_repository_ids + + project_id = var.project_ids[each.key] + repository_id = each.value + + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_project_repository (v2 parity). + # v2 used try(project.id) / try(repo.id) from YAML for source_*_id; wire when uncommenting. + # resource_metadata = { + # source_project_id = null + # source_id = null + # source_identity = "PREP:${each.key}" + # source_key = each.key + # source_project_key = each.key + # source_name = each.key + # } + + lifecycle { + prevent_destroy = true + } } diff --git a/modules/project_repository/outputs.tf b/modules/project_repository/outputs.tf index c34cf5f..d2c46e4 100644 --- a/modules/project_repository/outputs.tf +++ b/modules/project_repository/outputs.tf @@ -1,4 +1,7 @@ output "project_repository_ids" { description = "Map of project key to project_repository resource ID" - value = { for k, pr in dbtcloud_project_repository.project_repositories : k => pr.id } + value = merge( + { for k, pr in dbtcloud_project_repository.project_repositories : k => pr.id }, + { for k, pr in dbtcloud_project_repository.protected_project_repositories : k => pr.id } + ) } diff --git a/modules/project_repository/variables.tf b/modules/project_repository/variables.tf index 5a33508..b92179c 100644 --- a/modules/project_repository/variables.tf +++ b/modules/project_repository/variables.tf @@ -7,3 +7,9 @@ variable "repository_ids" { description = "Map of project key to repository_id (the integer ID used for project_repository links, not the resource ID)" type = map(string) } + +variable "protected_repository_keys" { + description = "Project keys that use protected dbtcloud_repository resources; matching links get lifecycle.prevent_destroy (v2 projects.tf parity)." + type = list(string) + default = [] +} diff --git a/modules/repository/main.tf b/modules/repository/main.tf index 086a6a5..ea88a01 100644 --- a/modules/repository/main.tf +++ b/modules/repository/main.tf @@ -22,6 +22,17 @@ locals { try(p.key, p.name) => p } + privatelink_endpoints_map = { + for ple in var.privatelink_endpoints : + ple.key => ple + } + + # COMPAT(v1-schema): resolve PrivateLink via privatelink_endpoints[] + account data when only private_link_endpoint_key is set (v2 projects.tf parity). + needs_privatelink_data = length([ + for k, repo in local.repos_map : k + if try(repo.private_link_endpoint_key, null) != null && try(repo.private_link_endpoint_id, null) == null + ]) > 0 + # Auto-detect provider from remote_url detected_provider = { for k, repo in local.repos_map : @@ -34,27 +45,56 @@ locals { ) } - # Effective git clone strategy per repo (with fallbacks) - effective_git_clone_strategy = { + # GitHub owner segment from remote_url (github.com only), for installation lookup + # With a single capture group, regexall returns one element per match; inner list is [ capture ] (see terraform console). + github_owner_from_url = { + for k, repo in local.repos_map : + k => try(regexall("github\\.com[/:]([^/]+)/", trimspace(try(repo.remote_url, "")))[0][0], null) + } + + installation_id_from_discovery = { + for k, repo in local.repos_map : + k => lookup( + var.github_installation_by_owner, + try(local.github_owner_from_url[k], null) != null && trimspace(tostring(local.github_owner_from_url[k])) != "" ? lower(local.github_owner_from_url[k]) : "", + null + ) + } + + # YAML github_installation_id > owner match from discovery map > account fallback installation + resolved_github_installation_id = { for k, repo in local.repos_map : k => ( - # Azure DevOps: downgrade if required IDs are missing + try(repo.github_installation_id, null) != null ? try(repo.github_installation_id, null) : ( + local.installation_id_from_discovery[k] != null ? + local.installation_id_from_discovery[k] : + try(var.github_installation_fallback_id, null) + ) + ) + } + + # Effective git clone strategy per repo (with fallbacks). + # nonsensitive() keeps the strategy string from inheriting sensitivity from var.dbt_pat (v2 projects.tf). + effective_git_clone_strategy = { + for k, repo in local.repos_map : + k => nonsensitive( try(repo.git_clone_strategy, "") == "azure_active_directory_app" && ( trimspace(tostring(try(repo.azure_active_directory_project_id, ""))) == "" || trimspace(tostring(try(repo.azure_active_directory_repository_id, ""))) == "" ) ? "deploy_key" : - # GitLab deploy_token: downgrade to deploy_key unless explicitly enabled try(repo.git_clone_strategy, "") == "deploy_token" ? ( var.enable_gitlab_deploy_token ? "deploy_token" : "deploy_key" ) : - # GitHub App: use if installation ID available, else deploy_key try(repo.git_clone_strategy, "") == "github_app" ? ( - try(repo.github_installation_id, null) != null || var.dbt_pat != null ? "github_app" : "deploy_key" + try(repo.github_installation_id, null) != null || + local.resolved_github_installation_id[k] != null || + var.dbt_pat != null + ? "github_app" : "deploy_key" ) : - # Explicit strategy set to something other than the above try(repo.git_clone_strategy, null) != null ? try(repo.git_clone_strategy, "deploy_key") : - # Auto-detect defaults - local.detected_provider[k] == "github" ? "github_app" : + local.detected_provider[k] == "github" ? ( + local.resolved_github_installation_id[k] != null || var.dbt_pat != null ? "github_app" : "deploy_key" + ) : local.detected_provider[k] == "gitlab" ? "deploy_token" : local.detected_provider[k] == "azure_devops" ? "azure_active_directory_app" : "deploy_key" @@ -109,6 +149,27 @@ locals { for k, repo in local.repos_map : k => repo if local.repo_protected[k] != true } + + private_link_endpoint_id_by_repo_key = { + for k, repo in local.repos_map : k => ( + try(repo.private_link_endpoint_id, null) != null && trimspace(tostring(try(repo.private_link_endpoint_id, ""))) != "" ? + try(repo.private_link_endpoint_id, null) : + ( + length(data.dbtcloud_privatelink_endpoints.all) > 0 && + try(repo.private_link_endpoint_key, null) != null && + lookup(local.privatelink_endpoints_map, repo.private_link_endpoint_key, null) != null + ) ? data.dbtcloud_privatelink_endpoints.all[0].endpoints[ + index( + [for ep in data.dbtcloud_privatelink_endpoints.all[0].endpoints : ep.id], + lookup(local.privatelink_endpoints_map, repo.private_link_endpoint_key, { endpoint_id = null }).endpoint_id + ) + ].id : null + ) + } +} + +data "dbtcloud_privatelink_endpoints" "all" { + count = local.needs_privatelink_data ? 1 : 0 } ############################################# @@ -125,7 +186,7 @@ resource "dbtcloud_repository" "repositories" { github_installation_id = ( local.effective_git_clone_strategy[each.key] == "github_app" - ? try(each.value.github_installation_id, null) + ? local.resolved_github_installation_id[each.key] : null ) @@ -144,8 +205,18 @@ resource "dbtcloud_repository" "repositories" { each.value.azure_bypass_webhook_registration_failure, false ) - private_link_endpoint_id = try(each.value.private_link_endpoint_id, null) + private_link_endpoint_id = local.private_link_endpoint_id_by_repo_key[each.key] pull_request_url_template = try(each.value.pull_request_url_template, null) + + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_repository (v2 parity). + # resource_metadata = { + # source_project_id = try(local.all_projects_map[each.key].id, null) + # source_id = try(each.value.id, null) + # source_identity = "REP:${each.key}" + # source_key = each.key + # source_project_key = each.key + # source_name = try(each.value.name, each.key) + # } } ############################################# @@ -162,7 +233,7 @@ resource "dbtcloud_repository" "protected_repositories" { github_installation_id = ( local.effective_git_clone_strategy[each.key] == "github_app" - ? try(each.value.github_installation_id, null) + ? local.resolved_github_installation_id[each.key] : null ) @@ -181,9 +252,19 @@ resource "dbtcloud_repository" "protected_repositories" { each.value.azure_bypass_webhook_registration_failure, false ) - private_link_endpoint_id = try(each.value.private_link_endpoint_id, null) + private_link_endpoint_id = local.private_link_endpoint_id_by_repo_key[each.key] pull_request_url_template = try(each.value.pull_request_url_template, null) + # Deferred until dbt-labs/dbtcloud supports resource_metadata on dbtcloud_repository (v2 parity). + # resource_metadata = { + # source_project_id = try(local.all_projects_map[each.key].id, null) + # source_id = try(each.value.id, null) + # source_identity = "REP:${each.key}" + # source_key = each.key + # source_project_key = each.key + # source_name = try(each.value.name, each.key) + # } + lifecycle { prevent_destroy = true } diff --git a/modules/repository/outputs.tf b/modules/repository/outputs.tf index 444507a..256e242 100644 --- a/modules/repository/outputs.tf +++ b/modules/repository/outputs.tf @@ -5,3 +5,8 @@ output "repository_ids" { { for k, r in dbtcloud_repository.protected_repositories : k => tostring(r.repository_id) } ) } + +output "protected_repository_keys" { + description = "Project keys whose dbtcloud_repository uses lifecycle.prevent_destroy (for module.project_repository split)." + value = keys(local.protected_repos_map) +} diff --git a/modules/repository/tests/unit.tftest.hcl b/modules/repository/tests/unit.tftest.hcl index f0a955d..e07084a 100644 --- a/modules/repository/tests/unit.tftest.hcl +++ b/modules/repository/tests/unit.tftest.hcl @@ -8,7 +8,7 @@ mock_provider "dbtcloud" {} # ── GitHub URL auto-detection ───────────────────────────────────────────────── -run "github_url_without_explicit_strategy_auto_detects_github_app" { +run "github_url_without_pat_or_discovery_map_falls_back_to_deploy_key" { command = plan variables { @@ -26,8 +26,8 @@ run "github_url_without_explicit_strategy_auto_detects_github_app" { } assert { - condition = dbtcloud_repository.repositories["analytics"].git_clone_strategy == "github_app" - error_message = "GitHub URL without explicit strategy auto-detects to github_app (no installation_id check in auto-detect path)" + condition = dbtcloud_repository.repositories["analytics"].git_clone_strategy == "deploy_key" + error_message = "GitHub without installation id, PAT, or discovery map should use deploy_key (github_app requires a resolvable installation or PAT)" } } @@ -78,6 +78,66 @@ run "github_url_with_pat_uses_github_app" { } } +run "github_url_resolves_installation_from_discovery_map_without_pat" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + repository = { + remote_url = "https://github.com/my-org/analytics" + } + } + ] + project_ids = { analytics = "1001" } + dbt_pat = null + github_installation_by_owner = { "my-org" = 88332211 } + github_installation_fallback_id = null + } + + assert { + condition = nonsensitive(dbtcloud_repository.repositories["analytics"].git_clone_strategy) == "github_app" + error_message = "GitHub with owner match in github_installation_by_owner should use github_app" + } + + assert { + condition = nonsensitive(dbtcloud_repository.repositories["analytics"].github_installation_id) == 88332211 + error_message = "github_installation_id should come from discovery map keyed by lowercase org from remote_url" + } +} + +run "github_url_uses_installation_fallback_when_owner_unknown" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + repository = { + remote_url = "https://github.com/other-org/analytics" + } + } + ] + project_ids = { analytics = "1001" } + dbt_pat = null + github_installation_by_owner = { "my-org" = 111 } + github_installation_fallback_id = 99988877 + } + + assert { + condition = dbtcloud_repository.repositories["analytics"].git_clone_strategy == "github_app" + error_message = "GitHub should still use github_app when fallback installation id is provided" + } + + assert { + condition = dbtcloud_repository.repositories["analytics"].github_installation_id == 99988877 + error_message = "github_installation_id should fall back when remote_url owner is not in the map" + } +} + # ── GitLab URL auto-detection ───────────────────────────────────────────────── run "gitlab_url_without_explicit_strategy_auto_detects_deploy_token" { diff --git a/modules/repository/variables.tf b/modules/repository/variables.tf index 1771144..5c2dfbf 100644 --- a/modules/repository/variables.tf +++ b/modules/repository/variables.tf @@ -20,3 +20,21 @@ variable "enable_gitlab_deploy_token" { type = bool default = false } + +variable "github_installation_by_owner" { + description = "Lowercase GitHub org/user login → installation id (from module data_lookups when dbt_pat is set). Used to set github_installation_id when not in YAML." + type = map(any) + default = {} +} + +variable "github_installation_fallback_id" { + description = "First GitHub App installation id in the account when owner-based match fails (from module data_lookups)" + type = number + default = null +} + +variable "privatelink_endpoints" { + description = "Optional account-level PrivateLink registry (key + endpoint_id) for resolving repository.private_link_endpoint_key" + type = list(any) + default = [] +} diff --git a/modules/semantic_layer/.terraform.lock.hcl b/modules/semantic_layer/.terraform.lock.hcl new file mode 100644 index 0000000..8dc8cba --- /dev/null +++ b/modules/semantic_layer/.terraform.lock.hcl @@ -0,0 +1,27 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/dbt-labs/dbtcloud" { + version = "1.8.2" + constraints = "~> 1.8" + hashes = [ + "h1:3Hw1in/qcHdAp2BRhEaFRr/ez9lFjBv3V5WIUxHXcSs=", + "h1:ASwPE1I3U7cufOLFvGV+yHmV4/24y2Jrmt8V3Pn9lEY=", + "h1:BIVjUC8FxluKSvwm2jvySD0AvSkFPKwGn/SHez082Oc=", + "h1:IpC3g4srpeI3iJ5GL3GcxSMO/+XHBktS/zumEm/8xbc=", + "zh:01df4a2592c7d66152aa18bae2fb6abcca205f2c0d75486a67e3c8307ea70af2", + "zh:020e8c80eb8973ac0571b8cb2c990d76c82d2032f590ffaec418fc6728a48594", + "zh:0c30dd1d7ac516efe921362143c0372b2dedab59950662c08513c8c1e0430ee4", + "zh:192bc343bfe193d79b4f2e5d5340a44230e169670d190c7034c7db67fd97b3e7", + "zh:24b8175f22a97844000facea8742bd2f0ca5eedb214c900e6b50153308874e86", + "zh:26ef5f40358401f7fadae72b16782d0a8a8761af325aac17b5507234148c19f9", + "zh:7b9cd9637bf607854dfdeceab06dd361686f54864050cfb770d4214a2d0265db", + "zh:8df586c709b83ca62056fd3d9ff5e9d73833acdcb3194d7e5bbc0cde98f87ed9", + "zh:8e561719c50ae83caefa0dd8ebf3cf4907a512d1be90ecb80eb5db6593f9e9b2", + "zh:9071202c4b459dd28fa792a62e229815bae41d637428fa66dca5eb7fa0fc5801", + "zh:926cca7812b272ab976ce49c0858de78bd5c4c6fcffe16cb2d33cdce63522bc2", + "zh:ae927fc0b059c800b0342b6c6f7217dd191349334a6d6ed1cad578f13f64722c", + "zh:b0e484d5c2dc79b1f573aa3bdac4150a2b4094b6fd9c92033b855e7b8f07045c", + "zh:f96b0ce96fe679653e5dd3820f15e7916855ce8deb832f6b92befedc2387f1f6", + ] +} diff --git a/modules/semantic_layer/main.tf b/modules/semantic_layer/main.tf index 6720150..b38d212 100644 --- a/modules/semantic_layer/main.tf +++ b/modules/semantic_layer/main.tf @@ -9,21 +9,41 @@ terraform { } locals { - # Only create semantic layer config for projects that have a semantic_layer block semantic_layer_map = { for p in var.projects : try(p.key, p.name) => p - if try(p.semantic_layer, null) != null + if try(p.semantic_layer_config, null) != null + } + + semantic_environment_id = { + for k, p in local.semantic_layer_map : k => ( + try(p.semantic_layer_config.environment_id, null) != null && tostring(try(p.semantic_layer_config.environment_id, null)) != "" + ? tostring(p.semantic_layer_config.environment_id) + : ( + length(compact([ + try(p.semantic_layer_config.environment_key, null), + try(p.semantic_layer_config.environment, null), + ])) > 0 + ? lookup( + var.environment_ids, + "${k}_${coalesce( + try(p.semantic_layer_config.environment_key, null), + try(p.semantic_layer_config.environment, null), + )}", + null + ) + : null + ) + ) } } resource "dbtcloud_semantic_layer_configuration" "semantic_layer" { for_each = local.semantic_layer_map - project_id = var.project_ids[each.key] - environment_id = lookup( - var.environment_ids, - "${each.key}_${each.value.semantic_layer.environment}", - null - ) + project_id = var.project_ids[each.key] + environment_id = local.semantic_environment_id[each.key] } + +# Deferred: stock dbtcloud provider has no resource_metadata on dbtcloud_semantic_layer_configuration (terraform providers schema). +# v2/importer semantic_layer_config.id would map to source_id when supported. diff --git a/modules/semantic_layer/tests/unit.tftest.hcl b/modules/semantic_layer/tests/unit.tftest.hcl new file mode 100644 index 0000000..e2a1914 --- /dev/null +++ b/modules/semantic_layer/tests/unit.tftest.hcl @@ -0,0 +1,53 @@ +# Unit tests for modules/semantic_layer — run from modules/semantic_layer/: terraform test + +mock_provider "dbtcloud" {} + +run "semantic_layer_config_environment_id" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + semantic_layer_config = { + environment_id = "9001" + } + } + ] + project_ids = { analytics = "1001" } + environment_ids = { + "analytics_dev" = "8001" + } + } + + assert { + condition = tostring(dbtcloud_semantic_layer_configuration.semantic_layer["analytics"].environment_id) == "9001" + error_message = "semantic_layer_config.environment_id should pass through directly" + } +} + +run "semantic_layer_config_environment_key_lookup" { + command = plan + + variables { + projects = [ + { + key = "analytics" + name = "Analytics" + semantic_layer_config = { + environment_key = "dev" + } + } + ] + project_ids = { analytics = "1001" } + environment_ids = { + "analytics_dev" = "8001" + } + } + + assert { + condition = tostring(dbtcloud_semantic_layer_configuration.semantic_layer["analytics"].environment_id) == "8001" + error_message = "semantic_layer_config.environment_key should resolve via environment_ids composite key" + } +} diff --git a/modules/semantic_layer/variables.tf b/modules/semantic_layer/variables.tf index 88b0718..26b9a04 100644 --- a/modules/semantic_layer/variables.tf +++ b/modules/semantic_layer/variables.tf @@ -1,5 +1,5 @@ variable "projects" { - description = "List of project configurations. Each project may have a 'semantic_layer' block with an 'environment' key." + description = "Project configs. Optional semantic_layer_config (environment_id and/or environment_key); see modules/semantic_layer." type = any } diff --git a/modules/service_tokens/main.tf b/modules/service_tokens/main.tf index c95ec81..e7b1905 100644 --- a/modules/service_tokens/main.tf +++ b/modules/service_tokens/main.tf @@ -11,7 +11,12 @@ terraform { locals { tokens_map = { for t in var.service_tokens_data : - try(t.key, t.name) => t + t.key => t + } + + service_tokens_permissions_by_key = { + for k, t in local.tokens_map : + k => try(t.permissions, []) } protected_tokens_map = { @@ -30,14 +35,42 @@ locals { resource "dbtcloud_service_token" "service_tokens" { for_each = local.unprotected_tokens_map - name = each.value.name + name = each.value.name + state = try(each.value.state, 1) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "TOK:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } dynamic "service_token_permissions" { - for_each = try(each.value.permissions, []) + for_each = var.skip_global_project_permissions ? [] : tolist(try(local.service_tokens_permissions_by_key[each.key], [])) content { permission_set = service_token_permissions.value.permission_set - project_id = try(service_token_permissions.value.project_id, null) - all_projects = try(service_token_permissions.value.all_projects, service_token_permissions.value.project_id == null) + all_projects = try( + service_token_permissions.value.all_projects, + try(service_token_permissions.value.project_key, null) == null && + try(service_token_permissions.value.project_id, null) == null, + ) + project_id = ( + try(service_token_permissions.value.project_key, null) != null + ? try(var.project_ids[service_token_permissions.value.project_key], null) + : try(service_token_permissions.value.project_id, null) + ) + writable_environment_categories = try(service_token_permissions.value.writable_environment_categories, []) + } + } + + dynamic "service_token_permissions" { + for_each = var.skip_global_project_permissions ? tolist(try(local.service_tokens_permissions_by_key[each.key], [])) : [] + content { + permission_set = service_token_permissions.value.permission_set + all_projects = true + project_id = null + writable_environment_categories = try(service_token_permissions.value.writable_environment_categories, []) } } } @@ -45,14 +78,42 @@ resource "dbtcloud_service_token" "service_tokens" { resource "dbtcloud_service_token" "protected_service_tokens" { for_each = local.protected_tokens_map - name = each.value.name + name = each.value.name + state = try(each.value.state, 1) + + # resource_metadata: pending official dbtcloud provider support (see importer projects_v2/globals.tf). + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "TOK:${each.key}" + # source_key = each.key + # source_name = each.value.name + # } dynamic "service_token_permissions" { - for_each = try(each.value.permissions, []) + for_each = var.skip_global_project_permissions ? [] : tolist(try(local.service_tokens_permissions_by_key[each.key], [])) content { permission_set = service_token_permissions.value.permission_set - project_id = try(service_token_permissions.value.project_id, null) - all_projects = try(service_token_permissions.value.all_projects, service_token_permissions.value.project_id == null) + all_projects = try( + service_token_permissions.value.all_projects, + try(service_token_permissions.value.project_key, null) == null && + try(service_token_permissions.value.project_id, null) == null, + ) + project_id = ( + try(service_token_permissions.value.project_key, null) != null + ? try(var.project_ids[service_token_permissions.value.project_key], null) + : try(service_token_permissions.value.project_id, null) + ) + writable_environment_categories = try(service_token_permissions.value.writable_environment_categories, []) + } + } + + dynamic "service_token_permissions" { + for_each = var.skip_global_project_permissions ? tolist(try(local.service_tokens_permissions_by_key[each.key], [])) : [] + content { + permission_set = service_token_permissions.value.permission_set + all_projects = true + project_id = null + writable_environment_categories = try(service_token_permissions.value.writable_environment_categories, []) } } diff --git a/modules/service_tokens/variables.tf b/modules/service_tokens/variables.tf index 18a26ac..a043137 100644 --- a/modules/service_tokens/variables.tf +++ b/modules/service_tokens/variables.tf @@ -3,3 +3,15 @@ variable "service_tokens_data" { type = any default = [] } + +variable "project_ids" { + description = "Map of project key to dbt Cloud project ID (resolves permissions[].project_key)" + type = map(number) + default = {} +} + +variable "skip_global_project_permissions" { + description = "When true, create permissions without per-project IDs (all_projects only); for when projects are managed outside this root module" + type = bool + default = false +} diff --git a/modules/user_groups/main.tf b/modules/user_groups/main.tf index c2ef7e1..5623390 100644 --- a/modules/user_groups/main.tf +++ b/modules/user_groups/main.tf @@ -8,10 +8,15 @@ terraform { } } +############################################# +# User Groups (account-level, maps users to group IDs) +# No delete support — manages group membership only. +############################################# + locals { user_groups_map = { for ug in var.user_groups_data : - tostring(ug.user_id) => ug + ug.key => ug } } @@ -19,6 +24,15 @@ resource "dbtcloud_user_groups" "user_groups" { for_each = local.user_groups_map user_id = each.value.user_id + + # resource_metadata: pending official dbtcloud provider support on dbtcloud_user_groups. + # resource_metadata = { + # source_id = try(each.value.id, null) + # source_identity = "USERGRP:${each.key}" + # source_key = each.key + # source_name = coalesce(try(each.value.name, null), format("user_%s", each.value.user_id)) + # } + group_ids = [ for gk in try(each.value.group_keys, []) : tonumber(lookup(var.group_ids, gk, null)) diff --git a/modules/user_groups/outputs.tf b/modules/user_groups/outputs.tf index 8fa9b08..03811ce 100644 --- a/modules/user_groups/outputs.tf +++ b/modules/user_groups/outputs.tf @@ -1,4 +1,4 @@ output "user_group_ids" { - description = "Map of user_id (string) to user_groups resource ID" + description = "Map of assignment key (YAML key or string user_id) to dbtcloud_user_groups resource ID" value = { for k, ug in dbtcloud_user_groups.user_groups : k => ug.id } } diff --git a/modules/user_groups/variables.tf b/modules/user_groups/variables.tf index c77be8b..6a8a2ee 100644 --- a/modules/user_groups/variables.tf +++ b/modules/user_groups/variables.tf @@ -1,5 +1,5 @@ variable "user_groups_data" { - description = "List of user-to-group assignments from YAML user_groups[]. Each entry has user_id and group_keys list." + description = "List of user-to-group assignments from YAML user_groups[]. Each entry has user_id, group_keys; optional key (for_each key, default user_id) and optional id (resource_metadata.source_id when provider supports it)." type = any default = [] } diff --git a/outputs.tf b/outputs.tf index f1b90df..987cb88 100644 --- a/outputs.tf +++ b/outputs.tf @@ -30,7 +30,7 @@ output "environment_ids" { ############################################# output "credential_ids" { - description = "Map of composite key (project_key_env_key) to credential ID" + description = "Map of composite key (project_key_env_key or project_key_profile_key) to credential ID" value = module.credentials.credential_ids } @@ -48,8 +48,23 @@ output "job_ids" { ############################################# output "connection_ids" { - description = "Map of global connection key to dbt Cloud connection ID" - value = length(try(local.yaml_content.global_connections, [])) > 0 ? module.global_connections[0].connection_ids : {} + description = "Map of global connection key (and LOOKUP:… placeholders) to dbt Cloud connection ID — merged managed global_connections + data_lookups" + value = local.global_connection_ids_effective +} + +output "lookup_connection_ids" { + description = "Subset of connection_ids from LOOKUP:… resolution only (empty if module.data_lookups not used)" + value = length(module.data_lookups) > 0 ? module.data_lookups[0].lookup_connection_ids : {} +} + +output "github_installation_by_owner" { + description = "GitHub App installation id by org/user login (from dbt integrations API when dbt_pat is set)" + value = length(module.data_lookups) > 0 ? module.data_lookups[0].github_installation_by_owner : {} +} + +output "github_installation_fallback_id" { + description = "First GitHub installation id when owner-based match is not used" + value = length(module.data_lookups) > 0 ? module.data_lookups[0].github_installation_fallback_id : null } output "service_token_ids" { @@ -62,6 +77,11 @@ output "group_ids" { value = length(try(local.yaml_content.groups, [])) > 0 ? module.groups[0].group_ids : {} } +output "notification_ids" { + description = "Map of notification key to dbt Cloud notification ID" + value = length(try(local.yaml_content.notifications, [])) > 0 ? module.notifications[0].notification_ids : {} +} + output "oauth_configuration_ids" { description = "Map of OAuth configuration key to dbt Cloud OAuth configuration ID" value = length(try(local.yaml_content.oauth_configurations, [])) > 0 ? module.oauth_configurations[0].oauth_configuration_ids : {} @@ -77,12 +97,12 @@ output "ip_rule_ids" { ############################################# output "extended_attribute_ids" { - description = "Map of composite key (project_key_ea_key) to extended_attributes resource ID" + description = "Map of composite key (project_key_ea_key) to dbt Cloud extended_attributes_id (numeric API id)" value = length(flatten([for p in local.projects : try(p.extended_attributes, [])])) > 0 ? module.extended_attributes[0].extended_attribute_ids : {} } output "profile_ids" { - description = "Map of composite key (project_key_profile_key) to dbt Cloud profile ID" + description = "Map of composite key (project_key_profile_key) to dbt Cloud profile_id (numeric API id)" value = length(flatten([for p in local.projects : try(p.profiles, [])])) > 0 ? module.profiles[0].profile_ids : {} } @@ -90,3 +110,23 @@ output "lineage_integration_ids" { description = "Map of composite key (project_key_integration_key) to lineage integration ID" value = length(flatten([for p in local.projects : try(p.lineage_integrations, [])])) > 0 ? module.lineage_integrations[0].lineage_integration_ids : {} } + +output "semantic_layer_ids" { + description = "Map of project key to dbt Cloud semantic layer configuration ID" + value = length(module.semantic_layer) > 0 ? module.semantic_layer[0].semantic_layer_ids : {} +} + +output "project_artefact_ids" { + description = "Map of project key to dbt Cloud project_artefacts resource ID" + value = length(module.project_artefacts) > 0 ? module.project_artefacts[0].project_artefact_ids : {} +} + +output "yaml_schema_version" { + description = "YAML version key from the config file (must be 1; see schemas/v1.json)" + value = try(local._raw_yaml.version, null) +} + +output "yaml_account" { + description = "The YAML account block (name, host_url, id)" + value = try(local._raw_yaml.account, null) +} diff --git a/providers.tf b/providers.tf index ba0bab7..a9169c0 100644 --- a/providers.tf +++ b/providers.tf @@ -5,9 +5,15 @@ terraform { source = "dbt-labs/dbtcloud" version = "~> 1.8" } + http = { + source = "hashicorp/http" + version = "~> 3.0" + } } } +provider "http" {} + provider "dbtcloud" { account_id = var.dbt_account_id token = var.dbt_token diff --git a/schemas/v1.json b/schemas/v1.json index dd89cb5..a467e47 100644 --- a/schemas/v1.json +++ b/schemas/v1.json @@ -1,507 +1,928 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://raw.githubusercontent.com/trouze/terraform-dbtcloud-yaml/refs/heads/main/schemas/v1.json", - "title": "dbt Cloud Terraform YAML Configuration Schema v1", - "description": "JSON Schema for validating dbt Cloud YAML configuration files (v1). Supports both singular project: and list-based projects: shapes. Use with VS Code YAML extension or JetBrains built-in YAML validation.", + "title": "dbt Cloud Terraform Configuration Schema v1", + "description": "Canonical JSON Schema for dbt Cloud Terraform YAML (version: 1). Root Terraform hoists globals.connections → global_connections, globals.privatelink_endpoints → privatelink_endpoints, globals.service_tokens / groups / notifications to top-level keys modules read, and converts project.environment_variables[].environment_values from a map to a list internally. The dbtcloud provider uses Terraform variables (e.g. TF_VAR_dbt_host_url), not YAML — mirror account.host_url there for applies.", "type": "object", "properties": { - - "account_features": { + "version": { + "const": 1, + "description": "Schema version identifier. Must be set to 1 for this document." + }, + "account": { "type": "object", - "description": "Account-level feature flags", + "description": "Account-level metadata applied to all projects in this file.", "properties": { - "advanced_ci": { "type": "boolean", "description": "Enable Advanced CI (compare changes) feature" }, - "partial_parsing": { "type": "boolean", "description": "Enable partial parsing for faster runs" }, - "repo_caching": { "type": "boolean", "description": "Enable repository caching" } + "name": { + "type": "string", + "description": "Friendly dbt Cloud account name", + "minLength": 1, + "maxLength": 128 + }, + "host_url": { + "type": "string", + "description": "dbt Cloud host URL (e.g., https://cloud.getdbt.com)", + "format": "uri" + }, + "id": { + "type": ["integer", "null"], + "description": "Optional dbt Cloud account ID", + "minimum": 1 + } }, + "required": ["name", "host_url"], "additionalProperties": false }, - - "global_connections": { - "type": "array", - "description": "Account-level global connections referenced by environments via connection_key", - "items": { - "type": "object", - "description": "Global connection configuration", - "properties": { - "name": { "type": "string", "description": "Display name for the connection", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key used to reference this connection in environments", "pattern": "^[a-z0-9_]+$" }, - "type": { - "type": "string", - "description": "Warehouse type", - "enum": ["databricks", "snowflake", "bigquery", "redshift", "postgres", "spark", "starburst_trino", "apache_spark", "athena", "fabric", "synapse"] - }, - "protected": { "type": "boolean", "description": "Protect from accidental deletion", "default": false }, - "host": { "type": "string", "description": "[Databricks] Workspace hostname (e.g. adb-1234.azuredatabricks.net)" }, - "http_path": { "type": "string", "description": "[Databricks] HTTP path for the warehouse or cluster" }, - "catalog": { "type": "string", "description": "[Databricks] Unity Catalog catalog name" }, - "account": { "type": "string", "description": "[Snowflake] Account identifier (e.g. xy12345.us-east-1)" }, - "database": { "type": "string", "description": "[Snowflake] Default database" }, - "warehouse": { "type": "string", "description": "[Snowflake] Virtual warehouse name" }, - "role": { "type": "string", "description": "[Snowflake] Default role" }, - "allow_sso": { "type": "boolean", "description": "[Snowflake] Allow SSO authentication" } - }, - "required": ["name", "key", "type"], - "additionalProperties": true - } - }, - - "service_tokens": { - "type": "array", - "description": "Account-level service tokens", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Service token display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for this service token", "pattern": "^[a-z0-9_]+$" }, - "permissions": { - "type": "array", - "description": "Permission sets granted to this token", - "items": { - "type": "object", - "properties": { - "permission_set": { - "type": "string", - "description": "Permission set name", - "enum": ["owner", "member", "developer", "analyst", "stakeholder", "readonly", "job_runner", "git_admin", "account_admin", "admin", "database_admin", "write_artifacts", "view_environment", "webhook_admin", "billing_admin", "semantic_layer_only"] - }, - "all_projects": { "type": "boolean", "description": "Apply to all projects", "default": false }, - "project_key": { "type": "string", "description": "Restrict to a specific project key" } - }, - "required": ["permission_set"], - "additionalProperties": false - } - } + "globals": { + "type": "object", + "description": "Reusable account-level resources referenced by projects.", + "properties": { + "connections": { + "type": "array", + "items": { "$ref": "#/$defs/connection" } }, - "required": ["name", "key", "permissions"], - "additionalProperties": false - } - }, - - "groups": { - "type": "array", - "description": "Account-level groups for RBAC", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Group display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for this group", "pattern": "^[a-z0-9_]+$" }, - "assign_by_default": { "type": "boolean", "description": "Automatically assign new users to this group", "default": false }, - "sso_mapping_groups": { "type": "array", "description": "SSO group names to map to this group", "items": { "type": "string" } } - }, - "required": ["name", "key"], - "additionalProperties": false - } + "repositories": { + "type": "array", + "items": { "$ref": "#/$defs/repository" } + }, + "privatelink_endpoints": { + "type": "array", + "items": { "$ref": "#/$defs/privateLinkEndpoint" } + }, + "service_tokens": { + "type": "array", + "items": { "$ref": "#/$defs/serviceToken" } + }, + "groups": { + "type": "array", + "items": { "$ref": "#/$defs/group" } + }, + "semantic_layer_configs": { + "type": "array", + "items": { "$ref": "#/$defs/semanticLayerConfig" } + }, + "notifications": { + "type": "array", + "items": { "$ref": "#/$defs/notificationTarget" } + } + }, + "additionalProperties": false }, - - "user_groups": { + "projects": { "type": "array", - "description": "Assign users to groups", - "items": { - "type": "object", - "properties": { - "user_id": { "type": "integer", "description": "dbt Cloud user ID", "minimum": 1 }, - "group_keys": { "type": "array", "description": "List of group keys to assign the user to", "items": { "type": "string" }, "minItems": 1 } - }, - "required": ["user_id", "group_keys"], - "additionalProperties": false - } + "description": "List of dbt Cloud projects managed by this file.", + "minItems": 1, + "items": { "$ref": "#/$defs/project" } }, - - "notifications": { - "type": "array", - "description": "Account-level job notifications", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Notification display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for this notification", "pattern": "^[a-z0-9_]+$" }, - "notification_type": { - "type": "integer", - "description": "Destination type: 1=email, 2=slack, 4=pagerduty, 5=webhook", - "enum": [1, 2, 4, 5] - }, - "user_id": { "type": ["integer", "null"], "description": "dbt Cloud user ID to notify (email notifications)", "minimum": 1 }, - "slack_channel_id": { "type": ["string", "null"], "description": "[Slack] Slack channel ID" }, - "slack_channel_name": { "type": ["string", "null"], "description": "[Slack] Slack channel display name" }, - "webhook_id": { "type": ["string", "null"], "description": "[Webhook] Webhook ID" }, - "on_failure": { "type": "array", "description": "Job IDs to notify on failure", "items": { "type": "integer", "minimum": 1 }, "default": [] }, - "on_success": { "type": "array", "description": "Job IDs to notify on success", "items": { "type": "integer", "minimum": 1 }, "default": [] }, - "on_cancel": { "type": "array", "description": "Job IDs to notify on cancel", "items": { "type": "integer", "minimum": 1 }, "default": [] }, - "on_warning": { "type": "array", "description": "Job IDs to notify on warning", "items": { "type": "integer", "minimum": 1 }, "default": [] } - }, - "required": ["name", "key", "notification_type"], - "additionalProperties": false - } + "metadata": { + "type": "object", + "description": "Auxiliary metadata such as placeholder descriptions.", + "properties": { + "placeholders": { + "type": "array", + "items": { "$ref": "#/$defs/placeholder" } + } + }, + "additionalProperties": false }, - "oauth_configurations": { "type": "array", - "description": "Account-level OAuth configurations. Client secrets are supplied via var.oauth_client_secrets[key].", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "OAuth config display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key (also used as lookup key in oauth_client_secrets)", "pattern": "^[a-z0-9_]+$" }, - "type": { "type": "string", "description": "OAuth provider type", "enum": ["snowflake"] }, - "authorize_url": { "type": "string", "description": "OAuth authorization endpoint URL" }, - "token_url": { "type": "string", "description": "OAuth token exchange endpoint URL" }, - "redirect_uri": { "type": "string", "description": "OAuth redirect URI registered with the provider" }, - "client_id": { "type": "string", "description": "OAuth client ID" } - }, - "required": ["name", "key", "type", "authorize_url", "token_url", "redirect_uri", "client_id"], - "additionalProperties": false - } + "description": "Account OAuth configurations (passed through to modules/oauth_configurations).", + "items": { "type": "object", "additionalProperties": true } }, - "ip_restrictions": { "type": "array", - "description": "Account-level IP restriction rules", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Rule display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for this rule", "pattern": "^[a-z0-9_]+$" }, - "type": { "type": "string", "description": "Rule type: allow or deny", "enum": ["allow", "deny"] }, - "cidrs": { - "type": "array", - "description": "CIDR blocks for this rule", - "items": { - "type": "object", - "properties": { - "cidr": { "type": "string", "description": "CIDR block (e.g. 203.0.113.0/24)", "pattern": "^([0-9]{1,3}\\.){3}[0-9]{1,3}/[0-9]{1,2}$" } - }, - "required": ["cidr"], - "additionalProperties": false - }, - "minItems": 1 - }, - "rule_set_enabled": { "type": "boolean", "description": "Whether the IP restriction rule set is active", "default": false } - }, - "required": ["name", "key", "type", "cidrs"], - "additionalProperties": false - } + "description": "Account IP restriction rules (modules/ip_restrictions).", + "items": { "type": "object", "additionalProperties": true } }, - - "projects": { - "type": "array", - "description": "List of dbt Cloud projects. Use this (plural) for multi-project configs. See also: project (singular).", - "minItems": 1, - "items": { "$ref": "#/$defs/project" } + "account_features": { + "type": ["object", "null"], + "description": "Optional account feature flags (modules/account_features)." }, - - "project": { - "$ref": "#/$defs/project", - "description": "Single dbt Cloud project. Legacy singular form — prefer projects: (list)." + "user_groups": { + "type": "array", + "description": "User ↔ group assignments (modules/user_groups); each entry must include key.", + "items": { "type": "object", "additionalProperties": true } } }, - - "anyOf": [ - { "required": ["projects"] }, - { "required": ["project"] } - ], - + "required": ["version", "account", "projects"], "additionalProperties": false, - "$defs": { - - "project": { + "slug": { + "type": "string", + "pattern": "^[A-Za-z0-9_.-]+$", + "minLength": 1, + "maxLength": 128, + "description": "Reusable key/slug used for cross references." + }, + "lookupString": { + "type": "string", + "pattern": "^LOOKUP:[A-Za-z0-9_.-]+$", + "description": "Placeholder instructing Terraform to resolve a resource via data source." + }, + "resourceRef": { + "description": "Reference to an existing resource by numeric ID, slug key, or lookup placeholder.", + "oneOf": [ + { + "type": "integer", + "minimum": 1 + }, + { + "$ref": "#/$defs/slug" + }, + { + "$ref": "#/$defs/lookupString" + } + ] + }, + "repository": { "type": "object", - "description": "dbt Cloud project configuration", + "description": "Git repository configuration with provider-specific fields.", "properties": { - "name": { "type": "string", "description": "Project display name", "minLength": 1, "maxLength": 128 }, - "key": { "type": "string", "description": "Unique key for cross-referencing this project", "pattern": "^[a-z0-9_]+$" }, - "protected": { "type": "boolean", "description": "Protect from accidental deletion via lifecycle ignore", "default": false }, - - "repository": { - "type": "object", - "description": "Git repository configuration", - "properties": { - "remote_url": { - "type": "string", - "description": "Repository path or URL. For GitHub/GitLab/Azure DevOps native integrations this is typically 'org/repo'. For deploy-key connections use HTTPS or SSH URL.", - "minLength": 1 - }, - "git_clone_strategy": { - "type": ["string", "null"], - "description": "Clone strategy. Auto-detected from integration IDs if omitted. Options: deploy_key, github_app, deploy_token, azure_active_directory_app", - "enum": ["deploy_key", "github_app", "deploy_token", "azure_active_directory_app", null], - "default": null - }, - "github_installation_id": { - "type": ["integer", "null"], - "description": "[GitHub native] GitHub App installation ID", - "minimum": 1, - "default": null - }, - "gitlab_project_id": { - "type": ["integer", "null"], - "description": "[GitLab native] GitLab project ID (numeric, found in project Settings > General)", - "minimum": 1, - "default": null - }, - "pull_request_url_template": { - "type": ["string", "null"], - "description": "Custom URL template for opening pull requests. Use {{source}} and {{destination}} placeholders.", - "default": null - }, - "azure_active_directory_project_id": { - "type": ["string", "null"], - "description": "[Azure DevOps native] Azure DevOps project ID (UUID)", - "default": null - }, - "azure_active_directory_repository_id": { - "type": ["string", "null"], - "description": "[Azure DevOps native] Azure DevOps repository ID (UUID)", - "default": null - }, - "azure_bypass_webhook_registration_failure": { - "type": ["boolean", "null"], - "description": "[Azure DevOps native] If true, connection succeeds even if webhook registration fails", - "default": false - }, - "private_link_endpoint_id": { - "type": ["string", "null"], - "description": "Private Link endpoint ID for VPC access", - "default": null - }, - "is_active": { - "type": ["boolean", "null"], - "description": "Whether the repository connection is active", - "default": true - } - }, - "required": ["remote_url"], - "additionalProperties": false + "key": { + "$ref": "#/$defs/slug" + }, + "remote_url": { + "type": "string", + "description": "Git repository URL (HTTPS or SSH format). Supports GitHub, GitLab, Azure DevOps, and Bitbucket.", + "minLength": 1 + }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false + }, + "git_clone_strategy": { + "type": ["string", "null"], + "description": "Git clone strategy. If omitted, auto-detected based on provider: 'deploy_key' (default), 'github_app' (GitHub native), 'deploy_token' (GitLab native), 'azure_active_directory_app' (Azure DevOps native)", + "enum": ["deploy_key", "github_app", "deploy_token", "azure_active_directory_app", null], + "default": null + }, + "is_active": { + "type": ["boolean", "null"], + "description": "Whether the repository is active and available for use", + "default": true + }, + "github_installation_id": { + "type": ["integer", "null"], + "description": "[GitHub native integration only] GitHub App installation ID.", + "minimum": 1, + "default": null + }, + "gitlab_project_id": { + "type": ["integer", "null"], + "description": "[GitLab native integration only] GitLab project ID (numeric).", + "minimum": 1, + "default": null + }, + "azure_active_directory_project_id": { + "type": ["string", "null"], + "description": "[Azure DevOps native integration only] Azure DevOps project ID (UUID format).", + "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$", + "default": null + }, + "azure_active_directory_repository_id": { + "type": ["string", "null"], + "description": "[Azure DevOps native integration only] Azure DevOps repository ID (UUID format).", + "pattern": "^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$", + "default": null + }, + "azure_bypass_webhook_registration_failure": { + "type": ["boolean", "null"], + "description": "[Azure DevOps native integration only] If false (default), connection fails if service user can't set webhooks. If true, connection succeeds but auto-triggering won't work.", + "default": false + }, + "private_link_endpoint_key": { + "type": ["string", "null"], + "description": "Reference to a PrivateLink endpoint defined in globals.", + "default": null + }, + "pull_request_url_template": { + "type": ["string", "null"], + "description": "Custom URL template for creating pull requests.", + "default": null + }, + "id": { + "type": ["integer", "null"], + "description": "Optional repository ID for lookups", + "minimum": 1 + } + }, + "required": ["remote_url"], + "additionalProperties": true + }, + "repositoryRef": { + "description": "Reference to a repository defined in globals or inline object.", + "oneOf": [ + { "$ref": "#/$defs/slug" }, + { "$ref": "#/$defs/repository" } + ] + }, + "credential": { + "type": "object", + "description": "Credential configuration.", + "properties": { + "token_name": { + "type": "string", + "description": "Key in token_map variable that contains the warehouse token", + "minLength": 1 + }, + "schema": { + "type": "string", + "description": "Default warehouse schema name", + "minLength": 1 + }, + "catalog": { + "type": ["string", "null"], + "description": "Catalog name (e.g., for Databricks Unity Catalog)", + "default": null + } + }, + "required": ["token_name", "schema"], + "additionalProperties": false + }, + "profileCredential": { + "type": "object", + "description": "Standalone profile credential configuration used to materialize profile-owned credentials.", + "properties": { + "id": { + "type": ["integer", "null"], + "minimum": 1 + }, + "credential_type": { + "type": "string", + "minLength": 1 + }, + "schema": { + "type": "string", + "minLength": 1 + }, + "num_threads": { + "type": ["integer", "null"], + "minimum": 1 + }, + "token_name": { + "type": ["string", "null"], + "minLength": 1 + }, + "catalog": { + "type": ["string", "null"], + "default": null + }, + "auth_type": { + "type": ["string", "null"] + }, + "user": { + "type": ["string", "null"] + }, + "warehouse": { + "type": ["string", "null"] + }, + "role": { + "type": ["string", "null"] + }, + "database": { + "type": ["string", "null"] + }, + "dataset": { + "type": ["string", "null"] + }, + "default_schema": { + "type": ["string", "null"] + }, + "username": { + "type": ["string", "null"] + }, + "target_name": { + "type": ["string", "null"] + }, + "tenant_id": { + "type": ["string", "null"] + }, + "client_id": { + "type": ["string", "null"] + }, + "schema_authorization": { + "type": ["string", "null"] + }, + "authentication": { + "type": ["string", "null"] + } + }, + "required": ["credential_type", "schema"], + "additionalProperties": false + }, + "extendedAttributes": { + "type": "object", + "description": "Extended attributes for environment-level connection overrides (flexible JSON config).", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "description": "Display label; Terraform uses try(key, name) as the identifier when key is omitted." + }, + "id": { + "type": ["integer", "null"], + "description": "Optional extended attributes ID for reference.", + "minimum": 1 + }, + "state": { + "type": ["integer", "null"], + "description": "State: 1 = active, 2 = inactive.", + "enum": [1, 2, null], + "default": 1 + }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false }, - "extended_attributes": { + "type": "object", + "description": "Arbitrary JSON key-value pairs for connection overrides (e.g., catalog, schema).", + "additionalProperties": true + } + }, + "required": ["extended_attributes"], + "anyOf": [{ "required": ["key"] }, { "required": ["name"] }], + "additionalProperties": false + }, + "environment": { + "type": "object", + "description": "Environment configuration referenced by jobs.", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "minLength": 1, + "maxLength": 128 + }, + "type": { + "type": "string", + "enum": ["development", "deployment"], + "default": "development" + }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false + }, + "connection": { + "$ref": "#/$defs/resourceRef", + "description": "Connection reference (ID, key, or LOOKUP placeholder)." + }, + "connection_id": { + "type": ["integer", "null"], + "description": "Deprecated: connection ID (use connection instead).", + "minimum": 1 + }, + "credential": { "$ref": "#/$defs/credential" }, + "dbt_version": { + "type": ["string", "null"], + "description": "dbt version (e.g., '1.5.0', '1.6.0'). Null uses default.", + "pattern": "^(\\d+\\.\\d+\\.\\d+)?$" + }, + "custom_branch": { + "type": ["string", "null"], + "description": "Custom git branch for this environment", + "default": null + }, + "enable_model_query_history": { + "type": ["boolean", "null"], + "description": "Enable query history in dbt Cloud", + "default": null + }, + "target_name": { + "type": ["string", "null"], + "description": "Override default target name", + "default": null + }, + "extended_attributes_key": { + "oneOf": [ + { "$ref": "#/$defs/slug" }, + { "type": "null" } + ], + "description": "Key of extended attributes (defined in project.extended_attributes) to use for this environment.", + "default": null + }, + "primary_profile_key": { + "oneOf": [ + { "$ref": "#/$defs/slug" }, + { "type": "null" } + ], + "description": "Key of a project profile to use as the environment's primary profile.", + "default": null + }, + "id": { + "type": ["integer", "null"], + "description": "Optional environment ID for reference.", + "minimum": 1 + } + }, + "required": ["key", "name", "type", "connection"], + "additionalProperties": false + }, + "jobTriggers": { + "type": "object", + "description": "Job trigger configuration (at least one must be true).", + "properties": { + "schedule": { + "type": "boolean", + "description": "Enable scheduled runs", + "default": false + }, + "github_webhook": { + "type": "boolean", + "description": "Trigger on GitHub push", + "default": false + }, + "git_provider_webhook": { + "type": "boolean", + "description": "Trigger on git provider webhook", + "default": false + }, + "on_merge": { + "type": "boolean", + "description": "Trigger on merge to main branch", + "default": false + } + }, + "required": ["schedule", "github_webhook", "git_provider_webhook", "on_merge"], + "additionalProperties": false + }, + "job": { + "type": "object", + "description": "Job configuration referencing an environment.", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "minLength": 1, + "maxLength": 256 + }, + "environment_key": { + "$ref": "#/$defs/slug", + "description": "Key of the environment this job runs in." + }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false + }, + "description": { + "type": ["string", "null"], + "maxLength": 512, + "default": null + }, + "is_active": { + "type": ["boolean", "null"], + "default": true + }, + "execute_steps": { "type": "array", - "description": "Extended attribute sets that can be applied to environments", + "minItems": 1, "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for referencing in environments via extended_attributes_key", "pattern": "^[a-z0-9_]+$" }, - "content": { "type": "object", "description": "Free-form YAML object with adapter-specific overrides (e.g. databricks.http_path)", "additionalProperties": true } - }, - "required": ["name", "key", "content"], - "additionalProperties": false + "type": "string", + "minLength": 1 } }, - + "triggers": { "$ref": "#/$defs/jobTriggers" }, + "schedule_type": { + "type": ["string", "null"], + "enum": ["every_day", "every_week", "every_month", "days_of_week", null], + "default": null + }, + "schedule_hours": { + "type": ["array", "null"], + "items": { + "type": "integer", + "minimum": 0, + "maximum": 23 + }, + "default": null + }, + "schedule_days": { + "type": ["array", "null"], + "items": { + "type": "integer", + "minimum": 0, + "maximum": 6 + }, + "default": null + }, + "schedule_cron": { + "type": ["string", "null"], + "pattern": "^(([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|\\*|\\*/[0-9]+|[0-9]+-[0-9]+)(,([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|\\*|\\*/[0-9]+|[0-9]+-[0-9]+))*\\s+){4}([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|\\*|\\*/[0-9]+|[0-9]+-[0-9]+)(,([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9]|\\*|\\*/[0-9]+|[0-9]+-[0-9]+))*$", + "default": null + }, + "num_threads": { + "type": ["integer", "null"], + "minimum": 1, + "maximum": 16, + "default": null + }, + "timeout_seconds": { + "type": ["integer", "null"], + "minimum": 300, + "maximum": 86400, + "default": null + }, + "target_name": { + "type": ["string", "null"], + "default": null + }, + "dbt_version": { + "type": ["string", "null"], + "pattern": "^(\\d+\\.\\d+\\.\\d+)?$", + "default": null + }, + "generate_docs": { + "type": ["boolean", "null"], + "default": null + }, + "run_lint": { + "type": ["boolean", "null"], + "default": null + }, + "run_generate_sources": { + "type": ["boolean", "null"], + "default": null + }, + "run_compare_changes": { + "type": ["boolean", "null"], + "default": null + }, + "errors_on_lint_failure": { + "type": ["boolean", "null"], + "default": null + }, + "triggers_on_draft_pr": { + "type": ["boolean", "null"], + "default": null + }, + "self_deferring": { + "type": ["boolean", "null"], + "default": null + }, + "deferring_job_id": { + "type": ["integer", "null"], + "minimum": 1, + "default": null + }, + "deferring_environment_id": { + "type": ["integer", "null"], + "minimum": 1, + "default": null + }, + "deferring_job_key": { + "$ref": "#/$defs/slug" + }, + "deferring_environment_key": { + "$ref": "#/$defs/slug" + }, + "force_node_selection": { + "type": ["boolean", "null"], + "description": "(Deprecated) Controls SAO. true=disabled, false/null=enabled. Omit for CI/Merge jobs. Use cost_optimization_features instead.", + "default": null + }, + "cost_optimization_features": { + "type": ["array", "null"], + "items": { + "type": "string", + "enum": ["state_aware_orchestration"] + }, + "description": "Cost optimization features. Include 'state_aware_orchestration' to enable SAO. Requires dbt_version='latest-fusion'.", + "default": null + }, + "job_type": { + "type": ["string", "null"], + "enum": ["scheduled", "ci", "merge", "other", null], + "description": "Job type for categorization. CI and merge jobs cannot use force_node_selection.", + "default": null + }, + "compare_changes_flags": { + "type": ["string", "null"], + "description": "Flags to pass to dbt when comparing changes (for run_compare_changes).", + "default": "--select state:modified" + }, + "notification_keys": { + "type": "array", + "items": { "$ref": "#/$defs/slug" }, + "description": "References to notification targets defined in globals/projects." + }, + "environment_variable_overrides": { + "type": "object", + "description": "Per-job environment variable overrides (dbtcloud_environment_variable_job_override). Values prefixed with secret_ are resolved via the token_map Terraform variable (remainder of the string is the map key).", + "additionalProperties": { "type": "string" } + }, + "id": { + "type": ["integer", "null"], + "minimum": 1, + "description": "Optional job ID for backward compatibility." + } + }, + "required": ["key", "name", "environment_key", "execute_steps", "triggers"], + "additionalProperties": false + }, + "environmentVariable": { + "type": "object", + "properties": { + "name": { + "type": "string", + "pattern": "^[A-Z_][A-Z0-9_]*$", + "minLength": 1, + "maxLength": 256 + }, + "environment_values": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "string", + "maxLength": 4096 + } + } + }, + "required": ["name", "environment_values"], + "additionalProperties": false + }, + "profile": { + "type": "object", + "description": "Profile linking a connection and credentials for use in deployment environments.", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false + }, + "connection_key": { "$ref": "#/$defs/slug" }, + "credentials_key": { "$ref": "#/$defs/slug" }, + "credential": { "$ref": "#/$defs/profileCredential" }, + "extended_attributes_key": { + "oneOf": [ + { "$ref": "#/$defs/slug" }, + { "type": "null" } + ], + "default": null + }, + "id": { + "type": ["integer", "null"], + "minimum": 1, + "description": "Optional source profile ID for migration metadata." + }, + "connection_id": { + "type": ["integer", "null"], + "minimum": 1, + "description": "Optional source connection ID for migration metadata." + }, + "credentials_id": { + "type": ["integer", "null"], + "minimum": 1, + "description": "Optional source credential ID for migration metadata." + }, + "extended_attributes_id": { + "type": ["integer", "null"], + "minimum": 1, + "description": "Optional source extended attributes ID for migration metadata." + } + }, + "required": ["key", "connection_key", "credentials_key"], + "additionalProperties": false + }, + "project": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "pattern": "^[a-zA-Z0-9_-]+$", + "minLength": 1, + "maxLength": 128 + }, + "protected": { + "type": ["boolean", "null"], + "description": "If true, Terraform will prevent this resource from being destroyed (lifecycle.prevent_destroy = true)", + "default": false + }, + "repository": { + "$ref": "#/$defs/repositoryRef" + }, "environments": { "type": "array", - "description": "dbt Cloud environments for this project", "minItems": 1, - "items": { - "type": "object", - "description": "Environment configuration", - "properties": { - "name": { "type": "string", "description": "Environment display name", "minLength": 1, "maxLength": 128 }, - "key": { "type": "string", "description": "Unique key within this project for cross-referencing", "pattern": "^[a-z0-9_]+$" }, - "type": { "type": "string", "description": "Environment type", "enum": ["development", "deployment"], "default": "development" }, - "connection_key": { "type": "string", "description": "References global_connections[].key to attach a global connection" }, - "deployment_type": { - "type": ["string", "null"], - "description": "Deployment environment classification (required for type: deployment)", - "enum": ["production", "staging", "other", null], - "default": null - }, - "custom_branch": { "type": ["string", "null"], "description": "Custom git branch. Null uses the project default branch.", "default": null }, - "dbt_version": { "type": ["string", "null"], "description": "dbt version override for this environment (e.g. '1.8.0')", "default": null }, - "extended_attributes_key": { "type": ["string", "null"], "description": "References extended_attributes[].key to apply adapter overrides", "default": null }, - "protected": { "type": "boolean", "description": "Protect from accidental deletion", "default": false }, - "enable_model_query_history": { "type": ["boolean", "null"], "description": "Enable model query history", "default": null }, - "credential": { - "type": "object", - "description": "Warehouse credential for this environment", - "properties": { - "credential_type": { - "type": "string", - "description": "Warehouse adapter type", - "enum": ["databricks", "snowflake", "bigquery", "redshift", "postgres"] - }, - "schema": { "type": "string", "description": "Default target schema", "minLength": 1 }, - "catalog": { "type": "string", "description": "[Databricks] Unity Catalog catalog name" }, - "token_name": { "type": "string", "description": "[Databricks] Key in token_map variable containing the personal access token" }, - "auth_type": { "type": "string", "description": "[Snowflake] Authentication type", "enum": ["password", "keypair"] }, - "user": { "type": "string", "description": "[Snowflake] Service user name" }, - "database": { "type": "string", "description": "[Snowflake] Default database" }, - "warehouse": { "type": "string", "description": "[Snowflake] Virtual warehouse name" }, - "role": { "type": "string", "description": "[Snowflake] Default role" }, - "num_threads": { "type": "integer", "description": "Number of parallel threads", "minimum": 1, "maximum": 64 } - }, - "required": ["credential_type"], - "additionalProperties": false - } - }, - "required": ["name", "key", "type"], - "additionalProperties": false - } + "items": { "$ref": "#/$defs/environment" } }, - - "jobs": { + "extended_attributes": { "type": "array", - "description": "Jobs for this project, referencing environments by environment_key", - "items": { - "type": "object", - "description": "Job configuration", - "properties": { - "name": { "type": "string", "description": "Job display name", "minLength": 1, "maxLength": 256 }, - "key": { "type": "string", "description": "Unique key within this project", "pattern": "^[a-z0-9_]+$" }, - "environment_key": { "type": "string", "description": "References environments[].key to place this job" }, - "execute_steps": { - "type": "array", - "description": "dbt commands to execute (e.g. 'dbt build', 'dbt test')", - "minItems": 1, - "items": { "type": "string", "minLength": 1 } - }, - "triggers": { - "type": "object", - "description": "Job trigger configuration", - "properties": { - "schedule": { "type": "boolean", "description": "Enable scheduled runs", "default": false }, - "github_webhook": { "type": "boolean", "description": "Trigger on GitHub webhook (requires GitHub App integration)", "default": false }, - "git_provider_webhook": { "type": "boolean", "description": "Trigger on git provider webhook (GitLab, Azure DevOps, etc.)", "default": false }, - "on_merge": { "type": "boolean", "description": "Trigger when a PR is merged", "default": false } - }, - "required": ["schedule", "github_webhook", "git_provider_webhook", "on_merge"], - "additionalProperties": false - }, - "description": { "type": ["string", "null"], "description": "Job description", "maxLength": 512, "default": null }, - "num_threads": { "type": ["integer", "null"], "description": "Parallel thread count", "minimum": 1, "maximum": 64, "default": null }, - "timeout_seconds": { "type": ["integer", "null"], "description": "Job timeout in seconds", "minimum": 0, "default": null }, - "target_name": { "type": ["string", "null"], "description": "dbt target name override", "default": null }, - "dbt_version": { "type": ["string", "null"], "description": "Job-level dbt version override", "default": null }, - "schedule_type": { - "type": ["string", "null"], - "description": "Schedule frequency type", - "enum": ["days_of_week", "every_day", "every_hour", "custom_cron", null], - "default": null - }, - "schedule_days": { - "type": ["array", "null"], - "description": "Days of week to run (0=Sunday … 6=Saturday). Used with days_of_week schedule_type.", - "items": { "type": "integer", "minimum": 0, "maximum": 6 }, - "default": null - }, - "schedule_hours": { - "type": ["array", "null"], - "description": "Hours of day to run (0–23, UTC). Used with days_of_week and every_day schedule types.", - "items": { "type": "integer", "minimum": 0, "maximum": 23 }, - "default": null - }, - "schedule_cron": { - "type": ["string", "null"], - "description": "Custom cron expression (5-field, UTC). Used with custom_cron schedule_type.", - "default": null - }, - "deferring_environment_key": { "type": ["string", "null"], "description": "References environments[].key to defer to that environment's most recent successful run", "default": null }, - "deferring_job_key": { "type": ["string", "null"], "description": "References another job[].key to defer to a specific job", "default": null }, - "self_deferring": { "type": ["boolean", "null"], "description": "Defer to this job's own latest successful run", "default": null }, - "generate_docs": { "type": ["boolean", "null"], "description": "Generate dbt docs after run", "default": null }, - "run_lint": { "type": ["boolean", "null"], "description": "Run SQLFluff linting", "default": null }, - "run_generate_sources": { "type": ["boolean", "null"], "description": "Run dbt source freshness", "default": null }, - "run_compare_changes": { "type": ["boolean", "null"], "description": "Enable Advanced CI change comparison", "default": null }, - "errors_on_lint_failure": { "type": ["boolean", "null"], "description": "Fail job if linting errors are found", "default": null }, - "triggers_on_draft_pr": { "type": ["boolean", "null"], "description": "Allow webhook triggers on draft PRs", "default": null }, - "is_active": { "type": ["boolean", "null"], "description": "Whether this job is active", "default": true } - }, - "required": ["name", "key", "environment_key", "execute_steps", "triggers"], - "additionalProperties": false - } + "description": "Extended attributes for environment-level connection overrides (referenced by environment.extended_attributes_key).", + "items": { "$ref": "#/$defs/extendedAttributes" }, + "default": [] + }, + "profiles": { + "type": "array", + "description": "Profiles linking connection and credentials for deployment environments.", + "items": { "$ref": "#/$defs/profile" }, + "default": [] }, - "environment_variables": { "type": "array", - "description": "Project-level dbt environment variables", - "items": { - "type": "object", - "properties": { - "name": { - "type": "string", - "description": "Variable name. Must start with DBT_.", - "pattern": "^DBT_[A-Z0-9_]+$", - "minLength": 1 - }, - "environment_values": { - "type": "array", - "description": "Per-environment values. Use env: 'project' for the project-level default.", - "minItems": 1, - "items": { - "type": "object", - "properties": { - "env": { "type": "string", "description": "Environment name or 'project' for the project-level default", "minLength": 1 }, - "value": { "type": "string", "description": "Variable value. Prefix with 'secret_' to reference a token_map key.", "minLength": 0 } - }, - "required": ["env", "value"], - "additionalProperties": false - } - } - }, - "required": ["name", "environment_values"], - "additionalProperties": false - } + "items": { "$ref": "#/$defs/environmentVariable" } }, - - "profiles": { + "jobs": { "type": "array", - "description": "dbt Cloud IDE connection profiles", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Profile display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key for this profile", "pattern": "^[a-z0-9_]+$" }, - "connection_key": { "type": "string", "description": "References global_connections[].key" }, - "credential_key": { "type": "string", "description": "Composite key (project_key_env_key) or environment key referencing the credential to use" }, - "extended_attributes_key": { "type": ["string", "null"], "description": "References extended_attributes[].key for adapter overrides", "default": null } - }, - "required": ["name", "key", "connection_key", "credential_key"], - "additionalProperties": false - } + "items": { "$ref": "#/$defs/job" } + }, + "notifications": { + "type": "array", + "items": { "$ref": "#/$defs/notificationTarget" } + }, + "project_artefacts": { + "type": "object", + "description": "Docs and freshness job links (modules/project_artefacts).", + "additionalProperties": true + }, + "semantic_layer_config": { + "type": "object", + "description": "Semantic layer target environment (modules/semantic_layer).", + "additionalProperties": true }, - "lineage_integrations": { "type": "array", - "description": "External lineage integrations (e.g. Tableau)", - "items": { - "type": "object", - "properties": { - "name": { "type": "string", "description": "Integration display name", "minLength": 1 }, - "key": { "type": "string", "description": "Unique key. Auth token supplied via var.lineage_tokens[project_key_integration_key].", "pattern": "^[a-z0-9_]+$" }, - "host": { "type": "string", "description": "Integration host URL (e.g. https://tableau.example.com)" }, - "site_id": { "type": ["string", "null"], "description": "[Tableau] Tableau site ID", "default": null }, - "token_name": { "type": ["string", "null"], "description": "[Tableau] Tableau personal access token name", "default": null } - }, - "required": ["name", "key", "host"], - "additionalProperties": false - } + "description": "Tableau / Looker lineage integrations (modules/lineage_integrations).", + "items": { "type": "object", "additionalProperties": true } }, - - "artefacts": { + "id": { + "type": ["integer", "null"], + "description": "Optional source project ID for migration metadata.", + "minimum": 1 + } + }, + "required": ["name", "repository", "environments"], + "additionalProperties": false + }, + "connection": { + "type": "object", + "description": "Connection metadata referenced by environments.", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "minLength": 1 + }, + "type": { + "type": "string", + "description": "Data platform type (e.g., snowflake, databricks, bigquery)" + }, + "id": { + "type": ["integer", "null"], + "minimum": 1 + }, + "private_link_endpoint_key": { + "type": ["string", "null"], + "description": "Reference to PrivateLink endpoint if applicable." + }, + "details": { "type": "object", - "description": "Project artefact job assignments for docs and source freshness", - "properties": { - "docs_job": { "type": "string", "description": "References jobs[].key for the job that generates dbt docs" }, - "freshness_job": { "type": "string", "description": "References jobs[].key for the job that runs source freshness" } - }, - "additionalProperties": false + "description": "Provider-specific connection details.", + "additionalProperties": true }, - - "semantic_layer": { + "protected": { + "type": ["boolean", "null"], + "description": "When true, Terraform uses lifecycle.prevent_destroy on this global connection (matches modules/global_connections and root check global_connections_protected).", + "default": false + } + }, + "required": ["key", "name", "type"], + "additionalProperties": false + }, + "serviceToken": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "minLength": 1 + }, + "permissions": { + "type": "array", + "description": "RBAC blocks for dbtcloud_service_token (permission_set, all_projects, project_key, etc.).", + "items": { "type": "object", "additionalProperties": true } + }, + "id": { + "type": ["integer", "null"], + "minimum": 1 + }, + "description": { + "type": ["string", "null"], + "default": null + }, + "protected": { + "type": ["boolean", "null"], + "default": false + }, + "state": { + "type": ["integer", "null"], + "enum": [1, 2, null] + } + }, + "required": ["key", "name", "permissions"], + "additionalProperties": false + }, + "group": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "name": { + "type": "string", + "minLength": 1 + }, + "members": { + "type": "array", + "items": { "type": "string" }, + "description": "User emails or IDs." + }, + "group_permissions": { + "type": "array", + "items": { "type": "object", "additionalProperties": true } + }, + "assign_by_default": { + "type": ["boolean", "null"], + "default": false + }, + "sso_mapping_groups": { + "type": "array", + "items": { "type": "string" } + }, + "protected": { + "type": ["boolean", "null"], + "default": false + }, + "id": { + "type": ["integer", "null"], + "minimum": 1 + } + }, + "required": ["key", "name"], + "additionalProperties": false + }, + "privateLinkEndpoint": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "cloud": { "type": "string" }, + "region": { "type": "string" }, + "endpoint_id": { "type": "string" } + }, + "required": ["key", "cloud", "region", "endpoint_id"], + "additionalProperties": false + }, + "notificationTarget": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "type": { + "type": "string", + "enum": ["slack", "email", "pagerduty", "webhook"] + }, + "target": { "type": "object", - "description": "Semantic Layer configuration", - "properties": { - "environment_key": { "type": "string", "description": "References environments[].key for the Semantic Layer environment" } - }, - "required": ["environment_key"], - "additionalProperties": false + "description": "Provider-specific payload (channel, address, webhook URL, etc.)", + "additionalProperties": true + }, + "id": { + "type": ["integer", "null"], + "minimum": 1 } }, - - "required": ["name", "key", "repository", "environments"], + "required": ["key", "type", "target"], + "additionalProperties": false + }, + "semanticLayerConfig": { + "type": "object", + "properties": { + "key": { "$ref": "#/$defs/slug" }, + "project_key": { "$ref": "#/$defs/slug" }, + "enabled": { "type": "boolean" }, + "settings": { + "type": "object", + "additionalProperties": true + } + }, + "required": ["key", "project_key", "enabled"], + "additionalProperties": false + }, + "placeholder": { + "type": "object", + "properties": { + "id": { "$ref": "#/$defs/lookupString" }, + "description": { + "type": "string", + "minLength": 1 + } + }, + "required": ["id", "description"], "additionalProperties": false } } } + diff --git a/tests/fixtures/basic.yml b/tests/fixtures/basic.yml index 3d35519..3965d47 100644 --- a/tests/fixtures/basic.yml +++ b/tests/fixtures/basic.yml @@ -1,23 +1,43 @@ -project: - name: My Test Project - key: my_project - environments: - - name: Production - key: prod - type: deployment - deployment_type: production +version: 1 +account: + name: Test Account + host_url: https://cloud.getdbt.com +globals: + connections: + - name: Test Warehouse + key: test_wh + type: snowflake protected: true - jobs: - - name: Daily Run - key: daily_run - environment_key: prod - execute_steps: - - dbt build - triggers: - schedule: true - github_webhook: false - git_provider_webhook: false - on_merge: false - schedule_type: days_of_week - schedule_days: [0, 1, 2, 3, 4] - schedule_hours: [6] + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH +projects: + - name: My Test Project + key: my_project + repository: + remote_url: acme/analytics + environments: + - name: Production + key: prod + type: deployment + deployment_type: production + protected: true + connection: test_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS + jobs: + - name: Daily Run + key: daily_run + environment_key: prod + execute_steps: + - dbt build + triggers: + schedule: true + github_webhook: false + git_provider_webhook: false + on_merge: false + schedule_type: days_of_week + schedule_days: [0, 1, 2, 3, 4] + schedule_hours: [6] diff --git a/tests/fixtures/complete.yml b/tests/fixtures/complete.yml index 2720482..24d0754 100644 --- a/tests/fixtures/complete.yml +++ b/tests/fixtures/complete.yml @@ -1,15 +1,39 @@ +version: 1 +account: + name: Test Account + host_url: https://cloud.getdbt.com +globals: + connections: + - name: Shared Warehouse + key: shared_wh + type: snowflake + protected: true + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH projects: - name: Analytics key: analytics + repository: + remote_url: acme/analytics environments: - name: Development key: dev type: development + connection: shared_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS - name: Production key: prod type: deployment deployment_type: production protected: true + connection: shared_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS jobs: - name: CI Check key: ci_check @@ -36,9 +60,15 @@ projects: - name: Finance key: finance protected: true + repository: + remote_url: acme/finance environments: - name: Production key: prod type: deployment deployment_type: production protected: true + connection: shared_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS diff --git a/tests/fixtures/integration-core.yml b/tests/fixtures/integration-core.yml index 3a5c5c1..a3fee05 100644 --- a/tests/fixtures/integration-core.yml +++ b/tests/fixtures/integration-core.yml @@ -1,10 +1,30 @@ +version: 1 +account: + name: TF Integration + host_url: https://cloud.getdbt.com +globals: + connections: + - name: TF Int WH + key: tf_int_wh + type: snowflake + protected: true + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH projects: - name: TF Integration Core key: tf_int_core + repository: + remote_url: org/tf-int-core environments: - name: Dev key: dev type: development + connection: tf_int_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS jobs: - name: Daily Run key: daily_run diff --git a/tests/fixtures/integration-multi-project.yml b/tests/fixtures/integration-multi-project.yml index 927751c..789a517 100644 --- a/tests/fixtures/integration-multi-project.yml +++ b/tests/fixtures/integration-multi-project.yml @@ -1,10 +1,30 @@ +version: 1 +account: + name: TF Integration Multi + host_url: https://cloud.getdbt.com +globals: + connections: + - name: TF Int WH + key: tf_int_wh + type: snowflake + protected: true + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH projects: - name: TF Integration Alpha key: tf_int_alpha + repository: + remote_url: org/tf-int-alpha environments: - name: Dev key: dev type: development + connection: tf_int_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS jobs: - name: Nightly key: nightly @@ -22,7 +42,13 @@ projects: - name: TF Integration Beta key: tf_int_beta + repository: + remote_url: org/tf-int-beta environments: - name: Dev key: dev type: development + connection: tf_int_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS diff --git a/tests/fixtures/integration-service-tokens.yml b/tests/fixtures/integration-service-tokens.yml index f60de4d..cbf6e75 100644 --- a/tests/fixtures/integration-service-tokens.yml +++ b/tests/fixtures/integration-service-tokens.yml @@ -1,10 +1,33 @@ +version: 1 +account: + name: TF Integration Svc Acct + host_url: https://cloud.getdbt.com +globals: + connections: + - key: tf_int_wh + name: TF Int WH + type: snowflake + protected: true + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH + service_tokens: + - name: TF Integration Token + key: tf_int_svc_token + permissions: + - permission_set: readonly + all_projects: true projects: - name: TF Integration Svc key: tf_int_svc - -service_tokens: - - name: TF Integration Token - key: tf_int_svc_token - permissions: - - permission_set: readonly - all_projects: true + repository: + remote_url: org/repo + environments: + - name: Dev + key: dev + type: development + connection: tf_int_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS diff --git a/tests/fixtures/v1_minimal.yml b/tests/fixtures/v1_minimal.yml new file mode 100644 index 0000000..eabcd13 --- /dev/null +++ b/tests/fixtures/v1_minimal.yml @@ -0,0 +1,49 @@ +# Minimal version: 1 layout — globals.connections + projects[] (normalized at Terraform root). +version: 1 +account: + name: Test Account + host_url: https://cloud.getdbt.com +globals: + connections: + - key: test_wh + name: Test Warehouse + type: snowflake + protected: true + details: + account: xy12345 + database: ANALYTICS + warehouse: COMPUTE_WH +projects: + - name: My Test Project + key: my_project + repository: + remote_url: acme/analytics + environments: + - name: Production + key: prod + type: deployment + deployment_type: production + protected: true + connection: test_wh + credential: + token_name: SNOWFLAKE_PASSWORD + schema: ANALYTICS + jobs: + - name: Daily Run + key: daily_run + environment_key: prod + execute_steps: + - dbt build + triggers: + schedule: true + github_webhook: false + git_provider_webhook: false + on_merge: false + schedule_type: days_of_week + schedule_days: [0, 1, 2, 3, 4] + schedule_hours: [6] + environment_variables: + - name: DBT_V1_MAP_FORM + environment_values: + prod: "map_value" + project: "project_default" diff --git a/tests/root.tftest.hcl b/tests/root.tftest.hcl index c395a34..3954e30 100644 --- a/tests/root.tftest.hcl +++ b/tests/root.tftest.hcl @@ -16,7 +16,7 @@ variables { yaml_file = "tests/fixtures/basic.yml" } -# ── Single-project YAML (project: key) ─────────────────────────────────────── +# ── Single-project YAML (version: 1) ───────────────────────────────────────── run "single_project_yaml_produces_one_project" { command = plan @@ -128,3 +128,38 @@ run "multi_project_jobs_keyed_correctly" { error_message = "Expected composite key 'analytics_daily_run' in job_ids" } } + +# ── version: 1 YAML (globals.* normalized at root) ───────────────────────────── + +run "v1_yaml_flattens_globals_and_reports_schema_version" { + command = plan + + variables { + yaml_file = "tests/fixtures/v1_minimal.yml" + } + + assert { + condition = output.yaml_schema_version == 1 + error_message = "Expected yaml_schema_version output 1 for version: 1 fixture" + } + + assert { + condition = output.yaml_account != null && output.yaml_account.host_url == "https://cloud.getdbt.com" + error_message = "Expected yaml_account.host_url from version: 1 YAML" + } + + assert { + condition = contains(keys(output.project_ids), "my_project") + error_message = "Expected project key my_project after globals normalization" + } + + assert { + condition = contains(keys(output.environment_ids), "my_project_prod") + error_message = "Expected environment composite key my_project_prod" + } + + assert { + condition = contains(keys(output.job_ids), "my_project_daily_run") + error_message = "Expected job composite key my_project_daily_run" + } +} diff --git a/validation.tf b/validation.tf index 955937a..03ea238 100644 --- a/validation.tf +++ b/validation.tf @@ -7,10 +7,16 @@ ############################################# locals { + # ── V-00a: YAML must declare version: 1 ───────────────────────────────────── + + _errors_yaml_version = try(local._raw_yaml.version, null) == 1 ? [] : [ + "YAML must set version: 1. See schemas/v1.json.", + ] + # ── Index locals: pre-computed key sets for cross-reference lookups ──────── _valid_global_connection_keys = toset([ - for c in try(local.yaml_content.global_connections, []) : try(c.key, c.name) + for c in try(local.yaml_content.global_connections, []) : c.key ]) _valid_project_keys = toset([ @@ -18,7 +24,7 @@ locals { ]) _valid_group_keys = toset([ - for g in try(local.yaml_content.groups, []) : try(g.key, g.name) + for g in try(local.yaml_content.groups, []) : g.key ]) # Environment keys per project: { project_key => set(env_key) } @@ -37,6 +43,16 @@ locals { ]) } + _artefact_docs_job_by_project = { + for p in local.projects : + try(p.key, p.name) => try(p.project_artefacts.docs_job_key, null) + } + + _artefact_freshness_job_by_project = { + for p in local.projects : + try(p.key, p.name) => try(p.project_artefacts.freshness_job_key, null) + } + # Extended attribute keys per project: { project_key => set(ea_key) } _ea_keys_by_project = { for p in local.projects : @@ -45,6 +61,14 @@ locals { ]) } + # Profile keys per project: { project_key => set(profile_key) } + _profile_keys_by_project = { + for p in local.projects : + try(p.key, p.name) => toset([ + for prof in try(p.profiles, []) : try(prof.key, prof.name) + ]) + } + _valid_credential_types = toset([ "databricks", "snowflake", "bigquery", "redshift", "postgres", "athena", "fabric", "synapse", "starburst", "trino", @@ -53,17 +77,92 @@ locals { _valid_connection_types = toset([ "databricks", "snowflake", "bigquery", "redshift", "postgres", - "spark", "starburst_trino", "apache_spark", "athena", "fabric", "synapse", + "spark", "starburst", "starburst_trino", "apache_spark", "athena", "fabric", "synapse", "teradata", "salesforce", + ]) + + _privatelink_endpoint_keys = toset([ + for ple in try(local.yaml_content.privatelink_endpoints, []) : ple.key + ]) + + # ── V-00: environments need connection or primary_profile_key ─────────────── + + _errors_env_connection_or_profile = flatten([ + for p in local.projects : [ + for env in try(p.environments, []) : + ( + try(env.connection, null) == null && ( + try(env.primary_profile_key, null) == null || try(env.primary_profile_key, "") == "" + ) + ) ? [ + "Environment '${try(env.key, env.name)}' in project '${try(p.key, p.name)}' must set connection (global connection key, id, or LOOKUP:…), or set primary_profile_key to a profiles[].key." + ] : [] + ] + ]) + + # ── V-00b: primary_profile_key → profiles[].key (same project) ───────────── + + _errors_primary_profile_key = flatten([ + for p in local.projects : [ + for env in try(p.environments, []) : + try( + env.primary_profile_key != null && env.primary_profile_key != "" && !contains( + try(local._profile_keys_by_project[try(p.key, p.name)], toset([])), + env.primary_profile_key + ) + ? ["Environment '${try(env.key, env.name)}' in project '${try(p.key, p.name)}' references primary_profile_key '${env.primary_profile_key}' but no profile with that key exists. Available profile keys: [${join(", ", tolist(try(local._profile_keys_by_project[try(p.key, p.name)], toset([]))))}]"] + : [], + [] + ) + ] ]) - # ── V-01: connection_key in environments → global_connections[].key ──────── + # ── V-01: connection in environments → global_connections[].key (skip LOOKUP:…) _errors_connection_key = flatten([ for p in local.projects : [ for env in try(p.environments, []) : try( - env.connection_key != null && !contains(local._valid_global_connection_keys, env.connection_key) - ? ["Environment '${try(env.key, env.name)}' in project '${try(p.key, p.name)}' references connection_key '${env.connection_key}' but no global_connection with that key exists. Available keys: [${join(", ", tolist(local._valid_global_connection_keys))}]"] + ( + try(env.primary_profile_key, null) == null || try(env.primary_profile_key, "") == "" + ) && ( + try(env.connection, null) != null && + !startswith(tostring(env.connection), "LOOKUP:") && + !contains(local._valid_global_connection_keys, env.connection) + ) + ? ["Environment '${try(env.key, env.name)}' in project '${try(p.key, p.name)}' references connection '${env.connection}' but no global_connection with that key exists. Available keys: [${join(", ", tolist(local._valid_global_connection_keys))}]"] + : [], + [] + ) + ] + ]) + + # ── V-01b: connection_key on profiles → global_connections[].key (skip LOOKUP:…) + + _errors_profile_connection_key = flatten([ + for p in local.projects : [ + for prof in try(p.profiles, []) : + try( + try(prof.connection_key, null) != null && + !startswith(tostring(prof.connection_key), "LOOKUP:") && + !contains(local._valid_global_connection_keys, prof.connection_key) + ? ["Profile '${try(prof.key, prof.name)}' in project '${try(p.key, p.name)}' references connection_key '${prof.connection_key}' but no global_connection with that key exists. Available keys: [${join(", ", tolist(local._valid_global_connection_keys))}]"] + : [], + [] + ) + ] + ]) + + # ── V-01c: extended_attributes_key on profiles → extended_attributes[].key + + _errors_profile_ea_key = flatten([ + for p in local.projects : [ + for prof in try(p.profiles, []) : + try( + prof.extended_attributes_key != null && prof.extended_attributes_key != "" && !contains( + try(local._ea_keys_by_project[try(p.key, p.name)], toset([])), + prof.extended_attributes_key + ) + ? ["Profile '${try(prof.key, prof.name)}' in project '${try(p.key, p.name)}' references extended_attributes_key '${prof.extended_attributes_key}' but no extended_attribute with that key exists. Available keys: [${join(", ", tolist(try(local._ea_keys_by_project[try(p.key, p.name)], toset([]))))}]"] : [], [] ) @@ -92,7 +191,7 @@ locals { for p in local.projects : [ for env in try(p.environments, []) : try( - env.extended_attributes_key != null && !contains( + env.extended_attributes_key != null && env.extended_attributes_key != "" && !contains( try(local._ea_keys_by_project[try(p.key, p.name)], toset([])), env.extended_attributes_key ) @@ -103,6 +202,19 @@ locals { ] ]) + # ── V-03b: each extended_attributes[] must supply non-empty extended_attributes object + + _errors_ea_payload = flatten([ + for p in local.projects : [ + for ea in try(p.extended_attributes, []) : + ( + try(ea.extended_attributes, null) == null || length(keys(try(ea.extended_attributes, {}))) == 0 + ) ? [ + "Project '${try(p.key, p.name)}' extended_attributes entry '${try(ea.key, ea.name)}' must set non-empty 'extended_attributes'." + ] : [] + ] + ]) + # ── V-04: deferring_environment_key in jobs → environments[].key ─────────── _errors_deferring_env_key = flatten([ @@ -120,23 +232,52 @@ locals { ] ]) - # ── V-05: artefacts.docs_job / freshness_job → jobs[].key ───────────────── + # ── V-05: project_artefacts job keys → jobs[].key ─────────────────────────── _errors_artefact_job_keys = flatten([ - for p in local.projects : try(p.artefacts, null) != null ? concat( - try(p.artefacts.docs_job, null) != null && !contains( + for p in local.projects : + try(p.project_artefacts, null) != null + ? concat( + local._artefact_docs_job_by_project[try(p.key, p.name)] != null && !contains( try(local._job_keys_by_project[try(p.key, p.name)], toset([])), - p.artefacts.docs_job + local._artefact_docs_job_by_project[try(p.key, p.name)] ) ? [ - "Project '${try(p.key, p.name)}' artefacts.docs_job references job key '${p.artefacts.docs_job}' which does not exist. Available job keys: [${join(", ", tolist(try(local._job_keys_by_project[try(p.key, p.name)], toset([]))))}]" + "Project '${try(p.key, p.name)}' project_artefacts docs_job_key references key '${local._artefact_docs_job_by_project[try(p.key, p.name)]}' which does not exist. Available job keys: [${join(", ", tolist(try(local._job_keys_by_project[try(p.key, p.name)], toset([]))))}]" ] : [], - try(p.artefacts.freshness_job, null) != null && !contains( + local._artefact_freshness_job_by_project[try(p.key, p.name)] != null && !contains( try(local._job_keys_by_project[try(p.key, p.name)], toset([])), - p.artefacts.freshness_job + local._artefact_freshness_job_by_project[try(p.key, p.name)] ) ? [ - "Project '${try(p.key, p.name)}' artefacts.freshness_job references job key '${p.artefacts.freshness_job}' which does not exist. Available job keys: [${join(", ", tolist(try(local._job_keys_by_project[try(p.key, p.name)], toset([]))))}]" + "Project '${try(p.key, p.name)}' project_artefacts freshness_job_key references key '${local._artefact_freshness_job_by_project[try(p.key, p.name)]}' which does not exist. Available job keys: [${join(", ", tolist(try(local._job_keys_by_project[try(p.key, p.name)], toset([]))))}]" ] : [], - ) : [] + ) + : [] + ]) + + # ── V-05b: semantic_layer_config must resolve an environment ──────────────── + + _errors_semantic_layer_env = flatten([ + for p in local.projects : + try(p.semantic_layer_config, null) != null + ? ( + try(p.semantic_layer_config.environment_id, null) != null + ? [] + : length(compact([ + try(p.semantic_layer_config.environment_key, null), + try(p.semantic_layer_config.environment, null), + ])) == 0 + ? ["Project '${try(p.key, p.name)}' has semantic_layer_config but no environment_id and no environment_key / environment to resolve against environments[].key."] + : !contains( + try(local._env_keys_by_project[try(p.key, p.name)], toset([])), + coalesce( + try(p.semantic_layer_config.environment_key, null), + try(p.semantic_layer_config.environment, null), + ) + ) ? [ + "Project '${try(p.key, p.name)}' semantic_layer_config references environment '${coalesce(try(p.semantic_layer_config.environment_key, null), try(p.semantic_layer_config.environment, null))}' which does not exist. Available environment keys: [${join(", ", tolist(try(local._env_keys_by_project[try(p.key, p.name)], toset([]))))}]" + ] : [] + ) + : [] ]) # ── V-06: Every project must have a name ────────────────────────────────── @@ -185,10 +326,35 @@ locals { _errors_connection_type = [ for conn in try(local.yaml_content.global_connections, []) : - "Global connection '${try(conn.key, conn.name)}' has type '${try(conn.type, "")}' which is not a recognized warehouse type. Valid types: [${join(", ", tolist(local._valid_connection_types))}]" + "Global connection '${conn.key}' has type '${try(conn.type, "")}' which is not a recognized warehouse type. Valid types: [${join(", ", tolist(local._valid_connection_types))}]" if !contains(local._valid_connection_types, try(conn.type, "")) ] + # ── V-10b: global_connections[].private_link_endpoint_key → privatelink_endpoints[].key ─ + + _errors_connection_privatelink_key = [ + for conn in try(local.yaml_content.global_connections, []) : + "Global connection '${conn.key}' references private_link_endpoint_key '${conn.private_link_endpoint_key}' but no privatelink_endpoints[] entry has that key. Define globals.privatelink_endpoints or set private_link_endpoint_id." + if( + try(conn.private_link_endpoint_key, null) != null && + try(conn.private_link_endpoint_id, null) == null && + !contains(local._privatelink_endpoint_keys, conn.private_link_endpoint_key) + ) + ] + + # ── V-10c: project.repository.private_link_endpoint_key → privatelink_endpoints[].key ─ + + _errors_repository_privatelink_key = [ + for p in local.projects : + "Project '${try(p.key, p.name)}' repository references private_link_endpoint_key '${p.repository.private_link_endpoint_key}' but no privatelink_endpoints[] entry has that key. Define privatelink_endpoints at the YAML root or set private_link_endpoint_id." + if( + try(p.repository, null) != null && + try(p.repository.private_link_endpoint_key, null) != null && + try(p.repository.private_link_endpoint_id, null) == null && + !contains(local._privatelink_endpoint_keys, p.repository.private_link_endpoint_key) + ) + ] + # ── V-11: schedule coherence — schedule:true requires schedule_type or cron ─ _errors_schedule_config = flatten([ @@ -218,7 +384,7 @@ locals { _errors_service_token_project_keys = flatten([ for st in try(local.yaml_content.service_tokens, []) : [ for perm in try(st.permissions, []) : - "Service token '${try(st.key, st.name)}' has a permission referencing project_key '${perm.project_key}' which is not a defined project. Available project keys: [${join(", ", tolist(local._valid_project_keys))}]" + "Service token '${st.key}' has a permission referencing project_key '${perm.project_key}' which is not a defined project. Available project keys: [${join(", ", tolist(local._valid_project_keys))}]" if( try(perm.project_key, null) != null && !contains(local._valid_project_keys, perm.project_key) @@ -229,16 +395,25 @@ locals { # ── Aggregated error list ────────────────────────────────────────────────── _all_validation_errors = compact(concat( + local._errors_yaml_version, + local._errors_env_connection_or_profile, + local._errors_primary_profile_key, local._errors_connection_key, + local._errors_profile_connection_key, + local._errors_profile_ea_key, local._errors_job_env_key, local._errors_ea_key, + local._errors_ea_payload, local._errors_deferring_env_key, local._errors_artefact_job_keys, + local._errors_semantic_layer_env, local._errors_project_name, local._errors_deployment_type, local._errors_execute_steps, local._errors_credential_type, local._errors_connection_type, + local._errors_connection_privatelink_key, + local._errors_repository_privatelink_key, local._errors_schedule_config, local._errors_user_group_keys, local._errors_service_token_project_keys, @@ -297,9 +472,9 @@ check "global_connections_protected" { assert { condition = length([ for c in try(local.yaml_content.global_connections, []) : - try(c.key, c.name) + c.key if !try(c.protected, false) ]) == 0 - error_message = "Best practice: the following global connections have protected: false. Deleting a connection detaches all environments that reference it. Add protected: true to prevent accidental removal. Connections: ${join(", ", [for c in try(local.yaml_content.global_connections, []) : try(c.key, c.name) if !try(c.protected, false)])}" + error_message = "Best practice: the following global connections have protected: false. Deleting a connection detaches all environments that reference it. Add protected: true to prevent accidental removal. Connections: ${join(", ", [for c in try(local.yaml_content.global_connections, []) : c.key if !try(c.protected, false)])}" } } diff --git a/variables.tf b/variables.tf index 522397b..fa3cbfc 100644 --- a/variables.tf +++ b/variables.tf @@ -33,7 +33,7 @@ variable "dbt_pat" { } variable "dbt_host_url" { - description = "dbt Cloud host URL (e.g., https://cloud.getdbt.com or custom domain)" + description = "dbt Cloud host URL (e.g., https://cloud.getdbt.com or custom domain). Required by the Terraform dbtcloud provider; version: 1 YAML account.host_url is used only for HTTP lookups (module data_lookups) when this variable is null — mirror account.host_url here for real applies." type = string default = null @@ -48,7 +48,7 @@ variable "dbt_host_url" { ############################################# variable "yaml_file" { - description = "Path to the YAML file defining dbt Cloud resources (projects, environments, jobs, etc.)" + description = "Path to the YAML file defining dbt Cloud resources. Must set version: 1 with account, globals.* (connections, optional groups, service_tokens, notifications, privatelink_endpoints), and projects[] — see schemas/v1.json. Root locals hoist globals into top-level keys modules consume and normalize environment_variables[].environment_values from maps to lists." type = string validation { @@ -73,7 +73,7 @@ variable "target_name" { ############################################# variable "token_map" { - description = "Map of credential token names to their actual values (e.g., Databricks tokens). Token names correspond to credential.token_name in YAML." + description = "Map of token names to secret values. Used for legacy Databricks credential.token_name in YAML and for jobs[].environment_variable_overrides values prefixed with secret_ (lookup key is the string after the prefix)." type = map(string) default = {} sensitive = true @@ -87,7 +87,7 @@ variable "connection_credentials" { } variable "environment_credentials" { - description = "Map of environment credential keys to their warehouse credential objects. Key format: project_key_env_key. Supports 14 warehouse types via credential_type field." + description = "Map of credential keys to warehouse credential objects. Key format: project_key_env_key for environments, or project_key_profile_key for standalone profile-owned credentials. Supports 14 warehouse types via credential_type field." type = map(any) default = {} sensitive = true @@ -117,18 +117,88 @@ variable "enable_gitlab_deploy_token" { default = false } +variable "skip_global_project_permissions" { + description = "When true, account-level group permissions from YAML are applied as all_projects-only blocks so Terraform does not add edges to project resources (scoped adoption of globals)." + type = bool + default = false +} + ############################################# # Locals ############################################# locals { - yaml_content = yamldecode(file(var.yaml_file)) - - # Support both single project (project:) and multi-project (projects:) YAML shapes. - # Single-project users keep their existing YAML unchanged — the project: key is - # automatically wrapped into a one-element list. - projects = try( - local.yaml_content.projects, - [local.yaml_content.project] + _raw_yaml = yamldecode(file(var.yaml_file)) + + # version: 1 — hoist globals into the top-level keys root modules read. + yaml_content = merge( + { + for k, v in local._raw_yaml : k => v + if !contains([ + "version", + "account", + "globals", + "metadata", + "projects", + "global_connections", + "privatelink_endpoints", + "service_tokens", + "groups", + "notifications", + ], k) + }, + { + global_connections = try(local._raw_yaml.globals.connections, []) + privatelink_endpoints = try(local._raw_yaml.globals.privatelink_endpoints, []) + projects = local._raw_yaml.projects + service_tokens = try(local._raw_yaml.globals.service_tokens, []) + groups = try(local._raw_yaml.globals.groups, []) + notifications = try(local._raw_yaml.globals.notifications, []) + }, + ) + + # environment_variables[].environment_values: map env_key → value (YAML) → list of { env, value } for modules. + projects = [ + for p in local.yaml_content.projects : merge(p, { + environment_variables = [ + for ev in try(p.environment_variables, []) : merge(ev, { + environment_values = [ + for k, v in try(tomap(ev.environment_values), tomap({})) : { env = k, value = tostring(v) } + ] + }) + ] + }) + ] + + # HTTP helpers (module data_lookups): var.dbt_host_url, then version: 1 account.host_url, then public default (matches modules/data_lookups). + dbt_host_url_effective = coalesce( + var.dbt_host_url, + try(local._raw_yaml.account.host_url, null) != null && try(trimspace(tostring(local._raw_yaml.account.host_url)), "") != "" ? trimspace(tostring(local._raw_yaml.account.host_url)) : null, + "https://cloud.getdbt.com", + ) + + # Gating for module.data_lookups — keep in sync with modules/data_lookups LOOKUP extraction. + _lookup_connection_ref_strings = toset([ + for conn_ref in flatten([ + for p in local.projects : concat( + [ + for env in try(p.environments, []) : + try(env.connection, null) + if try(env.connection, null) != null && startswith(tostring(env.connection), "LOOKUP:") + ], + [ + for prof in try(p.profiles, []) : + try(prof.connection_key, null) + if try(prof.connection_key, null) != null && startswith(tostring(prof.connection_key), "LOOKUP:") + ] + ) + ]) : + tostring(conn_ref) if startswith(tostring(conn_ref), "LOOKUP:") + ]) + + # Merged map for environments/profiles: Terraform-managed global_connections + pre-existing account connections (LOOKUP:…). + global_connection_ids_effective = merge( + length(module.data_lookups) > 0 ? module.data_lookups[0].lookup_connection_ids : {}, + length(try(local.yaml_content.global_connections, [])) > 0 ? module.global_connections[0].connection_ids : {}, ) }