Merge pull request #134 from dbt-labs/import-with-template

Add ability to define field templates when importing jobs

Author: b-per

.pre-commit-config.yaml

@@ -5,6 +5,6 @@ repos:
     hooks:
       # Run the linter.
       - id: ruff
-        args: [ --select, I, --select, B, --fix ]
+        args: [ --fix ]
       # Run the formatter.
       - id: ruff-format

docs/advanced_config/glob_config_files.md

@@ -9,5 +9,7 @@ Those patterns are also called "Unix style pathname pattern expansion", and in a
 For example, to run the `plan` command on all the files stored in subdirectoris under the `jobs` directory, you can use the following command:
 
 ```bash
-dbt-jobs-as-code validate --config-file jobs/**/*.yml
-```
+dbt-jobs-as-code plan "jobs/**/*.yml" # (1)!
+```
+
+1. Depending on your shell you might have to quote the pattern or not. For example, for `zsh` quoting is required as otherwise the shell will try to expand the pattern before passing it to the command.

docs/advanced_config/index.md

@@ -1,6 +1,8 @@
-More advanced features of the tools that can be combined to match more complex requirements in regards to dbt Cloud jobs management.
+# Advanced configuration
 
-- [YAML Templating](templating.md)
-- [glob config files](glob_config_files.md)
-- [YAML anchors](yaml_anchors.md)
-- [Advanced jobs importing](jobs_importing.md)
+More advanced features of `dbt-jobs-as-code` can be combined to match complex requirements in regards to dbt Cloud jobs management.
+
+- [YAML Templating](templating.md) - for using the same YAML file to update different dbt Cloud projects and/or environments
+- [glob config files](glob_config_files.md) - for using glob patterns to match config files at once
+- [YAML anchors](yaml_anchors.md) - to reuse the same parameters in different jobs
+- [Advanced jobs importing](jobs_importing.md) - for importing jobs from dbt Cloud to a YAML file

docs/advanced_config/jobs_importing.md

@@ -1 +1,98 @@
-# WIP
+# Advanced job importing
+
+## Generating jobs with templated fields with `import-jobs`
+
+`plan` and `sync` commands allow adding Jinja variables to the jobs in order to use the same YAML file for different environments (see [YAML templating](templating.md)).
+
+While it is possible to import the jobs from dbt Cloud using the `import-jobs` command and to modify the outputted YAML by hand to add variables, there is also the ability to use a YAML file to specify variables for the different jobs.
+
+By providing the `--templated-fields` parameter, it is possible to use a YAML file to specify the variables for specific fields of the jobs.
+
+For example, the following YAML file:
+
+```yaml title="templ.yml"
+environment_id: {{ environment_id }}
+deferring_environment_id: {{ deferring_environment_id }}
+triggers.schedule: {{ env_name == 'prod' }}
+```
+
+will be used to set the `project_id`, `environment_id`, `deferring_environment_id` and `triggers.schedule` fields for all the jobs in the generated YAML file and can be called with 
+
+```bash
+dbt-jobs-as-code import-jobs --account-id 1234 --project-id 3213 --environment-id 423432 --templated-fields templ.yml --managed-only
+```
+
+The outputted YAML will look like the following:
+
+```yaml title="jobs.yml"
+# yaml-language-server: $schema=https://raw.githubusercontent.com/dbt-labs/dbt-jobs-as-code/main/src/dbt_jobs_as_code/schemas/load_job_schema.json
+
+jobs:
+  my-id:
+    account_id: 1234
+    project_id: 3213
+    environment_id: {{ environment_id }}
+    dbt_version:
+    name: My job name created from the UI
+    settings:
+      threads: 10
+      target_name: default
+    execution:
+      timeout_seconds: 100
+    deferring_job_definition_id:
+    deferring_environment_id: {{ deferring_environment_id }}
+    run_generate_sources: true
+    execute_steps:
+      - dbt build
+    generate_docs: true
+    schedule:
+      cron: 0 1,5 * * 0,1,2,3,4,5
+    triggers:
+      github_webhook: false
+      git_provider_webhook: false
+      schedule: {{ env_name == 'prod' }}
+      on_merge: false
+    description: ''
+    run_compare_changes: false
+    compare_changes_flags: --select state:modified
+    job_type: other
+    triggers_on_draft_pr: false
+    job_completion_trigger_condition:
+    custom_environment_variables: []
+```
+
+
+## Automatically promote jobs between environments
+
+The import command from above can also be used to automatically promote jobs between environments. In that case, as part of a CI/CD process, or on a schedule, the following command can be used to generate the YAML content and save it to a file:
+
+```bash
+dbt-jobs-as-code import-jobs --account-id 1234 --project-id 3213 --environment-id 423432 --templated-fields templ.yml --managed-only > jobs.yml
+```
+
+It would then be possible to automate the creation of PRs whenever the `jobs.yml` file is updated, meaning that some jobs would have been updated in the dbt Cloud UI. The GitHub action [Create Pull Request](https://github.com/marketplace/actions/create-pull-request) could be used to implement this flow.
+
+
+Then, the `jobs.yml` file can be used to import the jobs in a different environment with the following command, like described in [YAML templating](templating.md) and in the [typical flows](../typical_flows.md) page :
+
+```bash
+dbt-jobs-as-code plan jobs.yml -v qa_vars.yml --limit-projects-envs-to-yml
+```
+
+With `qa_vars.yml` being the YAML file containing the variables for the QA/staging environment.
+
+```yaml title="qa_vars.yml"
+env_name: "qa"
+environment_id: 456
+deferring_environment_id: # (1)!
+``` 
+
+1. The `deferring_environment_id` here is set to the null value, we could also set it to a specific ID
+
+And `prod_vars.yml` being the YAML file containing the variables for the Prod environment.
+
+```yaml title="prod_vars.yml"
+env_name: "prod"
+environment_id: 789
+deferring_environment_id: 
+``` 

docs/advanced_config/templating.md

@@ -10,7 +10,7 @@ To add templated values to your jobs YAML file:
 
 The file called in `--vars-yml` needs to be a valid YAML file like the following:
 
-```yaml
+```yaml title="vars_qa.yml"
 project_id: 123
 environment_id: 456
 ```
@@ -102,7 +102,7 @@ Templating also allows people to version control those YAML files and to have di
     deferring_environment_id: 789
     ```
 
-    ``` yaml title="vars_prd.yml"
+    ``` yaml title="vars_prod.yml"
     env_type: prod
     project_id: 12
     environment_id: 231231

docs/changelog.md

@@ -1,6 +1,10 @@
 
 To see the details of all changes, head to the GitHub repo
 
+### 1.4
+
+- Add `--templated-fields` to `import-jobs` to add Jinja variables to the generated YAML file. This can be useful to allow users to maintain jobs in the dbt Cloud UI and set a process to automatically promote those to other environments.
+
 ### 1.3
 
 - Add this docs site

docs/typical_flows.md

@@ -1,4 +1,4 @@
-# Typical flows (WIP)
+# Typical flows
 
 This page descibes how `dbt-jobs-as-code` can be used in different scenarios, what commands to run in what order, what parameters to use, what CI actions to create etc...
 
@@ -282,8 +282,28 @@ sequenceDiagram
 
 ---
 
-## Adanced flows for automation of jobs promotion
+## Advanced flows
 
-### **WIP -- Not implemented yet** // Replicate all jobs from one environment to another (ongoing managed jobs)
+### Automated promotion of jobs created in the UI to other environments
 
-WIP
+In this mechanism, we use `dbt-jobs-as-code` to create a CI/CD pipeline to automatically promote jobs created in the dbt Cloud UI to other environments. It would synchronize jobs and allow creating new ones, deleting old ones and modifying existing ones.
+
+The key part of the process is [the advanced ability to import jobs](advanced_config/jobs_importing.md) while setting Jinja variables that will be rendered differently for each environment.
+
+```mermaid
+sequenceDiagram
+    actor D as dbt Developer
+    participant C as dbt Cloud
+    participant G as git repo
+    actor U as User
+
+    D -->>C: Maintain dbt Cloud jobs in the UI in dev,<br> adding [[..]] to the jobs that<br> need to be promoted/replicated
+    U -->>G: Create and commit the YAML vars files for each env
+    U -->>G: Create and commit the YAML vars files for the templated values
+    U -->>G: Create a pipeline to automatically run `dbt-jbos-as-code<br>on a schedule or triggered by somone
+    G ->> C: run `import-jobs` with `--templated-fields templ.yml`<br>,`--managed-only` and `-e` to specify the env
+    C ->> G: get yaml file
+    Note over G: Check difference with existing file<br> and auto-create PR if it is different
+    Note right of G: When the PR is created, we follow the<br> typical flow of GH actions<br> with plan/sync on PR creation/merge
+
+```

mkdocs.yml

@@ -8,7 +8,7 @@ nav:
   - Home: index.md
   - Getting Started: getting_started.md
   - Advanced configuration:
-    - Index: advanced_config/index.md
+    - advanced_config/index.md
     - YAML Templating: advanced_config/templating.md
     - glob config files: advanced_config/glob_config_files.md
     - Using YAML anchors: advanced_config/yaml_anchors.md
@@ -54,7 +54,10 @@ theme:
     - content.action.edit
     - navigation.tabs
     - toc.integrate # not sure, let's see
+    - toc.follow
     - content.code.copy
+    - content.code.annotate
+    - navigation.indexes
 
 markdown_extensions:
   - mkdocs-click

pyproject.toml

@@ -71,3 +71,8 @@ markers = [
 
 [tool.ruff]
 line-length = 99
+
+[tool.ruff.lint]
+select = ["E", "F", "I"]
+ignore = ["E501"] # line-length, need to fix those by hands first
+typing-modules = ["beartype.typing"]

src/dbt_jobs_as_code/client/__init__.py

@@ -13,7 +13,7 @@
 from dbt_jobs_as_code.schemas.job import JobDefinition, JobMissingFields
 
 if os.getenv("DBT_JOB_ID", "") == "":
-    VERSION = f'v{version("dbt-jobs-as-code")}'
+    VERSION = f"v{version('dbt-jobs-as-code')}"
 else:
     VERSION = "dev"
 
@@ -147,7 +147,7 @@ def get_job(self, job_id: int) -> JobDefinition:
         self._check_for_creds()
 
         response = self._session.get(
-            url=(f"{self.base_url}/api/v2/accounts/" f"{self.account_id}/jobs/{job_id}/"),
+            url=(f"{self.base_url}/api/v2/accounts/{self.account_id}/jobs/{job_id}/"),
             headers=self._headers,
             verify=self._verify,
         )
@@ -162,7 +162,7 @@ def get_job_missing_fields(self, job_id: int) -> Optional[JobMissingFields]:
         self._check_for_creds()
 
         response = self._session.get(
-            url=(f"{self.base_url}/api/v2/accounts/" f"{self.account_id}/jobs/{job_id}/"),
+            url=(f"{self.base_url}/api/v2/accounts/{self.account_id}/jobs/{job_id}/"),
             headers=self._headers,
             verify=self._verify,
         )