feat: store [[identifier]] in job description with --use-desc-for-id by b-per · Pull Request #208 · dbt-labs/dbt-jobs-as-code

b-per · 2026-05-04T11:45:02Z

Summary

Adds --use-desc-for-id flag (env var: DBT_JOBS_AS_CODE_USE_DESC_FOR_ID) that moves the [[identifier]] tag from the job name to the job description, keeping job names clean in the dbt Cloud UI
The YAML format is unchanged — identifiers remain as YAML keys; only the dbt Cloud API storage location changes
Applied to all commands: sync, plan, validate, import_jobs, link, unlink, deactivate_jobs

How it works

Field	Default mode	`--use-desc-for-id` mode
API `name`	`"Daily job [[daily_job]]"`	`"Daily job"`
API `description`	`"Runs nightly"`	`"Runs nightly [[daily_job]]"`

When reading back from the API, the identifier is extracted from whichever field carries it and stripped, so JobDefinition.name and .description are always clean internally.

Migration

To migrate existing jobs from name-mode to desc-mode: run unlink, then link --use-desc-for-id.

Test Plan

211 tests pass (uv run pytest tests/)
New test classes: TestIdentifierExtraction (desc methods), TestToPayloadDescMode, TestPreProcessJobData, TestGetJobsDescMode, TestUpdateCreateDescMode
CLI tests for all 7 commands verify the flag is wired through to DBTCloud/build_change_set
Both ON and OFF mode paths are tested for all client methods

Closes #174

… duplication

When use_desc_for_id=True, embed [[identifier]] at the end of the job description instead of the job name, keeping the name clean in the UI.

…data Adds use_desc_for_id: bool = False parameter to DBTCloud.__init__ (stored as _use_desc_for_id) and implements _pre_process_job_data which moves [[identifier]] tags from job description back to job name for internal processing.

…od, immutability test)

…c_for_id Wire _pre_process_job_data into get_job and get_jobs so that when use_desc_for_id=True, [[identifier]] tags are moved from description back to name before constructing JobDefinition objects.

When use_desc_for_id=True on the DBTCloud client, update_job and create_job now embed [[identifier]] in the job description instead of the name when calling to_payload().

Add use_desc_for_id: bool = False parameter to build_change_set() and forward it to the DBTCloud constructor call inside the function.

…t_jobs, link, unlink Exposes the use_desc_for_id flag as a reusable click option with envvar DBT_JOBS_AS_CODE_USE_DESC_FOR_ID and wires it through to build_change_set() (sync, plan) and DBTCloud() (validate, import_jobs, link, unlink).

… commands Add the missing @option_use_desc_for_id decorator and use_desc_for_id kwarg to DBTCloud() in deactivate_jobs, matching the pattern used by all other commands. Add five new tests (validate, import_jobs, link, unlink, deactivate_jobs) that verify DBTCloud is constructed with use_desc_for_id=True when the flag is passed.

…hen use_desc_for_id When use_desc_for_id=True, the API response has [[id]] in description. Now update_job and create_job call _pre_process_job_data on the response before building the returned JobDefinition, matching get_job's existing behavior.

github-actions · 2026-05-04T11:45:33Z

Coverage Report

File	Stmts	Miss	Cover	Missing
src/dbt_jobs_as_code
main.py	318	94	70%	131–134, 140, 180–181, 214–217, 223, 269, 292–295, 300–303, 314–322, 329–337, 341, 344–346, 419–424, 438–439, 457–458, 462–464, 502, 505, 513–526, 529–532, 573–579, 582, 586, 603, 605, 607, 609, 612–623, 627–628, 668–672, 686, 690, 692, 697–710, 717–718, 722
src/dbt_jobs_as_code/client
__init__.py	207	46	78%	17, 56–57, 65, 126–127, 150–151, 174–175, 190–191, 208–209, 225–226, 245, 255, 335–350, 373, 389–390, 400–412, 415–426, 431–440
src/dbt_jobs_as_code/cloud_yaml_mapping
change_set.py	235	30	87%	27–29, 59, 65–66, 155, 212, 224, 248, 252–253, 259–261, 276–279, 336–350, 403–425, 461–475
src/dbt_jobs_as_code/importer
__init__.py	27	7	74%	14–15, 22–27
src/dbt_jobs_as_code/schemas
__init__.py	23	2	91%	68, 78
common_types.py	60	3	95%	67–68, 91
config.py	14	1	93%	18
job.py	140	7	95%	252, 257, 273–283
TOTAL	1227	190	85%

Tests	Skipped	Failures	Errors	Time
211	0 💤	0 ❌	0 🔥	12.161s ⏱️

savagegipsy · 2026-05-12T03:44:05Z

Hi, @b-per , Thank you for the changes!

I tested them and they do work, but not exactly as I initially expected. I’ll outline my testing steps below and would really appreciate your support in reviewing them:

Current situation:
Our dbt Cloud instance already has over a hundred jobs. We want to introduce this package to manage all job configurations via YAML. To test this, I created a test job in the dbt Cloud Web UI and performed the following steps to simulate the migration:

step 0. set `DBT_JOBS_AS_CODE_USE_DESC_FOR_ID`

 export DBT_JOBS_AS_CODE_USE_DESC_FOR_ID=True

step 1. import & link the test job 1051254 :

dbt-jobs-as-code import-jobs --account-id xxx -j 1051254 --include-linked-id > jobs/test_job.yml

step 2. plan

dbt-jobs-as-code plan jobs/test_job.yml

2026-05-12 11:13:12.302 | DEBUG | dbt_jobs_as_code.client:_build_parameters:273 - Request parameters {'offset': 0}
2026-05-12 11:13:13.800 | DEBUG | dbt_jobs_as_code.client:_build_parameters:273 - Request parameters {'offset': 100}
2026-05-12 11:13:14.253 | DEBUG | dbt_jobs_as_code.client:_build_parameters:273 - Request parameters {'offset': 200}
2026-05-12 11:13:14.691 | INFO | dbt_jobs_as_code.cloud_yaml_mapping.change_set:build_change_set:328 - Detected 0 existing jobs.
2026-05-12 11:13:14.691 | INFO | dbt_jobs_as_code.cloud_yaml_mapping.change_set:build_change_set:358 - Detected 1 new jobs.
2026-05-12 11:13:14.691 | INFO | dbt_jobs_as_code.cloud_yaml_mapping.change_set:build_change_set:373 - Detected 0 deleted jobs.
2026-05-12 11:13:14.691 | DEBUG | dbt_jobs_as_code.cloud_yaml_mapping.change_set:build_change_set:390 - Mapping of job identifier to id: {}
2026-05-12 11:13:14.692 | INFO | dbt_jobs_as_code.main:plan:245 - -- PLAN -- 1 changes detected.
[11:13:14] Changes detected main.py:247
┏━━━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┓
┃ Action ┃ Type ┃ ID ┃ Proj ID ┃ Env ID ┃
┡━━━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━┩
│ CREATE │ Job │ import_1 │ xxxxxx│ xxxxxxxxx│
└────────┴──────┴──────────┴─────────┴────────┘

step 3. sync

dbt-jobs-as-code sync jobs/test_job.yml

Results:

The first job is created by Web UI, and the second one is created by the yaml file.

It felt unexpected when I tried the migration flow.

Suppose import already captured an existing production job whose job id is 1051254 . After I deploy from that same YAML, a second job is created instead of updating the one I imported. When I inspect the result, the job that YAML ends up managing has id 1051260, which does not match the job id that was in the file I originally pulled down with import.

From an infrastructure-as-code perspective, what runs in the cloud should match what’s in the YAML. In my view this is very problematic: during migration it can lead to many duplicate jobs being created.

What I was hoping for under this issue is a workflow like:
import → link → plan (no changes, because it’s still the same job in dbt Cloud) → sync (still no changes).

I suspect others hit the same migration pain. I’d like to hear your take, and whether it would be possible to add an option so users can explicitly choose how to reconcile existing cloud jobs vs jobs defined in YAML (instead of ending up with duplicates or mismatched ids).

b-per · 2026-05-12T16:28:43Z

Hi @savagegipsy, thanks for the detailed report!

The workflow you described at the end is actually the correct one — you just skipped the link step in your test:

export DBT_JOBS_AS_CODE_USE_DESC_FOR_ID=True

# Step 1 — generate the YAML
dbt-jobs-as-code import-jobs --account-id xxx -j 1051254 --include-linked-id > jobs/test_job.yml

# Step 2 — link (this is the step that was missing)
dbt-jobs-as-code link jobs/test_job.yml

# Step 3 — plan now shows 0 changes
dbt-jobs-as-code plan jobs/test_job.yml

# Step 4 — sync makes no changes
dbt-jobs-as-code sync jobs/test_job.yml

import-jobs --include-linked-id is read-only — it saves the cloud job's API ID as linked_id in the YAML so that link knows which cloud job to tag. The link command is what actually writes [[identifier]] into the cloud job's description (in desc-mode). Without that write-back, plan has no way to match your YAML job to the existing cloud job and treats it as new.

Running link before plan will give you exactly the zero-change result you expected. Please let me know how it goes!

savagegipsy · 2026-05-14T06:29:03Z

Hi @savagegipsy, thanks for the detailed report!

The workflow you described at the end is actually the correct one — you just skipped the link step in your test:
export DBT_JOBS_AS_CODE_USE_DESC_FOR_ID=True

# Step 1 — generate the YAML
dbt-jobs-as-code import-jobs --account-id xxx -j 1051254 --include-linked-id > jobs/test_job.yml

# Step 2 — link (this is the step that was missing)
dbt-jobs-as-code link jobs/test_job.yml

# Step 3 — plan now shows 0 changes
dbt-jobs-as-code plan jobs/test_job.yml

# Step 4 — sync makes no changes
dbt-jobs-as-code sync jobs/test_job.yml
import-jobs --include-linked-id is read-only — it saves the cloud job's API ID as linked_id in the YAML so that link knows which cloud job to tag. The link command is what actually writes [[identifier]] into the cloud job's description (in desc-mode). Without that write-back, plan has no way to match your YAML job to the existing cloud job and treats it as new.

Running link before plan will give you exactly the zero-change result you expected. Please let me know how it goes!

@b-per
I retried and the result is exactly as you described. Running link before plan solved the issue and now plan shows 0 changes as expected.

Thanks a lot for the clear explanation and the quick help!

Just wanted to check if there’s any estimated timeline for when this feature might be merged?

b-per · 2026-05-18T14:56:25Z

CI passed

b-per · 2026-05-18T14:57:38Z

@savagegipsy this has been added to 1.18.0

b-per added 14 commits May 4, 2026 11:51

feat: add _extract_identifier_from_description to JobDefinition

6f69a05

refactor: make _extract_identifier_from_description an alias to avoid…

4182e01

… duplication

feat: add use_desc_for_id parameter to to_payload()

ab132d7

When use_desc_for_id=True, embed [[identifier]] at the end of the job description instead of the job name, keeping the name clean in the UI.

refactor: fix misnamed test and extract description length constant

f4d7f41

refactor: improve _pre_process_job_data (targeted replace, staticmeth…

9f19d60

…od, immutability test)

feat: call _pre_process_job_data in get_job and get_jobs when use_des…

363a936

…c_for_id Wire _pre_process_job_data into get_job and get_jobs so that when use_desc_for_id=True, [[identifier]] tags are moved from description back to name before constructing JobDefinition objects.

test: fix off-mode test for get_job and add off-mode test for get_jobs

2a85618

feat: pass use_desc_for_id to to_payload() in update_job and create_job

00de756

When use_desc_for_id=True on the DBTCloud client, update_job and create_job now embed [[identifier]] in the job description instead of the name when calling to_payload().

test: add off-mode test for create_job payload

4ede751

feat: pass use_desc_for_id through build_change_set to DBTCloud

6254866

Add use_desc_for_id: bool = False parameter to build_change_set() and forward it to the DBTCloud constructor call inside the function.

Update docs for new release

95c07a9

b-per merged commit 1645dbc into main May 18, 2026
5 checks passed

b-per deleted the feature/use-desc-for-id branch May 18, 2026 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: store [[identifier]] in job description with --use-desc-for-id#208

feat: store [[identifier]] in job description with --use-desc-for-id#208
b-per merged 15 commits into
mainfrom
feature/use-desc-for-id

b-per commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026 •

edited

Loading

Uh oh!

savagegipsy commented May 12, 2026 •

edited

Loading

Uh oh!

b-per commented May 12, 2026

Uh oh!

savagegipsy commented May 14, 2026 •

edited

Loading

Uh oh!

b-per commented May 18, 2026

Uh oh!

Uh oh!

b-per commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

b-per commented May 4, 2026

Summary

How it works

Migration

Test Plan

Uh oh!

github-actions Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

savagegipsy commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

step 0. set DBT_JOBS_AS_CODE_USE_DESC_FOR_ID

step 1. import & link the test job 1051254 :

step 2. plan

step 3. sync

Results:

Uh oh!

b-per commented May 12, 2026

Uh oh!

savagegipsy commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-per commented May 18, 2026

Uh oh!

Uh oh!

b-per commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 4, 2026 •

edited

Loading

savagegipsy commented May 12, 2026 •

edited

Loading

step 0. set `DBT_JOBS_AS_CODE_USE_DESC_FOR_ID`

savagegipsy commented May 14, 2026 •

edited

Loading