Merge branch 'MagicBot/consolidate-source-package' into MagicBot/fy26-q2-updates

Author: fivetran-catfritz

CHANGELOG.md

@@ -1,21 +1,27 @@
-[PR #142](https://github.com/fivetran/dbt_jira/pull/142) includes the following updates:
+# dbt_jira v1.0.0
 
-### Under the Hood - July 2025 Updates
+[PR #145](https://github.com/fivetran/dbt_jira/pull/145) includes the following updates:
 
+## Breaking Changes
+
+### Source Package Consolidation
+- Removed the dependency on the `fivetran/jira_source` package.
+  - All functionality from the source package has been merged into this transformation package for improved maintainability and clarity.
+  - If you reference `fivetran/jira_source` in your `packages.yml`, you must remove this dependency to avoid conflicts.
+  - Any source overrides referencing the `fivetran/jira_source` package will also need to be removed or updated to reference this package.
+  - Update any jira_source-scoped variables to be scoped to only under this package. See the [README](https://github.com/fivetran/dbt_jira/blob/main/README.md) for how to configure the build schema of staging models.
+- As part of the consolidation, vars are no longer used to reference staging models, and only sources are represented by vars. Staging models are now referenced directly with `ref()` in downstream models.
+
+### dbt Fusion Compatibility Updates
+- Updated package to maintain compatibility with dbt-core versions both before and after v1.10.6, which introduced a breaking change to multi-argument test syntax (e.g., `unique_combination_of_columns`).
+- Temporarily removed unsupported tests to avoid errors and ensure smoother upgrades across different dbt-core versions. These tests will be reintroduced once a safe migration path is available.
+  - Removed all `dbt_utils.unique_combination_of_columns` tests.
+  - Removed all `accepted_values` tests.
+  - Moved `loaded_at_field: _fivetran_synced` under the `config:` block in `src_jira.yml`.
+
+## Under the Hood
 - Updated conditions in `.github/workflows/auto-release.yml`.
 - Added `.github/workflows/generate-docs.yml`.
-- Added `+docs: show: False` to `integration_tests/dbt_project.yml`.
-- Migrated `flags` (e.g., `send_anonymous_usage_stats`, `use_colors`) from `sample.profiles.yml` to `integration_tests/dbt_project.yml`.
-- Updated `maintainer_pull_request_template.md` with improved checklist.
-- Refreshed README tag block:
-  - Standardized Quickstart-compatible badge set
-  - Left-aligned and positioned below the H1 title.
-- Updated Python image version to `3.10.13` in `pipeline.yml`.
-- Added `CI_DATABRICKS_DBT_CATALOG` to:
-  - `.buildkite/hooks/pre-command` (as an export)
-  - `pipeline.yml` (under the `environment` block, after `CI_DATABRICKS_DBT_TOKEN`)
-- Added `certifi==2025.1.31` to `requirements.txt` (if missing).
-- Updated `.gitignore` to exclude additional DBT, Python, and system artifacts.
 
 # dbt_jira v0.21.0
 

README.md

@@ -16,7 +16,7 @@
 </p>
 
 ## What does this dbt package do?
-- Produces modeled tables that leverage Jira data from [Fivetran's connector](https://fivetran.com/docs/applications/jira) in the format described by [this ERD](https://fivetran.com/docs/applications/jira#schemainformation) and builds off the output of our [Jira source package](https://github.com/fivetran/dbt_jira_source).
+- Produces modeled tables that leverage Jira data from [Fivetran's connector](https://fivetran.com/docs/applications/jira) in the format described by [this ERD](https://fivetran.com/docs/applications/jira#schemainformation).
 - Enables you to better understand the workload, performance, and velocity of your team's work using Jira issues. It performs the following actions:
   - Creates a daily issue history table so you can quickly create agile reports, such as burndown charts, along any issue field.
   - Enriches the core issue table with relevant data regarding its workflow and current state.
@@ -72,7 +72,7 @@ Include the following jira package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/jira
-    version: [">=0.21.0", "<0.22.0"]
+    version: [">=1.0.0", "<1.1.0"]
 
 ```
 ### Step 3: Define database and schema variables
@@ -116,10 +116,10 @@ vars:
 The `jira__daily_issue_field_history` model generates historical data for the columns specified by the `issue_field_history_columns` variable. By default, the only columns tracked are `status`, `status_id`,`sprint`, `story_points` and `story_point_estimate`, but all fields found in the Jira `FIELD` table's `field_name` column can be included in this model. The most recent value of any tracked column is also captured in `jira__issue_enhanced`.
 
 If you would like to change these columns, add the following configuration to your `dbt_project.yml` file. After adding the columns to your `dbt_project.yml` file, run the `dbt run --full-refresh` command to fully refresh any existing models:
-
-> IMPORTANT: If you wish to use a custom field, be sure to list the `field_name` and not the `field_id`. The corresponding `field_name` can be found in the `stg_jira__field` model.
-
-```yml
+    jira:
+      +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
+      staging:
+        +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
 vars:
     issue_field_history_columns: ['the', 'list', 'of', 'field', 'names']
 ```
@@ -147,10 +147,10 @@ By default, this package builds the Jira staging models within a schema titled (
 
 ```yml
 models:
-    jira_source:
-      +schema: my_new_schema_name # leave blank for just the target_schema
     jira:
-      +schema: my_new_schema_name # leave blank for just the target_schema
+      +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
+      staging:
+        +schema: my_new_schema_name # Leave +schema: blank to use the default target_schema.
 ```
 
 #### Change the source table references
@@ -187,9 +187,6 @@ This dbt package is dependent on the following dbt packages. These dependencies
 
 ```yml
 packages:
-    - package: fivetran/jira_source
-      version: [">=0.9.0", "<0.10.0"]
-
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]
 

dbt_project.yml

@@ -1,27 +1,27 @@
 name: 'jira'
-version: '0.21.0'
+version: '1.0.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:
 
   jira:
-    issue: "{{ ref('stg_jira__issue') }}"
-    project: "{{ ref('stg_jira__project') }}"
-    user: "{{ ref('stg_jira__user') }}"
-    issue_type: "{{ ref('stg_jira__issue_type') }}"
-    status: "{{ ref('stg_jira__status') }}"
-    status_category: "{{ ref('stg_jira__status_category') }}"
-    resolution: "{{ ref('stg_jira__resolution') }}"
-    priority: "{{ ref('stg_jira__priority') }}"
-    issue_multiselect_history: "{{ ref('stg_jira__issue_multiselect_history') }}"
-    issue_field_history: "{{ ref('stg_jira__issue_field_history') }}"
-    comment: "{{ ref('stg_jira__comment') }}"
-    issue_link: "{{ ref('stg_jira__issue_link') }}"
-    component: "{{ ref('stg_jira__component') }}"
-    field: "{{ ref('stg_jira__field') }}"
-    field_option: "{{ ref('stg_jira__field_option') }}"
-    sprint: "{{ ref('stg_jira__sprint') }}"
-    version: "{{ ref('stg_jira__version') }}"
+    issue: "{{ source('jira', 'issue') }}"
+    project: "{{ source('jira', 'project') }}"
+    user: "{{ source('jira', 'user') }}"
+    issue_type: "{{ source('jira', 'issue_type') }}"
+    status: "{{ source('jira', 'status') }}"
+    status_category: "{{ source('jira', 'status_category') }}"
+    resolution: "{{ source('jira', 'resolution') }}"
+    priority: "{{ source('jira', 'priority') }}"
+    issue_multiselect_history: "{{ source('jira', 'issue_multiselect_history') }}"
+    issue_field_history: "{{ source('jira', 'issue_field_history') }}"
+    comment: "{{ source('jira', 'comment') }}"
+    issue_link: "{{ source('jira', 'issue_link') }}"
+    component: "{{ source('jira', 'component') }}"
+    field: "{{ source('jira', 'field') }}"
+    field_option: "{{ source('jira', 'field_option') }}"
+    sprint: "{{ source('jira', 'sprint') }}"
+    version: "{{ source('jira', 'version') }}"
     jira_issue_history_buffer: 1
 models:
   jira:
@@ -30,3 +30,6 @@ models:
     intermediate:
       +materialized: ephemeral
       +schema: int_jira
+    staging:
+      +schema: jira_source
+      +materialized: view

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '0.21.0'
+version: '1.0.0'
 config-version: 2
 profile: 'integration_tests'
 
@@ -34,6 +34,9 @@ models:
   jira:
     +schema: "{{ 'jira_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'jira' }}"
     # +schema: "jira_{{ var('directed_schema','dev') }}"
+    staging:
+      +schema: jira_source
+      +materialized: view
 
 seeds:
   +docs:

integration_tests/seeds/issue.csv

@@ -1,4 +1,4 @@
-id,_fivetran_deleted,_fivetran_synced,_original_estimate,_remaining_estimate,_time_spent,assignee,created,creator,description,due_date,environment,issue_type,key,last_viewed,original_estimate,parent_id,priority,project,remaining_estimate,reporter,resolution,resolved,status,status_category_changed,summary,time_spent,updated,work_ratio
-10011,FALSE,2020-11-06 22:23:38,,,,,2020-04-01 08:33:58,5c3326c24b248c315badf3ee,"Yes, this is an issue to test permissions",,,10103,TP-8,,,,3,10001,,5c3326c24b248c315badf3ee,,2020-05-29 11:33:53,3,2020-04-01 08:34:02,We want to test permissions,,2020-05-29 11:33:53,-1
-10015,FALSE,2020-11-06 22:23:41,,,,,2020-06-11 07:38:03,5ed0edbe2999b60c23255b4f,,,,10103,TP-12,,,,5,10001,,5ed0edbe2999b60c23255b4f,,2020-06-11 07:38:03,10003,2020-06-11 07:38:03,Part of test epic,,2020-06-11 07:38:03,-1
-10018,FALSE,2020-11-12 12:20:50,,,,,2020-11-10 19:11:45,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,,,10103,TCP-1,2020-11-10 19:11:46,,,3,10005,,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,2020-11-28 19:11:45,10003,2020-11-28 19:11:46,this is a story,,2020-11-28 19:11:45,-1
+id,_fivetran_deleted,_fivetran_synced,_original_estimate,_remaining_estimate,_time_spent,assignee,created,creator,description,due_date,environment,issue_type,key,last_viewed,original_estimate,parent_id,priority,project,remaining_estimate,reporter,resolution,resolved,status,status_category_changed,summary,time_spent,updated,work_ratio,work_type
+10011,FALSE,2020-11-06 22:23:38,,,,,2020-04-01 08:33:58,5c3326c24b248c315badf3ee,"Yes, this is an issue to test permissions",,,10103,TP-8,,,,3,10001,,5c3326c24b248c315badf3ee,,2020-05-29 11:33:53,3,2020-04-01 08:34:02,We want to test permissions,,2020-05-29 11:33:53,-1,10103
+10015,FALSE,2020-11-06 22:23:41,,,,,2020-06-11 07:38:03,5ed0edbe2999b60c23255b4f,,,,,TP-12,,,,5,10001,,5ed0edbe2999b60c23255b4f,,2020-06-11 07:38:03,10003,2020-06-11 07:38:03,Part of test epic,,2020-06-11 07:38:03,-1,10103
+10018,FALSE,2020-11-12 12:20:50,,,,,2020-11-10 19:11:45,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,,,10103,TCP-1,2020-11-10 19:11:46,,,3,10005,,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,2020-11-28 19:11:45,10003,2020-11-28 19:11:46,this is a story,,2020-11-28 19:11:45,-1,

macros/staging/get_comment_columns.sql

@@ -0,0 +1,17 @@
+{% macro get_comment_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "author_id", "datatype": dbt.type_string()},
+    {"name": "body", "datatype": dbt.type_string()},
+    {"name": "created", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "is_public", "datatype": "boolean"},
+    {"name": "issue_id", "datatype": dbt.type_int()},
+    {"name": "update_author_id", "datatype": dbt.type_string()},
+    {"name": "updated", "datatype": dbt.type_timestamp()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/staging/get_component_columns.sql

@@ -0,0 +1,13 @@
+{% macro get_component_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "description", "datatype": dbt.type_string()},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "project_id", "datatype": dbt.type_int()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/staging/get_epic_columns.sql

@@ -0,0 +1,14 @@
+{% macro get_epic_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "done", "datatype": "boolean"},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "key", "datatype": dbt.type_string()},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "summary", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/staging/get_field_columns.sql

@@ -0,0 +1,13 @@
+{% macro get_field_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "is_array", "datatype": "boolean"},
+    {"name": "is_custom", "datatype": "boolean"},
+    {"name": "name", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

macros/staging/get_field_option_columns.sql

@@ -0,0 +1,12 @@
+{% macro get_field_option_columns() %}
+
+{% set columns = [
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "parent_id", "datatype": dbt.type_int()},
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "name", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}