Feature/add teams to jira package (#149)

fivetran-savage · web-flow · commit ab172a68ea31 · 2025-10-23T11:32:27.000-07:00
* feature/add_teams_to_jira_package

* add_team_column_to_end_models

* update_seed_data

* add_teams_to_timestamp_model

* update_columns

* remove_team__hardcoding

* update_changelog
diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml
@@ -12,6 +12,8 @@ table_variables:
     - version
   jira_using_priorities:
     - priority
+  jira_using_teams:
+    - team
 
 destination_configurations:
   databricks:
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,19 +1,29 @@
 # dbt_jira v1.1.0
 
-## Schema Updates
+## Schema/Data Change
+**8 total changes • 6 possible breaking changes**
 
-**6 total changes • 4 possible breaking changes**
-
-| Data Model                                    | Change Type | Old  | New                               | Notes                                                             |
-|---------------------------------------------------|-------------|----------|-------------------------------------------|-------------------------------------------------------------------|
+| Data Model(s) | Change type | Old | New | Notes |
+| ---------- | ----------- | -------- | -------- | ----- |
+| [stg_jira__team_tmp](https://fivetran.github.io/dbt_jira/#!/model/model.jira.stg_jira__team) | New Model |  |  | Added temp model as Jira teams source. |
+| [stg_jira__team](https://fivetran.github.io/dbt_jira/#!/model/model.jira.stg_jira__team) |New Model |  |  | Added staging model for Jira teams functionality
 | [jira__timestamp_issue_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__timestamp_issue_field_history)       | New Model   |     |  | SCD Type 2 table tracking field changes at timestamp level with validity periods. Each record shows complete field state during a time period with `valid_from`/`valid_until` timestamps. |
 | [int_jira__pivot_timestamp_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.int_jira__pivot_timestamp_field_history)  | New Model   |          |    | Pivots field history into columns at timestamp level, capturing ALL field changes rather than daily snapshots. |
 | [int_jira__timestamp_field_history_scd](https://fivetran.github.io/dbt_jira/#!/model/model.jira.int_jira__timestamp_field_history_scd)        | New Model   |          |    | SCD logic that backfills field values at timestamp level, showing complete field state at each update timestamp. |
 | [stg_jira__issue_field_history](https://fivetran.github.io/dbt_jira_source/#!/model/model.jira_source.stg_jira__issue_field_history)        | New Column   |          |  `author_id`    |    Tracks which user made each field change for audit trail and change attribution.     |
 | [stg_jira__issue_multiselect_history](https://fivetran.github.io/dbt_jira_source/#!/model/model.jira_source.stg_jira__issue_multiselect_history)        | New Column   |          |  `author_id`    |    Tracks which user made each multiselect field change for audit trail and change attribution.       |
 | [jira__timestamp_issue_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__timestamp_issue_field_history)        | New Columns   |          |  `author_id`, `status_id`, `is_current_record` |    Adds user attribution, raw status ID alongside resolved status name, and current record indicator for SCD Type 2 functionality.    | 
- 
 
+## Feature Update
+- Added support for Jira teams functionality by introducing staging models `stg_jira__team` and `stg_jira__team_tmp`.
+- Enhanced issue field history models to include team information if teams are enabled for the customer.
+  - When teams are enabled, team data will populate in the `jira__daily_issue_field_history` model to track team assignments over time and the `jira__issue_enhanced` model to provide current team assignment for each issue.
+
+## Under the Hood
+- Added new seed file `team.csv` for integration testing.
+- Added team-related test data to `issue.csv`, `issue_field_history.csv`, and `field.csv` seed files.
+- Created `get_team_columns` macro for consistent team column definitions.
+- Updated package variables to include team source reference and team identifier configuration.
 
 # dbt_jira v1.0.0
 
diff --git a/dbt_project.yml b/dbt_project.yml
@@ -21,6 +21,7 @@ vars:
     field: "{{ source('jira', 'field') }}"
     field_option: "{{ source('jira', 'field_option') }}"
     sprint: "{{ source('jira', 'sprint') }}"
+    team: "{{ source('jira', 'team') }}"
     version: "{{ source('jira', 'version') }}"
     jira_issue_history_buffer: 1
 
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -5,7 +5,7 @@ profile: 'integration_tests'
 
 vars: 
   # Comment out the below when generating docs
-  issue_field_history_columns: ['summary', 'components'] # @docs-ignore
+  issue_field_history_columns: ['summary', 'components', 'team'] # @docs-ignore
 
   jira:
     jira_schema: jira_integrations_tests_42
@@ -30,6 +30,7 @@ vars:
     jira_user_group_identifier: "user_group"
     jira_user_identifier: "user"
     jira_version_identifier: "version"
+    jira_team_identifier: "team"
 
 models:
   jira:
diff --git a/integration_tests/seeds/field.csv b/integration_tests/seeds/field.csv
@@ -5,4 +5,5 @@ issuerestriction,2020-11-23 22:20:39.643,false,false,Restrict to
 components,2020-11-23 22:20:40.643,false,false,Components
 summary,2020-11-23 22:20:41.553,false,false,Summary
 epiclink,2020-11-23 22:20:41.553,false,false,Epic-Link
-epic,2020-11-23 22:20:41.553,false,false,Epic
+epic,2020-11-23 22:20:41.553,false,false,Epic
+customfield_10001,2020-11-23 22:20:41.553,false,true,Team
diff --git a/integration_tests/seeds/issue.csv b/integration_tests/seeds/issue.csv
@@ -2,3 +2,4 @@ id,_fivetran_deleted,_fivetran_synced,_original_estimate,_remaining_estimate,_ti
 10011,FALSE,2020-11-06 22:23:38,,,,,2020-04-01 08:33:58,5c3326c24b248c315badf3ee,"Yes, this is an issue to test permissions",,,10103,TP-8,,,,3,10001,,5c3326c24b248c315badf3ee,,2020-05-29 11:33:53,3,2020-04-01 08:34:02,We want to test permissions,,2020-05-29 11:33:53,-1,10103
 10015,FALSE,2020-11-06 22:23:41,,,,,2020-06-11 07:38:03,5ed0edbe2999b60c23255b4f,,,,,TP-12,,,,5,10001,,5ed0edbe2999b60c23255b4f,,2020-06-11 07:38:03,10003,2020-06-11 07:38:03,Part of test epic,,2020-06-11 07:38:03,-1,10103
 10018,FALSE,2020-11-12 12:20:50,,,,,2020-11-10 19:11:45,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,,,10103,TCP-1,2020-11-10 19:11:46,,,3,10005,,557058:a9572f6a-3041-435a-96ed-5286e3811b33,,2020-11-28 19:11:45,10003,2020-11-28 19:11:46,this is a story,,2020-11-28 19:11:45,-1,
+10027,FALSE,2020-11-12 12:20:50,,,,,2020-11-15 19:11:45,5ed0edbe2999b60c23255b4f,Test issue to see if teams data populates,,,10103,TP-10,,,,1,10005,,5ed0edbe2999b60c23255b4f,,,3,,just a regular issue,,,,
diff --git a/integration_tests/seeds/issue_field_history.csv b/integration_tests/seeds/issue_field_history.csv
@@ -1,8 +1,11 @@
-field_id,issue_id,time,_fivetran_synced,value,author_id,is_active
-created,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.478,2020-11-10T19:19:41.472Z,5e61c1c5c8bee8005f04c88b,true
-customfield_10104,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.472,3.0,5e61c1c5c8bee8005f04c88b,true
-summary,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.500,"As a developer, I'd like to update story status during the sprint >> Click the Active sprints link at the top right of the screen to go to the Active sprints where the current Sprint's items can be updated",5e61c1c5c8bee8005f04c88b,true
-components,10018,2020-11-10 19:19:41.472,2020-11-19 12:20:53.500,10019,5e61c1c5c8bee8005f04c88c,true
-summary,10018,2020-11-10 19:19:41.472,2020-11-19 12:20:53.500,10026,5e61c1c5c8bee8005f04c88c,true
-epic,10015,2020-11-10 19:19:41.472,2020-11-19 12:20:53.500,1111111,5e61c1c5c8bee8005f04c88d,true
-epiclink,10018,2020-11-10 19:19:41.472,2020-11-12 12:20:53.500,https://ulr-here/epic-key,5e61c1c5c8bee8005f04c88c,true
+field_id,issue_id,time,_fivetran_synced,value,is_active,author_id
+created,10027,2020-11-10 19:19:41.472,2020-11-12 12:20:53.478,2020-11-10T19:19:41.472Z,true,5e61c1c5c8bee8005f04c88b
+customfield_10104,10027,2020-11-10 19:19:41.473,2020-11-12 12:20:53.472,3.0,true,5e61c1c5c8bee8005f04c88b
+summary,10027,2020-11-10 19:19:41.474,2020-11-12 12:20:53.500,"As a developer, I'd like to update story status during the sprint >> Click the Active sprints link at the top right of the screen to go to the Active sprints where the current Sprint's items can be updated",true,5e61c1c5c8bee8005f04c88b
+components,10018,2020-11-10 19:19:41.475,2020-11-19 12:20:53.501,10019,true,5e61c1c5c8bee8005f04c88c
+summary,10018,2020-11-10 19:19:41.476,2020-11-19 12:20:53.502,10026,true,5e61c1c5c8bee8005f04c88c
+epic,10015,2020-11-10 19:19:41.477,2020-11-19 12:20:53.503,1111111,true,5e61c1c5c8bee8005f04c88d
+epiclink,10018,2020-11-10 19:19:41.478,2020-11-12 12:20:53.504,https://ulr-here/epic-key,true,5e61c1c5c8bee8005f04c88c
+customfield_10001,10027,2020-11-10 19:19:41.479,2020-11-12 12:20:53.505,01e9ac93-bccf-4ad2-af92-3317a37ec67b,true,5e61c1c5c8bee8005f04c88c
+customfield_10001,10018,2020-11-10 19:19:41.480,2020-11-12 12:20:53.506,1bc241c7-6926-4b46-94b7-82d25069f067,true,5e61c1c5c8bee8005f04c88c
+customfield_10001,10015,2020-11-10 19:19:41.481,2020-11-12 12:20:53.507,e74a1782-f463-4542-8440-66de07d4b083,true,5e61c1c5c8bee8005f04c88c
diff --git a/integration_tests/seeds/issue_multiselect_history.csv b/integration_tests/seeds/issue_multiselect_history.csv
@@ -1,5 +1,5 @@
 _fivetran_id,time,_fivetran_synced,field_id,issue_id,value,author_id,is_active
 w4UiT+hPMxJp3RhL/YFJm3uWL5U=,2020-11-10 19:19:41.472,2020-11-12 12:20:53.506,subtasks,10027,0,5e61c1c5c8bee8005f04c88b,true
-4pVgGn0qSqR2hCmMdo4wWHXmgew=,2020-11-10 19:19:41.472,2020-11-12 12:20:53.479,customfield_10021,10027,0,5e61c1c5c8bee8005f04c88b,true
-/zrY8m6q0VMW6ia1jGIerXqLIgQ=,2020-11-10 19:19:41.472,2020-11-12 12:20:53.479,customfield_10020,10027,0,5e61c1c5c8bee8005f04c88b,true
-3p3gGn0qSqR2hCmMdo4wWHXa32m=,2020-11-10 19:19:41.472,2020-11-12 12:20:53.479,components,10027,0,5e61c1c5c8bee8005f04c88b,true
+4pVgGn0qSqR2hCmMdo4wWHXmgew=,2020-11-10 19:19:42.472,2020-11-12 12:20:53.479,customfield_10021,10027,0,5e61c1c5c8bee8005f04c88b,true
+/zrY8m6q0VMW6ia1jGIerXqLIgQ=,2020-11-10 19:19:43.472,2020-11-12 12:20:53.479,customfield_10020,10027,0,5e61c1c5c8bee8005f04c88b,true
+3p3gGn0qSqR2hCmMdo4wWHXa32m=,2020-11-10 19:19:44.472,2020-11-12 12:20:53.479,components,10027,0,5e61c1c5c8bee8005f04c88b,true
diff --git a/integration_tests/seeds/team.csv b/integration_tests/seeds/team.csv
@@ -0,0 +1,6 @@
+id,is_shared,is_visible,name,title,_fivetran_synced
+01e9ac93-bccf-4ad2-af92-3317a37ec67b,true,true,HR,HR,2020-11-23 22:20:39.685
+891da445-73e5-49b9-9d0d-89f6a587822a,true,true,Cloud Security,Cloud Security,2020-11-23 22:20:39.711
+1bc241c7-6926-4b46-94b7-82d25069f067,true,true,People Ops,People Ops,2020-11-23 22:20:39.643
+e74a1782-f463-4542-8440-66de07d4b083,true,true,Detection & Response,Detection & Response,2020-11-23 22:20:40.643
+d8a8402f-bc1b-4990-9b11-06a52d2de68a,true,true,Platform Security,Platform Security,2020-11-23 22:20:41.553
diff --git a/macros/staging/get_team_columns.sql b/macros/staging/get_team_columns.sql
@@ -0,0 +1,14 @@
+{% macro get_team_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_string()},
+    {"name": "is_shared", "datatype": "boolean"},
+    {"name": "is_visible", "datatype": "boolean"},
+    {"name": "name", "datatype": dbt.type_string()},
+    {"name": "title", "datatype": dbt.type_string()},
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}
diff --git a/models/jira__daily_issue_field_history.sql b/models/jira__daily_issue_field_history.sql
@@ -69,6 +69,14 @@ issue_types as (
     from {{ ref('stg_jira__issue_type') }}
 ),
 
+{% if var('jira_using_teams', True) %}
+teams as ( 
+
+    select * 
+    from {{ ref('stg_jira__team') }} 
+),
+{% endif %}
+
 {% if var('jira_using_components', True) %}
 components as (
 
@@ -88,7 +96,10 @@ joined as (
                 {% if col.name|lower == 'components' and var('jira_using_components', True) %}
                 , coalesce(pivoted_daily_history.components, most_recent_data.components) as components
 
-                {% elif col.name|lower not in ['issue_day_id', 'issue_id', 'valid_starting_on', 'valid_starting_at_week', 'components'] %} 
+                {% elif col.name|lower == 'team' and var('jira_using_teams', True) %} 
+                , coalesce(pivoted_daily_history.team, most_recent_data.team) as team 
+
+                {% elif col.name|lower not in ['issue_day_id', 'issue_id', 'valid_starting_on', 'valid_starting_at_week', 'components', 'team'] %} 
                 , coalesce(pivoted_daily_history.{{ col.name }}, most_recent_data.{{ col.name }}) as {{ col.name }}
 
                 {% endif %}
@@ -97,9 +108,12 @@ joined as (
         {% else %}
             {% for col in pivot_data_columns %}
                 {% if col.name|lower == 'components' and var('jira_using_components', True) %}
-                , pivoted_daily_history.components   
+                , pivoted_daily_history.components
 
-                {% elif col.name|lower not in ['issue_day_id', 'issue_id', 'valid_starting_on', 'valid_starting_at_week', 'components'] %} 
+                {% elif col.name|lower == 'team' and var('jira_using_teams', True) %} 
+                , pivoted_daily_history.team
+  
+                {% elif col.name|lower not in ['issue_day_id', 'issue_id', 'valid_starting_on', 'valid_starting_at_week', 'components','team'] %} 
                 , pivoted_daily_history.{{ col.name }}
 
                 {% endif %}
@@ -127,7 +141,7 @@ set_values as (
             order by date_day rows unbounded preceding) as status_id_field_partition
 
         -- list of exception columns
-        {% set exception_cols = ['issue_id', 'issue_day_id', 'valid_starting_on', 'valid_starting_at_week', 'status', 'status_id', 'components', 'issue_type'] %}
+        {% set exception_cols = ['issue_id', 'issue_day_id', 'valid_starting_on', 'valid_starting_at_week', 'status', 'status_id', 'components', 'issue_type', 'team'] %}
 
         {% for col in pivot_data_columns %}
             {% if col.name|lower == 'components' and var('jira_using_components', True) %}
@@ -137,6 +151,11 @@ set_values as (
             {% elif col.name|lower == 'issue_type' %}
             , coalesce(issue_types.issue_type_name, joined.issue_type) as issue_type
             , sum(case when joined.issue_type is null then 0 else 1 end) over (partition by issue_id order by date_day rows unbounded preceding) as issue_type_field_partition
+            
+            {% elif col.name|lower == 'team' and var('jira_using_teams', True) %}
+            , coalesce(teams.team_name, joined.team) as team
+            , sum(case when joined.team is null then 0 else 1 end) over ( partition by issue_id
+                order by date_day rows unbounded preceding) as team_field_partition
 
             {% elif col.name|lower not in exception_cols %}
             , coalesce(field_option_{{ col.name }}.field_option_name, joined.{{ col.name }}) as {{ col.name }}
@@ -158,6 +177,10 @@ set_values as (
         left join issue_types   
             on cast(issue_types.issue_type_id as {{ dbt.type_string() }}) = joined.issue_type
 
+        {% elif col.name|lower == 'team'and var('jira_using_teams', True) %} 
+        left join teams 
+            on cast(teams.team_id as {{ dbt.type_string() }}) = joined.team
+
         {% elif col.name|lower not in exception_cols %}
         left join field_option as field_option_{{ col.name }}
             on cast(field_option_{{ col.name }}.field_id as {{ dbt.type_string() }}) = joined.{{ col.name }}
@@ -181,7 +204,12 @@ fill_values as (
                 partition by issue_id, component_field_partition 
                 order by date_day asc rows between unbounded preceding and current row) as components
 
-            {% elif col.name|lower not in ['issue_id', 'issue_day_id', 'valid_starting_on', 'valid_starting_at_week', 'status', 'status_id', 'components'] %}
+            {% elif col.name|lower == 'team' and var('jira_using_teams', True) %}
+            , first_value(team) over (
+                partition by issue_id, team_field_partition
+                order by date_day asc rows between unbounded preceding and current row) as team
+
+            {% elif col.name|lower not in ['issue_id', 'issue_day_id', 'valid_starting_on', 'valid_starting_at_week', 'status', 'status_id', 'components', 'team'] %}
             -- grab the value that started this batch/partition
             , first_value( {{ col.name }} ) over (
                 partition by issue_id, {{ col.name }}_field_partition 
@@ -203,7 +231,10 @@ fix_null_values as (
             {% if col.name|lower == 'components' and var('jira_using_components', True) %}
             , case when components = 'is_null' then null else components end as components
 
-            {% elif col.name|lower not in ['issue_id','issue_day_id','valid_starting_on', 'valid_starting_at_week', 'status', 'components'] %}
+            {% elif col.name|lower == 'team' and var('jira_using_teams', True) %}
+            , case when team = 'is_null' then null else team end as team
+
+            {% elif col.name|lower not in ['issue_id','issue_day_id','valid_starting_on', 'valid_starting_at_week', 'status', 'components','team'] %}
             -- we de-nulled the true null values earlier in order to differentiate them from nulls that just needed to be backfilled
             , case when {{ col.name }} = 'is_null' then null else {{ col.name }} end as {{ col.name }}
 
@@ -226,7 +257,10 @@ surrogate_key as (
             {% if col.name|lower == 'components' and var('jira_using_components', True) %}
             , fix_null_values.components as components
 
-            {% elif col.name|lower not in ['issue_id','issue_day_id','valid_starting_on', 'valid_starting_at_week', 'status', 'components'] %} 
+            {% elif col.name|lower == 'team' and var('jira_using_teams', True) %} 
+            , fix_null_values.team as team
+
+            {% elif col.name|lower not in ['issue_id','issue_day_id','valid_starting_on', 'valid_starting_at_week', 'status', 'components','team'] %} 
             , fix_null_values.{{ col.name }} as {{ col.name }}
 
             {% endif %}
diff --git a/models/jira__timestamp_issue_field_history.sql b/models/jira__timestamp_issue_field_history.sql
@@ -27,6 +27,14 @@ components as (
 ),
 {% endif %}
 
+{% if var('jira_using_teams', True) %}
+teams as (
+
+    select *
+    from {{ ref('stg_jira__team') }}
+),
+{% endif %}
+
 field_option as (
 
     select *
@@ -48,7 +56,7 @@ create_validity_periods as (
         author_id
 
         -- list of exception columns
-        {% set exception_cols = ['issue_id', 'issue_timestamp_id', 'updated_at', 'updated_at_week', 'status', 'author_id', 'components', 'issue_type'] %}
+        {% set exception_cols = ['issue_id', 'issue_timestamp_id', 'updated_at', 'updated_at_week', 'status', 'author_id', 'issue_type'] %}
 
         {% for col in pivot_data_columns %}
             {% if col.name|lower not in exception_cols %}
@@ -71,7 +79,7 @@ final as (
         create_validity_periods.author_id
 
         -- list of exception columns
-        {% set exception_cols = ['issue_id', 'issue_timestamp_id', 'updated_at', 'updated_at_week', 'status', 'author_id', 'components', 'issue_type'] %}
+        {% set exception_cols = ['issue_id', 'issue_timestamp_id', 'updated_at', 'updated_at_week', 'status', 'author_id', 'components', 'issue_type', 'team'] %}
 
         {% for col in pivot_data_columns %}
             {% if col.name|lower == 'components' and var('jira_using_components', True) %}
@@ -80,6 +88,9 @@ final as (
             {% elif col.name|lower == 'issue_type' %}
             , coalesce(issue_types.issue_type_name, create_validity_periods.issue_type) as issue_type
 
+            {% elif col.name|lower == 'team' and var('jira_using_teams', True) %}
+            , coalesce(teams.team_name, create_validity_periods.team) as team
+
             {% elif col.name|lower not in exception_cols %}
             , coalesce(field_option_{{ col.name }}.field_option_name, create_validity_periods.{{ col.name }}) as {{ col.name }}
 
@@ -104,6 +115,10 @@ final as (
         left join issue_types
             on cast(issue_types.issue_type_id as {{ dbt.type_string() }}) = create_validity_periods.issue_type
 
+        {% elif col.name|lower == 'team' and var('jira_using_teams', True) %}
+        left join teams
+            on cast(teams.team_id as {{ dbt.type_string() }}) = create_validity_periods.team
+
         {% elif col.name|lower not in exception_cols %}
         left join field_option as field_option_{{ col.name }}
             on cast(field_option_{{ col.name }}.field_id as {{ dbt.type_string() }}) = create_validity_periods.{{ col.name }}
diff --git a/models/staging/src_jira.yml b/models/staging/src_jira.yml
diff --git a/models/staging/stg_jira.yml b/models/staging/stg_jira.yml
diff --git a/models/staging/stg_jira__team.sql b/models/staging/stg_jira__team.sql
diff --git a/models/staging/tmp/stg_jira__team_tmp.sql b/models/staging/tmp/stg_jira__team_tmp.sql
