diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ee858b7..95d44f01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,8 @@ - When teams are enabled, team data will populate in the `jira__daily_issue_field_history` model to track team assignments over time and the `jira__issue_enhanced` model to provide current team assignment for each issue. ## Under the Hood +- Introduced relevant team related seed data, macros, and variables to ensure team support. +- Created new analysis folder with `jira__issue_transition_cumulative_flow_analysis` and `jira_daily_issue_status_category_analysis` models. See the analysis [README.md](https://github.com/fivetran/dbt_jira/blob/main/analysis/README.md) for more details on how to use these models for your Jira reporting. - Created new ephemeral intermediate models that perform necessary operations to build the new `jira__timestamp_issue_field_history` models. - `int_jira__pivot_timestamp_field_history`: Table pivoting out the fields in field history into columns at the timestamp level to capture all field changes rather than just daily snapshots. - `int_jira__timestamp_field_history_scd`: Slowly-changing-dimension model that backfills field values at the timestamp level. diff --git a/README.md b/README.md index c278c2a9..bf621360 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ The following table provides a detailed list of all tables materialized within t | **Table** | **Description** | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | | [jira__daily_issue_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__daily_issue_field_history) | History table with one row for each day an issue remained open, with additional details about the issue sprint, status, and story points (if enabled).

**Example Analytics Questions:**
• How many issues, by sprint, were Closed or Blocked each week in Q1, 2025?
• How many days, on average, does it take an issue to go from 'Accepted' to either 'Closed' or 'Blocked'?| -| [jira__timestamp_issue_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__timestamp_issue_field_history) | Table tracking field changes at timestamp level with validity periods. Each record shows complete field state during a time period with valid_from/valid_until timestamps.

**Example Analytics Questions:**
• What was the exact sequence of field changes for a specific issue?
• How long did an issue spend in each status with precise timing?| +| [jira__timestamp_issue_field_history](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__timestamp_issue_field_history) | Table tracking field changes at timestamp level with validity periods. Each record shows complete field state during a time period with `valid_from`/`valid_until` timestamps.

**Example Analytics Questions:**
• What was the exact sequence of field changes for a specific issue?
• How long did an issue spend in each status with precise timing?| | [jira__issue_status_transitions](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__issue_status_transitions) | Issue status transition tracking with workflow analysis. Provides chronological view of status changes with timing metrics, transition direction analysis, and lifecycle indicators.

**Example Analytics Questions:**
• What is the average time spent in each status across all issues?
• Which is the lead time and cycle time for issues based on when work is added/started/completed?
• What are the most common workflow transition paths?| | [jira__issue_enhanced](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__issue_enhanced) | One row per Jira issue with enriched details about assignee, reporter, sprint, project, and current status, plus metrics on assignments and re-openings.

**Example Analytics Questions:**
• How many issues are currently blocked and who owns them?
• What's the average time to resolution for high-priority bugs by assignee? | | [jira__project_enhanced](https://fivetran.github.io/dbt_jira/#!/model/model.jira.jira__project_enhanced) | One row per project with team member details, issue counts, work velocity metrics, and project scope information.

**Example Analytics Questions:**
• Which projects have the highest velocity in terms of issues closed per sprint?
• What is the ratio of unassigned open tickets to assigned open tickets by project? | diff --git a/analysis/README.md b/analysis/README.md new file mode 100644 index 00000000..232c11fc --- /dev/null +++ b/analysis/README.md @@ -0,0 +1,33 @@ +# Jira Analysis +> Note: The compiled sql within the analysis folder references the final models [jira__issue_status_transitions](https://github.com/fivetran/dbt_jira/blob/master/models/jira__issue_status_transitions.sql), [jira__timestamp_issue_field_history](https://github.com/fivetran/dbt_jira/blob/master/models/jira__timestamp_issue_field_history), and [jira__daily_issue_field_history](https://github.com/fivetran/dbt_jira/blob/master/models/jira__daily_issue_field_history). As such, prior to +compiling the provided sql to analyze issue status category metrics, you must first execute `dbt run`. + + +## Analysis SQL +| **sql** | **description** | +| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| [jira__daily_issue_status_category_analysis](https://github.com/fivetran/dbt_jira/blob/master/analysis/jira__daily_issue_status_category_analysis.sql) | The output of the compiled sql will generate daily counts of issues in each status category by date, project, and team. The analysis provides a snapshot view of how many issues are in each status category (To Do, In Progress, Done) on any given day. The SQL references the `jira__daily_issue_field_history` model and can be customized by adjusting the `issue_field_history_columns` var in your `dbt_project.yml`. `status` can be used in place of `status_category` if desired by modifying the model accordingly. | +| [jira__issue_transition_cumulative_flow_analysis](https://github.com/fivetran/dbt_jira/blob/master/analysis/jira__issue_transition_cumulative_flow_analysis.sql) | The output of the compiled sql will generate daily transition metrics for issue status categories with cumulative flow calculations. The analysis joins status transitions with field history data to provide: count of distinct issues transitioning into a new status category, cumulative count of issues in each status category over time, and counts of issues that started work, completed work, or reopened work. The SQL references the `jira__issue_status_transitions` and `jira__timestamp_issue_field_history` models. Aggregation granularity can be adjusted by adding/removing field names in the `issue_field_history_columns` var in your `dbt_project.yml`. `status` can be used in place of `status_category` if desired by modifying the model accordingly. | + +## SQL Compile Instructions +Leveraging the above sql is made possible by the [analysis functionality of dbt](https://docs.getdbt.com/docs/building-a-dbt-project/analyses/). In order to +compile the sql, you will perform the following steps: +- Execute `dbt run` to create the package models. +- Execute `dbt compile` to generate the target specific sql. +- Navigate to your project's `/target/compiled/jira/analysis` directory. +- Copy the desired analysis code (`jira__daily_issue_status_category_analysis` or `jira__issue_transition_cumulative_flow_analysis`) and run in your data warehouse. +- Confirm the issue status category metrics match your expected workflow patterns. +- Analyze the daily counts, transition metrics, and cumulative flow data to identify trends and bottlenecks in your development process. + +## Contributions +Don't see a compiled sql statement you would have liked to be included? Notice any bugs when compiling +and running the analysis sql? If so, we highly encourage and welcome contributions to this package! +Please create issues or open PRs against `master`. Check out [this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package. + +## Database Support +This package has been tested on BigQuery, Snowflake, Redshift, Postgres, and Databricks. + +## Are there any resources available? +- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_jira/issues/new/choose) section to find the right avenue of support for you. +- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW). + diff --git a/analysis/jira__daily_issue_status_category_analysis.sql b/analysis/jira__daily_issue_status_category_analysis.sql new file mode 100644 index 00000000..5823d989 --- /dev/null +++ b/analysis/jira__daily_issue_status_category_analysis.sql @@ -0,0 +1,66 @@ +with field_history as ( + + select * + from {{ ref('jira__daily_issue_field_history') }} +), + +statuses as ( + + select * + from {{ ref('stg_jira__status') }} +), + +status_category as ( + + select * + from {{ ref('stg_jira__status_category') }} +), + +status_mapping as ( + + select + statuses.status_id, + statuses.status_name, + status_category.status_category_id, + case when lower(status_category.status_category_name) = 'to do' then 'To Do' + when lower(status_category.status_category_name) = 'in progress' then 'In Progress' + when lower(status_category.status_category_name) = 'done' then 'Done' + else 'Other' + end as status_category_name + from statuses + left join status_category + on statuses.status_category_id = status_category.status_category_id +), + +--if you prefer to keep analytics at the status level, substitute 'status' for 'status_category' throughout this model and skip the status_mapping step +daily_issue_status_category as ( + + select + field_history.date_day as report_date, + -- edit field_history columns as desired based on the issue_field_history_columns var for your dbt project + -- project and team have been chosen for this example + field_history.project, + field_history.team, + field_history.issue_id, + field_history.status, + status_mapping.status_category_name as status_category, + from field_history + left join status_mapping + on lower(field_history.status) = lower(status_mapping.status_name) +), + +--count of issues in each status on report date +daily_counts as ( + + select + report_date, + project, + team, + status_category, + count(distinct issue_id) as issues_in_status + from daily_issue_status_category + group by 1,2,3,4 +) + +select * +from daily_counts diff --git a/analysis/jira__issue_transition_cumulative_flow_analysis.sql b/analysis/jira__issue_transition_cumulative_flow_analysis.sql new file mode 100644 index 00000000..17d415f0 --- /dev/null +++ b/analysis/jira__issue_transition_cumulative_flow_analysis.sql @@ -0,0 +1,62 @@ +with field_history as ( + + select * + from {{ ref('jira__timestamp_issue_field_history') }} + +), + +status_transitions as ( + + select * + from {{ ref('jira__issue_status_transitions') }} +), + +---if you prefer to keep analytics at the status level, substitute 'status' for 'status_category' throughout this model +joined as ( + + select + transitions.transition_at, + transitions.issue_id, + transitions.status_category_name, + -- edit field_history columns as desired based on the issue_field_history_columns var for your dbt project + -- project and team have been chosen for this example + field_history.team, + field_history.project + from status_transitions as transitions + join field_history + on transitions.issue_id = field_history.issue_id + and transitions.transition_at = field_history.valid_from +), + +---roll up to daily for reporting +daily_rollup as ( + + select + cast(transition_at as date) as status_category_transition_date, + project, + team, + status_category_name as status_category, + count(distinct issue_id) as issues_in_new_status_category, + from joined + group by 1,2,3,4 +), + +--add cumulative flow calculation +cumulative_flow as ( + + select + status_category_transition_date, + project, + team, + status_category, + issues_in_new_status_category, + sum(issues_in_new_status_category) over ( + partition by project, team, status_category + order by status_category_transition_date + rows unbounded preceding + ) as cumulative_issues_in_status_category + from daily_rollup +) + +select * +from cumulative_flow \ No newline at end of file diff --git a/dbt_project.yml b/dbt_project.yml index c58edb84..ddb3b1db 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -25,6 +25,12 @@ vars: version: "{{ source('jira', 'version') }}" jira_issue_history_buffer: 1 +analysis-paths: ["analysis"] +clean-targets: + - target + - dbt_modules + - dbt_packages + models: jira: @@ -36,3 +42,5 @@ models: staging: +materialized: view +schema: jira_source + + diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 998c0c3b..cbfce328 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -3,9 +3,10 @@ version: '1.1.0' config-version: 2 profile: 'integration_tests' -vars: - # Comment out the below when generating docs - issue_field_history_columns: ['summary', 'components', 'team'] # @docs-ignore + +vars: + # Comment out the below when generating docs + issue_field_history_columns: ['summary', 'components', 'team' ,'project'] # @docs-ignore jira: jira_schema: jira_integrations_tests_42 @@ -32,6 +33,7 @@ vars: jira_version_identifier: "version" jira_team_identifier: "team" + models: jira: +schema: "{{ 'jira_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'jira' }}"