bug/string-limit-bq-sf (#193)

* bug/string-limit-bq-sf

* fix redshift

* redshift and docs

* seed for redshift

* update seeds for testing

* Apply suggestion from @fivetran-savage

Co-authored-by: Savage Fivetran <sarah.savage@fivetran.com>

* Update CHANGELOG.md

* Generate dbt docs via GitHub Actions

---------

Co-authored-by: Savage Fivetran <sarah.savage@fivetran.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Author: 3 people

CHANGELOG.md

@@ -1,3 +1,10 @@
+# dbt_jira v1.10.1
+
+[PR #193](https://github.com/fivetran/dbt_jira/pull/193) includes the following updates:
+
+## Bug Fix
+- Fixes a runtime error on Snowflake and BigQuery that occurred when an issue's concatenated comments exceeded the warehouse's string length limit. The `conversation` field in `jira__issue_enhanced` now returns `'conversation too long to render'` for those issues instead of failing the entire run. `count_comments` is unaffected. The character threshold defaults to 16,777,216 (16MB) but can be lowered using the `jira_conversation_char_limit` variable. [See the README for configuration details](https://github.com/fivetran/dbt_jira#controlling-conversation-aggregations-in-jira__issue_enhanced) and the [DECISIONLOG](https://github.com/fivetran/dbt_jira/blob/main/DECISIONLOG.md#conversation-aggregation-guard-against-string-length-limits) for per-warehouse details.
+
 # dbt_jira v1.10.0
 
 [PR #191](https://github.com/fivetran/dbt_jira/pull/191) includes the following updates:
@@ -621,4 +628,4 @@ This release includes the following updates.
 - @thibonacci ([#45](https://github.com/fivetran/dbt_jira/pull/45))
 
 # dbt_jira v0.1.0 -> v0.5.1
-Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
+Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!

DECISIONLOG.md

@@ -1,7 +1,19 @@
 # Decision Log
 In creating this package, which is meant for a wide range of use cases, we had to take opinionated stances on a few different questions we came across during development. We've consolidated significant choices we made here, and will continue to update as the package evolves.
 
+## Conversation Aggregation Guard Against String Length Limits
+Jira issues with a large volume of comments can cause the `conversation` field to exceed the warehouse string size limit, resulting in a runtime error. To prevent this, the package now checks the total conversation length for each issue before aggregation on Snowflake and BigQuery.
+
+If an issue exceeds the configured limit, `conversation` returns `'conversation too long to render'` instead of failing the model. The `count_comments` field is still populated, so users can identify issues with large comment volumes even when the full conversation cannot be rendered.
+
+The default threshold is `16,777,216` characters for Snowflake and BigQuery. You can override this value with `jira_conversation_char_limit` if you need a lower threshold. See the [README](https://github.com/fivetran/dbt_jira#controlling-conversation-aggregations-in-jira__issue_enhanced) for configuration details.
+
+Per-warehouse behavior:
+- **Snowflake and BigQuery**: Overflow protection is enabled by default. If an issue has enough comments for `conversation` to exceed the warehouse string size limit, `conversation` returns `'conversation too long to render'` instead of failing. The default limit is `16,777,216` characters.
+- **Redshift**: Overflow protection is not currently applied. Redshift has a much lower string size limit, but it does not support the SQL pattern used by this guard, so adding the same protection would require a more significant model change. Conversations are disabled on Redshift by default with `jira_include_conversations`, so overflow is only a risk for users who explicitly enable conversations and have issues with very large comment volumes. If you need Redshift overflow protection, [open a GitHub issue](https://github.com/fivetran/dbt_jira/issues/new/choose).
+- **Postgres and Databricks**: Overflow protection is not applied because their string/text limits make this issue unlikely in practice.
+
 ## Enhancing Jira Sprint Reporting with Flexible Metrics
 To improve sprint reporting in the Jira dbt package, we introduced two new models, `jira__daily_sprint_issue_history` and `jira__sprint_enhanced`, designed to capture key sprint metrics such as velocity, time tracking, and story point completion.
 
-If further refinements or reporting is needed for the sprint report, customers can submit a feature request by clicking the [New Issue button in our package issue page](https://github.com/fivetran/dbt_jira/issues).
+If further refinements or reporting is needed for the sprint report, customers can submit a feature request by clicking the [New Issue button in our package issue page](https://github.com/fivetran/dbt_jira/issues).

README.md

@@ -144,14 +144,18 @@ vars:
 The `dbt_jira` package offers variables to enable or disable conversation aggregations in the `jira__issue_enhanced` table. These settings allow you to manage the amount of data processed and avoid potential performance or limit issues with large datasets.
 
 - `jira_include_conversations`: Controls only the `conversation` [column](https://github.com/fivetran/dbt_jira/blob/main/models/jira.yml#L125-L127) in the `jira__issue_enhanced` table. 
-  - Default: Disabled for Redshift due to string size constraints; enabled for other supported warehouses.
+  - Default: Disabled for Redshift due to string size constraints; enabled for all other supported warehouses.
   - Setting this to `false` removes the `conversation` column but retains the `count_comments` field if `jira_include_comments` is still enabled. This is useful if you want a comment count without the full conversation details.
+  - On Snowflake and BigQuery, the package automatically guards against string length limit errors. Issues whose aggregated comment text would exceed the warehouse limit return `'conversation too long to render'` instead of failing. The `count_comments` field is unaffected.
+
+- `jira_conversation_char_limit`: Sets the character threshold used on Snowflake and BigQuery to detect conversations that would exceed the warehouse string length limit.
 
 In your `dbt_project.yml` file:
 
 ```yml
 vars:
   jira_include_conversations: false/true # Disabled by default for Redshift; enabled for other supported warehouses.
+  jira_conversation_char_limit: 16777216 # Override the default character limit for conversation aggregation.
 ```
 
 #### Controlling team-level granularity in `jira__sprint_enhanced`

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira'
-version: '1.10.0'
+version: '1.10.1'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 

docs/catalog.json

@@ -27,6 +27,13 @@ test_scenarios:
       jira_field_grain: 'field_name'
     include_incremental: true
 
+  # Redshift is false by default
+  - name: "conversations disabled"
+    vars:
+      jira_include_conversations: false
+    include_incremental: false
+    warehouses: ["snowflake", "bigquery", "postgres", "databricks"]
+
   - name: "MDLS"
     vars:
       fivetran_using_source_casing: true

docs/manifest.json

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '1.10.0'
+version: '1.10.1'
 config-version: 2
 profile: 'integration_tests'
 
@@ -49,6 +49,15 @@ seeds:
     +column_types:
       _fivetran_synced: timestamp
     comment:
+      +enabled: "{{ target.type != 'redshift' }}"
+      +column_types:
+        id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        created: timestamp
+        issue_id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
+        updated: timestamp
+    comment_redshift:
+      +enabled: "{{ target.type == 'redshift' }}"
+      +alias: comment
       +column_types:
         id: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
         created: timestamp