Enable package to use Snowflake Organisation Account views #1
Conversation
bisset-a
changed the title
| {{ config(materialized='table') }} | ||
| {{ config( | ||
| materialized='table', | ||
| enabled=not(var('uses_org_view', false)) |
There was a problem hiding this comment.
Let's add a comment just to show that we can do this once metering_history is available at org level?
models/hourly_spend_org_level.sql
Outdated
There was a problem hiding this comment.
Oh just saw there's a separate org level model, how comes?
There was a problem hiding this comment.
I originally did this because there was too much conditional logic - but you're right I think we should just completely disable this and daily_spend until Snowflake makes the metering_history view available
| warehouse_snapshots.organization_name, | ||
| warehouse_snapshots.account_name, | ||
| warehouse_snapshots.account_locator, | ||
| {% endif %} |
There was a problem hiding this comment.
Would you be able to simplify this significantly if you set this to always include these new columns? I'm not sure the conditional rendering is actually necessary. In the single account version, they'd just be for one account, but it still works
| @@ -0,0 +1,7 @@ | |||
| {% macro generate_scoped_unique_key(base_fields) %} | |||
There was a problem hiding this comment.
I just realised that if we're now adding the account_name column everywhere, we can always add the account_name to the unique key config! :)
There was a problem hiding this comment.
I just realised this too!! 😆 my only concern was for backwards compatibility for incremental models, account_name would be null for historical data but I think in this case it might be ok to have nulls in the unique_key since the other fields will be unique anyway. other option would be to create a new column called _unique_key that uses generate_surrogate_keys to handle the nulls and use that as the unique_key. wdyt?
There was a problem hiding this comment.
I think a composite PK works totally fine without making a new column - honestly I could go either way.
There was a problem hiding this comment.
perfect - lets leave it as it! thanks Niall!
macros/add_account_columns.sql
Outdated
|
Sweet. Want to open this PR against our main repo? |
PR created: get-select#178 |
This PR adds support of views in the new Snowflake organisation account, while still being compatible with single account Snowflake views.
To use the dbt_snowflake_monitoring package in multi-account mode, Project variable
uses_org_viewshould be set totruein the downstream project.Notes
metering_historyview is not currently available in the organisation account. Snowflake have confirmed it will be available in Q2 (May - July) so right now we are making a pragmatic decision to exclude QA from the query cost calculation until this view is enabled.hourly_spendanddaily_spenddo not include serverless task spend. This is becauseseverless_task_historydoes not exist in the organisation account. Snowflake confirmed this will be enabled in Q2 aswell.