Skip to content

Enable package to use Snowflake Organisation Account views #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

Enable package to use Snowflake Organisation Account views #1

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5a9028b
update models
bisset-a Feb 26, 2025
1c2e124
resolving merge conflicts
bisset-a Feb 26, 2025
e1aa209
fix
bisset-a Feb 27, 2025
9f3ae59
fix
bisset-a Mar 18, 2025
9441a6b
fix
bisset-a Mar 18, 2025
b454928
create org level hourly_spend model
bisset-a Mar 18, 2025
c622d6f
remove account_name col from stg_remaining_balance_daily
bisset-a Mar 18, 2025
f575b39
remove account_name col from stg_remaining_balance_daily downstreams
bisset-a Mar 18, 2025
eec7af9
fix
bisset-a Mar 18, 2025
665d6b2
fix for daily_rates
bisset-a Mar 18, 2025
05986b9
removing invalid columns from hourly_spend_org_level
bisset-a Mar 18, 2025
f6596b1
adding account_name to hourly_spend_org_level
bisset-a Mar 18, 2025
da2a75b
adding aliases in hourly_spend_org_level
bisset-a Mar 18, 2025
2d5b055
syntax fix hourly_spend_org_level
bisset-a Mar 18, 2025
940683f
adding account_locator it unique key of incremental models
bisset-a Mar 18, 2025
b48479a
update cost_per_query to support org views
bisset-a Mar 18, 2025
920a041
syntax fix cost_per_query
bisset-a Mar 18, 2025
b89364e
typo
bisset-a Mar 18, 2025
fdd0501
create org level daily_spend model
bisset-a Mar 18, 2025
ee96481
dynamic unique key cost_per_query
bisset-a Mar 19, 2025
f0a7fff
dynamic unique key stg_warehouse_metering_history
bisset-a Mar 19, 2025
bf5679c
fix columns cost_per_query
bisset-a Mar 19, 2025
b53faa9
change daily_rates to use account_locator instead of account_name
bisset-a Mar 19, 2025
cb5585c
remove invalid cols
bisset-a Mar 19, 2025
5576c07
dynamic unique key incrementals
bisset-a Mar 19, 2025
f76aca5
add account_locator to daily_rates partition
bisset-a Mar 19, 2025
2cd82b7
replace org columns and unique key with macros
bisset-a Mar 24, 2025
a2b34c0
removing unique key macro
bisset-a Mar 24, 2025
701fe29
adding comments to conditionally disabled models
bisset-a Mar 25, 2025
29d325e
Merge branch 'main' into annab-support-org-views
bisset-a Mar 25, 2025
62cb54b
adding comments to conditionally disabled models
bisset-a Mar 25, 2025
2add855
remove redundant filter cost_per_query
bisset-a Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions macros/add_account_columns.sql
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{% macro add_account_columns() %}
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% else %}
current_organization_name() as organization_name,
current_account_name() as account_name,
current_account() as account_locator,
{% endif %}
{% endmacro %}
7 changes: 7 additions & 0 deletions macros/generate_scoped_unique_key.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{% macro generate_scoped_unique_key(base_fields) %}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realised that if we're now adding the account_name column everywhere, we can always add the account_name to the unique key config! :)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realised this too!! 😆 my only concern was for backwards compatibility for incremental models, account_name would be null for historical data but I think in this case it might be ok to have nulls in the unique_key since the other fields will be unique anyway. other option would be to create a new column called _unique_key that uses generate_surrogate_keys to handle the nulls and use that as the unique_key. wdyt?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a composite PK works totally fine without making a new column - honestly I could go either way.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect - lets leave it as it! thanks Niall!

{% if var('uses_org_view', false) %}
['account_name', {{ base_fields|join(', ') }}]
{% else %}
{{ base_fields }}
{% endif %}
{% endmacro %}
40 changes: 7 additions & 33 deletions models/cost_per_query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
{{ config(
materialized='incremental',
unique_key=
['query_id', 'start_time', 'account_locator'] if var('uses_org_view', false) else
['query_id', 'start_time']
unique_key=generate_scoped_unique_key(['query_id', 'start_time'])
) }}

with
Expand All @@ -13,11 +11,7 @@ stop_threshold as (

filtered_queries as (
select
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% endif %}
{{ add_account_columns() }}
query_id,
query_text as original_query_text,
credits_used_cloud_services,
Expand Down Expand Up @@ -127,25 +121,19 @@ query_cost as (
inner join credits_billed_hourly
on query_seconds_per_hour.warehouse_id = credits_billed_hourly.warehouse_id
and query_seconds_per_hour.hour = credits_billed_hourly.hour
{% if var('uses_org_view', false) %}
and query_seconds_per_hour.account_locator = credits_billed_hourly.account_locator
{% endif %}
and query_seconds_per_hour.account_name = credits_billed_hourly.account_name
inner join {{ ref('daily_rates') }} as daily_rates
on date(query_seconds_per_hour.start_time) = daily_rates.date
and daily_rates.service_type = 'WAREHOUSE_METERING'
and daily_rates.usage_type = 'compute'
{% if var('uses_org_view', false) %}
and daily_rates.account_locator = query_seconds_per_hour.account_locator
{% endif %}
and daily_rates.account_name = query_seconds_per_hour.account_name
),

cost_per_query as (
select
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% endif %}
query_id,
any_value(start_time) as start_time,
any_value(end_time) as end_time,
Expand All @@ -165,11 +153,9 @@ cost_per_query as (
credits_billed_daily as (
select
date(hour) as date,
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% endif %}
sum(credits_used_compute) as daily_credits_used_compute,
sum(credits_used_cloud_services) as daily_credits_used_cloud_services,
greatest(daily_credits_used_cloud_services - daily_credits_used_compute * 0.1, 0) as daily_billable_cloud_services
Expand All @@ -179,11 +165,9 @@ credits_billed_daily as (

all_queries as (
select
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% endif %}
query_id,
start_time,
end_time,
Expand All @@ -201,11 +185,9 @@ all_queries as (
union all

select
{% if var('uses_org_view', false) %}
organization_name,
account_name,
account_locator,
{% endif %}
query_id,
start_time,
end_time,
Expand All @@ -224,11 +206,9 @@ all_queries as (
)

select
{% if var('uses_org_view', false) %}
all_queries.organization_name,
all_queries.account_name,
all_queries.account_locator,
{% endif %}
all_queries.query_id,
all_queries.start_time,
all_queries.end_time,
Expand Down Expand Up @@ -257,21 +237,15 @@ select
from all_queries
inner join credits_billed_daily
on date(all_queries.start_time) = credits_billed_daily.date
{% if var('uses_org_view', false) %}
and all_queries.account_locator = credits_billed_daily.account_locator
{% endif %}
and all_queries.account_name = credits_billed_daily.account_name
left join {{ ref('daily_rates') }} as daily_rates
on date(all_queries.start_time) = daily_rates.date
and daily_rates.service_type = 'CLOUD_SERVICES'
and daily_rates.usage_type = 'cloud services'
{% if var('uses_org_view', false) %}
and daily_rates.account_locator = all_queries.account_locator
{% endif %}
and daily_rates.account_name = all_queries.account_name
inner join {{ ref('daily_rates') }} as current_rates
on current_rates.is_latest_rate
and current_rates.service_type = 'CLOUD_SERVICES'
and current_rates.usage_type = 'cloud services'
{% if var('uses_org_view', false) %}
and current_rates.account_locator = all_queries.account_locator
{% endif %}
and current_rates.account_name = all_queries.account_name
order by all_queries.start_time asc
58 changes: 15 additions & 43 deletions models/daily_rates.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,21 +84,17 @@ rates_date_range_w_usage_types as (
select
date_range.start_date,
date_range.end_date,
{% if var('uses_org_view', false) %}
usage_types.account_locator,
{% endif %}
usage_types.account_name,
usage_types.usage_type
from date_range
cross join (select distinct rate_sheet_daily.account_locator, rate_sheet_daily.usage_type from rate_sheet_daily) as usage_types
cross join (select distinct rate_sheet_daily.account_name, rate_sheet_daily.usage_type from rate_sheet_daily) as usage_types
),

base as (
select
db.date,
dr.usage_type,
{% if var('uses_org_view', false) %}
dr.account_locator,
{% endif %}
dr.account_name
from dates_base as db
inner join rates_date_range_w_usage_types as dr
on db.date between dr.start_date and dr.end_date
Expand All @@ -108,57 +104,43 @@ rates_w_overage as (
select
base.date,
base.usage_type,
{% if var('uses_org_view', false) %}
base.account_locator,
{% endif %}
base.account_name,
coalesce(
rate_sheet_daily.service_type,
lag(rate_sheet_daily.service_type) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
),
lead(rate_sheet_daily.service_type) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
)
) as service_type,
coalesce(
rate_sheet_daily.effective_rate,
lag(rate_sheet_daily.effective_rate) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
),
lead(rate_sheet_daily.effective_rate) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
)
) as effective_rate,
coalesce(
rate_sheet_daily.currency,
lag(rate_sheet_daily.currency) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
),
lead(rate_sheet_daily.currency) ignore nulls over (
partition by base.usage_type
{% if var('uses_org_view', false) %}
, base.account_locator
{% endif %}
, base.account_name
order by base.date
)
) as currency,
Expand All @@ -178,17 +160,13 @@ rates_w_overage as (
left join rate_sheet_daily
on base.date = rate_sheet_daily.date
and base.usage_type = rate_sheet_daily.usage_type
{% if var('uses_org_view', false) %}
and base.account_locator = rate_sheet_daily.account_locator
{% endif %}
and base.account_name = rate_sheet_daily.account_name
),

rates as (
select
date,
{% if var('uses_org_view', false) %}
account_locator,
{% endif %}
account_name,
usage_type,
associated_usage_type,
service_type,
Expand All @@ -198,28 +176,22 @@ rates as (
from rates_w_overage
qualify row_number() over (
partition by date, service_type, associated_usage_type
{% if var('uses_org_view', false) %}
, account_locator
{% endif %}
, account_name
order by rate_priority desc
) = 1
)

select
date,
{% if var('uses_org_view', false) %}
account_locator,
{% endif %}
account_name,
associated_usage_type as usage_type,
service_type,
effective_rate,
currency,
is_overage_rate,
row_number() over (
partition by service_type, associated_usage_type
{% if var('uses_org_view', false) %}
, account_locator
{% endif %}
, account_name
order by date desc
) = 1 as is_latest_rate
from rates
Expand Down
18 changes: 0 additions & 18 deletions models/daily_spend_org_level.sql
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion models/dbt_queries.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{{ config(
materialized='incremental',
unique_key=['query_id', 'account_locator', 'start_time'] if var('uses_org_view', false) else ['query_id', 'start_time']
unique_key=generate_scoped_unique_key(['query_id', 'start_time'])
) }}

select
Expand Down
Loading