Skip to content

Commit 0c08087

Browse files
callum-mcdatacallum-mcdata
authored
Merge pull request #115 from dbt-labs/adding_secondary_calc_limit
Secondary Calcs - Lets Limit These Inputs
2 parents 3b87227 + c7908a8 commit 0c08087

10 files changed

Lines changed: 265 additions & 103 deletions

File tree

‎README.md‎

Lines changed: 26 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
* [Secondary calculation column aliases](#secondary-calculation-column-aliases)
2929

3030
<!-- Created by https://github.com/ekalinin/github-markdown-toc -->
31-
<!-- Added by: runner, at: Tue Aug 23 15:13:02 UTC 2022 -->
31+
<!-- Added by: runner, at: Wed Sep 21 02:26:03 UTC 2022 -->
3232

3333
<!--te-->
3434

@@ -138,7 +138,7 @@ You may want to materialize the results as a fixed table for querying. This is n
138138
When [dbt server](https://blog.getdbt.com/licensing-dbt/) is released in late 2022, you will be able to access these macros interactively, without needing to build each variant as a single dbt model. For more information, check out the [keynote presentation from Coalesce 2021](https://www.getdbt.com/coalesce-2021/keynote-the-metrics-system/).
139139

140140
# Secondary calculations
141-
Secondary calculations are window functions which act on the primary metric. You can use them to compare a metric's value to an earlier period and calculate year-to-date sums or rolling averages.
141+
Secondary calculations are window functions which act on the primary metric or metrics. You can use them to compare values to an earlier period and calculate year-to-date sums or rolling averages.
142142

143143
Create secondary calculations using the convenience [constructor](https://en.wikipedia.org/wiki/Constructor_(object-oriented_programming)) macros. Alternatively, you can manually create a list of dictionary entries (not recommended).
144144

@@ -157,29 +157,40 @@ Create secondary calculations using the convenience [constructor](https://en.wik
157157
Column aliases are [automatically generated](#secondary-calculation-column-aliases), but you can override them by setting `alias`.
158158

159159
## Period over Period ([source](/macros/secondary_calculations/secondary_calculation_period_over_period.sql))
160+
The period over period secondary calculation performs a calculation against the metric(s) in question by either determining the difference or the ratio between two points of time. This other point in time is determined by the input variable which looks at the grain selected in the macro.
160161

161-
Constructor: `metrics.period_over_period(comparison_strategy, interval [, alias])`
162+
Constructor: `metrics.period_over_period(comparison_strategy, interval [, alias, metric_list])`
162163

163-
- `comparison_strategy`: How to calculate the delta between the two periods. One of [`"ratio"`, `"difference"`]. Required
164-
- `interval`: The number of periods to look back. Required
165-
- `alias`: The column alias for the resulting calculation. Optional
164+
| Parameter | Example | Description | Required |
165+
| -------------------------- | ----------- | ----------- | -----------|
166+
| `comparison_strategy` | `ratio` or `difference` | How to calculate the delta between the two periods | Yes |
167+
| `interval` | 1 | Integer - the number of time grains to look back | Yes |
168+
| `alias` | 'week_over_week' | The column alias for the resulting calculation | No |
169+
| `metric_list` | 'base_sum_metric' | List of metrics that the secondary calculation should be applied to. Default is all metrics selected | No |
166170

167171
## Period to Date ([source](/macros/secondary_calculations/secondary_calculation_period_to_date.sql))
172+
The period to date secondary calculation performs an aggregation on a defined **period** of time that is equal to or coarser (higher, more aggregated) than the grain selected. Great example of this is when you want to display a month_to_date value alongside your weekly grained metric.
168173

169-
Constructor: `metrics.period_to_date(aggregate, period [, alias])`
174+
Constructor: `metrics.period_to_date(aggregate, period [, alias, metric_list])`
170175

171-
- `aggregate`: The aggregation to use in the window function. Options vary based on the primary aggregation and are enforced in [validate_aggregate_coherence()](/macros/secondary_calculations/validate_aggregate_coherence.sql). Required
172-
- `period`: The time grain to aggregate to. One of [`"day"`, `"week"`, `"month"`, `"quarter"`, `"year"`]. Must be at equal or lesser granularity than the metric's grain (see [Time Grains](#time-grains) below). Required
173-
- `alias`: The column alias for the resulting calculation. Optional
176+
| Parameter | Example | Description | Required |
177+
| -------------------------- | ----------- | ----------- | -----------|
178+
| `aggregate` | `max`, `average` | The aggregation to use in the window function. Options vary based on the primary aggregation and are enforced in [validate_aggregate_coherence()](/macros/secondary_calculations/validate_aggregate_coherence.sql). | Yes |
179+
| `period` | `"day"`, `"week"` | The time grain to aggregate to. One of [`"day"`, `"week"`, `"month"`, `"quarter"`, `"year"`]. Must be at equal or coarser (higher, more aggregated) granularity than the metric's grain (see [Time Grains](#time-grains) below). In example grain of `month`, the acceptable periods would be `month`, `quarter`, or `year`. | Yes |
180+
| `alias` | 'month_to_date' | The column alias for the resulting calculation | No |
181+
| `metric_list` | 'base_sum_metric' | List of metrics that the secondary calculation should be applied to. Default is all metrics selected | No |
174182

175183
## Rolling ([source](/macros/secondary_calculations/secondary_calculation_rolling.sql))
184+
The rolling secondary calculation performs an aggregation on a defined number of rows in metric dataset. For example, if the user selects the `week` grain and sets a rolling secondary calculation to `4` then the value returned will be a rolling 4 week calculation of whatever aggregation type was selected.
176185

177-
Constructor: `metrics.rolling(aggregate, interval [, alias])`
178-
179-
- `aggregate`: The aggregation to use in the window function. Options vary based on the primary aggregation and are enforced in [validate_aggregate_coherence()](/macros/secondary_calculations/validate_aggregate_coherence.sql). Required
180-
- `interval`: The number of periods to look back. Required
181-
- `alias`: The column alias for the resulting calculation. Optional
186+
Constructor: `metrics.rolling(aggregate, interval [, alias, metric_list])`
182187

188+
| Parameter | Example | Description | Required |
189+
| -------------------------- | ----------- | ----------- | -----------|
190+
| `aggregate` | `max`, `average` | The aggregation to use in the window function. Options vary based on the primary aggregation and are enforced in [validate_aggregate_coherence()](/macros/secondary_calculations/validate_aggregate_coherence.sql). | Yes |
191+
| `interval` | 1 | Integer - the number of time grains to look back | Yes |
192+
| `alias` | 'month_to_date' | The column alias for the resulting calculation | No |
193+
| `metric_list` | 'base_sum_metric' | List of metrics that the secondary calculation should be applied to. Default is all metrics selected | No |
183194

184195
# Customisation
185196
Most behaviour in the package can be overridden or customised.

‎integration_tests/models/metric_testing_models/multiple_metrics__period_over_period.sql‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,11 @@
55
grain='day',
66
dimensions=['had_discount'],
77
secondary_calculations=[
8-
{"calculation": "period_over_period", "interval": 1, "comparison_strategy": "difference", "alias": "pop_1mth"},
9-
{"calculation": "period_over_period", "interval": 1, "comparison_strategy": "ratio"},
10-
]
8+
metrics.period_over_period(
9+
comparison_strategy="difference"
10+
,interval=1
11+
,metric_list='base_sum_metric'
12+
)
13+
]
1114
)
1215
}}

‎macros/secondary_calculations/secondary_calculation_period_over_period.sql‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -30,25 +30,3 @@
3030
{% macro default__metric_comparison_strategy_ratio(metric_name, calc_sql) %}
3131
cast(coalesce({{ metric_name }}, 0) as {{ type_float() }}) / nullif({{ calc_sql }}, 0)
3232
{% endmacro %}
33-
34-
{% macro period_over_period(comparison_strategy, interval, alias) %}
35-
36-
{% set missing_args = [] %}
37-
{% if not comparison_strategy %}
38-
{% set _ = missing_args.append("comparison_strategy") %}
39-
{% endif %}
40-
{% if not interval %}
41-
{% set _ = missing_args.append("interval") %}
42-
{% endif %}
43-
{% if missing_args | length > 0 %}
44-
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_over_period') %}
45-
{% endif %}
46-
47-
{% do return ({
48-
"calculation": "period_over_period",
49-
"comparison_strategy": comparison_strategy,
50-
"interval": interval,
51-
"alias": alias
52-
})
53-
%}
54-
{% endmacro %}

‎macros/secondary_calculations/secondary_calculation_period_to_date.sql‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -13,26 +13,4 @@
1313

1414
{%- do return (calc_sql) %}
1515

16-
{% endmacro %}
17-
18-
{% macro period_to_date(aggregate, period, alias) %}
19-
20-
{% set missing_args = [] %}
21-
{% if not aggregate %}
22-
{% set _ = missing_args.append("aggregate") %}
23-
{% endif %}
24-
{% if not period %}
25-
{% set _ = missing_args.append("period") %}
26-
{% endif %}
27-
{% if missing_args | length > 0 %}
28-
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_to_date') %}
29-
{% endif %}
30-
31-
{% do return ({
32-
"calculation": "period_to_date",
33-
"aggregate": aggregate,
34-
"period": period,
35-
"alias": alias
36-
})
37-
%}
3816
{% endmacro %}

‎macros/secondary_calculations/secondary_calculation_rolling.sql‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -12,26 +12,4 @@
1212

1313
{% do return (calc_sql) %}
1414

15-
{% endmacro %}
16-
17-
{% macro rolling(aggregate, interval, alias) %}
18-
19-
{% set missing_args = [] %}
20-
{% if not aggregate %}
21-
{% set _ = missing_args.append("aggregate") %}
22-
{% endif %}
23-
{% if not interval %}
24-
{% set _ = missing_args.append("interval") %}
25-
{% endif %}
26-
{% if missing_args | length > 0 %}
27-
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_over_period') %}
28-
{% endif %}
29-
30-
{% do return ({
31-
"calculation": "rolling",
32-
"aggregate": aggregate,
33-
"interval": interval,
34-
"alias": alias
35-
})
36-
%}
3715
{% endmacro %}

‎macros/secondary_calculations_configuration/period_over_period.sql‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{% macro period_over_period(comparison_strategy, interval, alias, metric_list = []) %}
2+
3+
{% set missing_args = [] %}
4+
{% if not comparison_strategy %}
5+
{% set _ = missing_args.append("comparison_strategy") %}
6+
{% endif %}
7+
{% if not interval %}
8+
{% set _ = missing_args.append("interval") %}
9+
{% endif %}
10+
{% if missing_args | length > 0 %}
11+
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_over_period') %}
12+
{% endif %}
13+
{% if metric_list is string %}
14+
{% set metric_list = [metric_list] %}
15+
{% endif %}
16+
17+
{% do return ({
18+
"calculation": "period_over_period",
19+
"comparison_strategy": comparison_strategy,
20+
"interval": interval,
21+
"alias": alias,
22+
"metric_list": metric_list
23+
})
24+
%}
25+
{% endmacro %}

‎macros/secondary_calculations_configuration/period_to_date.sql‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{% macro period_to_date(aggregate, period, alias, metric_list = []) %}
2+
3+
{% set missing_args = [] %}
4+
{% if not aggregate %}
5+
{% set _ = missing_args.append("aggregate") %}
6+
{% endif %}
7+
{% if not period %}
8+
{% set _ = missing_args.append("period") %}
9+
{% endif %}
10+
{% if missing_args | length > 0 %}
11+
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_to_date') %}
12+
{% endif %}
13+
{% if metric_list is string %}
14+
{% set metric_list = [metric_list] %}
15+
{% endif %}
16+
17+
{% do return ({
18+
"calculation": "period_to_date",
19+
"aggregate": aggregate,
20+
"period": period,
21+
"alias": alias,
22+
"metric_list": metric_list
23+
})
24+
%}
25+
{% endmacro %}

‎macros/secondary_calculations_configuration/rolling.sql‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{% macro rolling(aggregate, interval, alias, metric_list=[]) %}
2+
3+
{% set missing_args = [] %}
4+
{% if not aggregate %}
5+
{% set _ = missing_args.append("aggregate") %}
6+
{% endif %}
7+
{% if not interval %}
8+
{% set _ = missing_args.append("interval") %}
9+
{% endif %}
10+
{% if missing_args | length > 0 %}
11+
{% do exceptions.raise_compiler_error( missing_args | join(", ") ~ ' not provided to period_over_period') %}
12+
{% endif %}
13+
{% if metric_list is string %}
14+
{% set metric_list = [metric_list] %}
15+
{% endif %}
16+
17+
{% do return ({
18+
"calculation": "rolling",
19+
"aggregate": aggregate,
20+
"interval": interval,
21+
"alias": alias,
22+
"metric_list": metric_list
23+
})
24+
%}
25+
{% endmacro %}

‎macros/sql_gen/gen_secondary_calculations_cte.sql‎

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -12,35 +12,28 @@ easier for not having to update the working secondary calc logic #}
1212
,secondary_calculations as (
1313

1414
select *
15-
16-
{# Checking if base_set is a list - which you'd think would have its own test but no
17-
Jinja doesn't have that built in so we have to hack it by checking if it is an
18-
iterable variable and NOT sring /mapping #}
19-
{% if base_set is iterable and (base_set is not string and base_set is not mapping) %}
20-
21-
{% for metric_name in base_set -%}
2215

23-
{% for calc_config in secondary_calculations -%}
16+
{# {% set is_multiple_metrics = base_set | length > 1 %} #}
17+
{% for calc_config in secondary_calculations -%}
18+
{% if calc_config.metric_list | length > 0 %}
19+
{% for metric_name in calc_config.metric_list -%}
20+
, {{ metrics.perform_secondary_calculation(metric_name, grain, dimensions, calc_config) -}} as {{ metrics.generate_secondary_calculation_alias(metric_name,calc_config, grain, true) }}
21+
{% endfor %}
22+
{% else %}
23+
{% for metric_name in base_set -%}
2424
, {{ metrics.perform_secondary_calculation(metric_name, grain, dimensions, calc_config) -}} as {{ metrics.generate_secondary_calculation_alias(metric_name,calc_config, grain, true) }}
25-
2625
{% endfor %}
26+
{% endif%}
27+
28+
{% endfor %}
2729

28-
{% endfor %}
29-
30-
{% else %}
31-
32-
{% for calc_config in secondary_calculations -%}
33-
, {{ metrics.perform_secondary_calculation(base_set, grain, dimensions, calc_config) -}} as {{ metrics.generate_secondary_calculation_alias(base_set,calc_config, grain, false) }}
34-
35-
{% endfor %}
3630

37-
{% endif %}
3831

3932
from
4033
{% if full_set|length > 1 %}
4134
joined_metrics
4235
{% else %}
43-
{{base_set[0]}}__final
36+
{{ base_set[0] }}__final
4437
{% endif %}
4538
)
4639

0 commit comments

Comments
 (0)