Description
Is this your first time submitting a feature request?
- I have read the expectations for open source contributors
- I have searched the existing issues, and I could not find an existing issue for this feature
- I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion
Describe the feature
When we add cluster keys to an incremental model, there is an order by
added to the model SQL that sorts the model before the merge.
With extremely large tables (ie those most likely to require clustering), this 'front-loads' some of the clustering into the model run itself. I find this undesirable for two reasons:
- It increases the length of our dbt runs as opposed to letting the entire cluster process happen async
- It 'conceals' some of the cost associated with clustering — this sort is not part of the clustering credits consumed and it's harder to recognise that cost when understanding the true cost of your clustering on an ongoing basis.
I am proposing to add a flag, where the default value is to keep things as is, and when the flag is True
that the order by
is not added to the model. I do actually think that a change in default behaviour would be desirable for most folks, but I feel like I'm more likely to get a greenlight on a version of this that doesn't alter behavior.
Describe alternatives you've considered
Overriding the macro or creating a custom materialisation.
Who will this benefit?
Folks with big data™.
Are you interested in contributing this feature?
Yes!
Anything else?
No response