Skip to content

[ADAP-548] [Feature] Flag to opt out of order by when cluster keys are specified #735

Open
@jaypeedevlin

Description

@jaypeedevlin

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

When we add cluster keys to an incremental model, there is an order by added to the model SQL that sorts the model before the merge.

With extremely large tables (ie those most likely to require clustering), this 'front-loads' some of the clustering into the model run itself. I find this undesirable for two reasons:

  1. It increases the length of our dbt runs as opposed to letting the entire cluster process happen async
  2. It 'conceals' some of the cost associated with clustering — this sort is not part of the clustering credits consumed and it's harder to recognise that cost when understanding the true cost of your clustering on an ongoing basis.

I am proposing to add a flag, where the default value is to keep things as is, and when the flag is True that the order by is not added to the model. I do actually think that a change in default behaviour would be desirable for most folks, but I feel like I'm more likely to get a greenlight on a version of this that doesn't alter behavior.

Describe alternatives you've considered

Overriding the macro or creating a custom materialisation.

Who will this benefit?

Folks with big data™.

Are you interested in contributing this feature?

Yes!

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions