allow upstream dependencies in DBT assets that aren't sources #21077
mathew-mwi
started this conversation in
Ideas
Replies: 1 comment
-
|
I just want to reiterate that I would love for this to be possible. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
If I create a dbt asset the only way I can link to an upstream process is via the sources meta reference but this doesn't work in all situations, for example I am using DBT athena and one of the tasks my model is doing is loading external sources via DBT and not another asset, this means that I don't have an upstream dependency. However, I still want my processes to run in a specific sequence, for example, if I re-run a partition I need to clear out old data from S3 as part of AWS Athena doesn't support overwriting data for unloads, so I need to have an asset that clears the location before my DBT model runs.
Another example is database set up processes, I may need to run scripts in my database in parallel, to set up a tonne of schemas, stages, objects, permissions, etc. this process is run sequentially in DBT but is far more efficient in an asset as I can run in parallel.
Please add in a feature to define upstream dependencies for DBT assets that are just upstream processes of the dbt model but not sources.
Beta Was this translation helpful? Give feedback.
All reactions