Replies: 1 comment
-
|
It turns out that the bigquery method replaces the entire partition and so expects the full partition to be in the temp table, which it isn't for us. Therefore, we need to use standard incremental for bigquery as well! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The package is soon going to be tested for and then migrated to the new optimised
incrementalstrategy. See the utils PR here: snowplow/dbt-snowplow-utils#117In general the change is quite straight forward. For the models:
snowplow_optimisesetting in the config and set it totrueincremental strategy(technically not required but adds benefit for bigquery and lets users now override this in a single place in a variable).There are a few open questions around the best way to manage some of this still, which it would be good to get opinions on.
snowplow__incremental_materializationin the package or remove this entirely and just hard-code the incremental method for those tables?2) While theincremental strategydoes allow us to use the already optimisedinsert_overwritefor bigquery, it also adds an extra line to every config and an extra variable. In theory this method is the same across all incremental models where it is called, so the user could set it at a models level in their project if they wanted to. If we removed it and they did not set it, they still get the same benefit as our method is close to the way that method uses partitions anyway.snowplow__disable_upsert_lookbackshould be listed in the variables in the project or just in our docs site?Beta Was this translation helpful? Give feedback.
All reactions