feat: microbatch strategy

[canbekley]

Adds support for the incremental "microbatch" strategy #398, that comes with dbt-core 1.9.0 #403
Currently only works for non-distributed incremental model, as dbt-core has hard-coded config.materialization = 'incremental' into a microbatching condition. Maybe this can be patched by the adapater though?

Checklist

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG

[canbekley]

@BentsiLeviav may I ask for a review on this? it would be an important addition for our pipelines

[BentsiLeviav]

Hi @canbekley

I'll review this in the next following days.
This feature depends on #403 , so feel free to answer there once you have time.
In addition, could you please add an indication of this feature to the README.md file? an explanation of how this was implemented would be appreciated

[pheepa]

Hi, that's a great feature - thanks for implementing it!

I don't quite understand the logic behind requiring partition_by for this method if microbatching is handled via a series of delete_insert strategy operations.

Wouldn't it make more sense to either remove this restriction or implement it using insert_overwrite, similar to how it's done in dbt-bigquery? Reference.

In Snowflake, delete_insert are used without requiring partition_by: Reference.

Since ClickHouse struggles when handling a large number of complex mutations (my experience), I assume it would be better to go with insert_overwrite.

[canbekley]

@pheepa I don't see where the microbatch strategy is requiring partition_by. You can also check the integration test model, which doesn't have a partition_by configuration, but a required unique_key instead. I have refactored some of the existing incremental strategy validations by moving them to a adapter method, including some of the insert_overwrite validations. Maybe that part was confusing?

I opted for delete_insert here because I think it is more flexibel, in that it doesn't require a partitioned table, and we can have arbitrary batch sizes independent of existing partition keys.

[pheepa]

@canbekley I'm sorry, the refactoring of the incremental strategy validation confused me a little bit (but as an adapter method it makes more sense). Now it looks good to me.
Let's see how delete_insert goes

[canbekley]

hi @BentsiLeviav, I've added documentation and fixed an issue with previous python versions. could you have a second look?

[canbekley]

@BentsiLeviav Do you maybe have an idea how to circumvent this check in MicrobatchModelRunner._is_incremental(), which currently prevents distributed_incremental models being run in microbatches?

[maowerner]

Hi, is there any update on this? I believe this feature is waiting for review :)