Skip to content

Commit 3117153

Browse files
committed
docs: add Fabric adapter support; refactor get_dbtreplace_directory_pattern to dispatch
- Add Microsoft Fabric Data Warehouse and Fabric Spark to supported adapter lists in README and docs/index.md - Update Additional setup headings to include Fabric - Update Limitations section and max_depth_dag docs to include Fabric (no recursive CTE support, uses loop-based approach, defaults to 9) - Refactor get_dbtreplace_directory_pattern from if target.type guard to adapter.dispatch pattern, consistent with recursive_dag.sql
1 parent ef2a357 commit 3117153

4 files changed

Lines changed: 29 additions & 19 deletions

File tree

README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,16 @@ Currently, the following adapters are supported:
2424
- AWS Athena (tested manually)
2525
- Greenplum (tested manually)
2626
- ClickHouse (tested manually)
27+
- Microsoft Fabric Data Warehouse (tested manually)
28+
- Microsoft Fabric Spark (tested manually)
2729

2830
## Using This Package
2931

3032
### Cloning via dbt Package Hub
3133

3234
Check [dbt Hub](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) for the latest installation instructions, or [read the docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
3335

34-
### Additional setup for Databricks/Spark/DuckDB/Redshift/ClickHouse
36+
### Additional setup for Databricks/Spark/DuckDB/Redshift/ClickHouse/Fabric
3537

3638
In your `dbt_project.yml`, add the following config:
3739

docs/customization/overriding-variables.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,14 +103,14 @@ vars:
103103

104104
| variable | description | default |
105105
| ----------- | ----------- | ----------- |
106-
| `max_depth_dag` | limits the maximum distance between nodes calculated in `int_all_dag_relationships` | 9 for bigquery and spark, -1 for other adatpters |
106+
| `max_depth_dag` | limits the maximum distance between nodes calculated in `int_all_dag_relationships` | 9 for bigquery, spark, and fabric, -1 for other adapters |
107107
| `insert_batch_size` | number of records inserted per batch when unpacking the graph into models | 10000 |
108108

109109
**Note on max_depth_dag**
110110

111111
The default behavior for limiting the relationships calculated in the `int_all_dag_relationships` model differs depending on your adapter.
112112

113-
- For Bigquery & Spark/Databricks the maximum distance between two nodes in your DAG, calculated in `int_all_dag_relationships`, is set by the `max_depth_dag` variable, which is defaulted to 9. So by default, `int_all_dag_relationships` contains a row for every path less than or equal to 9 nodes in length between two nodes in your DAG. This is because these adapters do not currently support recursive SQL, and queries often fail on more than 9 recursive joins.
113+
- For BigQuery, Spark/Databricks, and Microsoft Fabric Data Warehouse the maximum distance between two nodes in your DAG, calculated in `int_all_dag_relationships`, is set by the `max_depth_dag` variable, which is defaulted to 9. So by default, `int_all_dag_relationships` contains a row for every path less than or equal to 9 nodes in length between two nodes in your DAG. This is because these adapters do not currently support recursive SQL, and queries often fail on more than 9 recursive joins.
114114
- For all other adapters `int_all_dag_relationships` by default contains a row for every single path between two nodes in your DAG. If you experience long runtimes for the `int_all_dag_relationships` model, you may consider limiting the length of your generated DAG paths. To do this, set `max_depth_dag: {{ whatever limit you want to enforce }}`. The value of `max_depth_dag` must be greater than 2 for all DAG tests to work, and greater than `chained_views_threshold` to ensure your performance tests to work. By default, the value of this variable for these adapters is -1, which the package interprets as "no limit".
115115

116116
```yaml title="dbt_project.yml"

docs/index.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,16 @@ Currently, the following adapters are supported:
2525
- AWS Athena (tested manually)
2626
- Greenplum (tested manually)
2727
- ClickHouse (tested manually)
28+
- Microsoft Fabric Data Warehouse (tested manually)
29+
- Microsoft Fabric Spark (tested manually)
2830

2931
## Using This Package
3032

3133
### Cloning via dbt Package Hub
3234

3335
Check [dbt Hub](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) for the latest installation instructions, or [read the docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
3436

35-
### Additional setup for Databricks/Spark/DuckDB/Redshift
37+
### Additional setup for Databricks/Spark/DuckDB/Redshift/Fabric
3638

3739
In your `dbt_project.yml`, add the following config:
3840

@@ -64,8 +66,8 @@ Each test warning indicates the presence of a type of misalignment. To troublesh
6466

6567
## Limitations
6668

67-
### BigQuery and Databricks
69+
### BigQuery, Databricks, and Microsoft Fabric Data Warehouse
6870

69-
BigQuery current support for recursive CTEs is limited and Databricks SQL doesn't support recursive CTEs.
71+
BigQuery has limited support for recursive CTEs, while Databricks SQL and Microsoft Fabric Data Warehouse do not support them.
7072

7173
For those Data Warehouses, the model `int_all_dag_relationships` needs to be created by looping CTEs instead. The number of loops is configured with `max_depth_dag` and defaulted to 9. This means that dependencies between models of more than 9 levels of separation won't show in the model `int_all_dag_relationships` but tests on the DAG will still be correct. With a number of loops higher than 9 BigQuery sometimes raises an error saying the query is too complex.

macros/get_directory_pattern.sql

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,21 +23,27 @@
2323
{% endmacro %}
2424

2525
{% macro get_dbtreplace_directory_pattern() %}
26+
{{ return(adapter.dispatch('get_dbtreplace_directory_pattern', 'dbt_project_evaluator')()) }}
27+
{% endmacro %}
28+
29+
{% macro default__get_dbtreplace_directory_pattern() %}
30+
{% if execute %}
31+
{%- set on_mac_or_linux = dbt_project_evaluator.is_os_mac_or_linux() -%}
32+
{%- if on_mac_or_linux -%}
33+
{{ dbt.replace("file_path", "regexp_replace(file_path,'.*/','')", "''") }}
34+
{% else %}
35+
{{ dbt.replace("file_path", "regexp_replace(file_path,'.*\\\\\\\\','')", "''") }}
36+
{% endif %}
37+
{% endif %}
38+
{% endmacro %}
39+
40+
{% macro fabric__get_dbtreplace_directory_pattern() %}
2641
{% if execute %}
27-
{%- if target.type == 'fabric' -%}
28-
{%- set on_mac_or_linux = dbt_project_evaluator.is_os_mac_or_linux() -%}
29-
{%- if on_mac_or_linux -%}
30-
left(file_path, len(file_path) - charindex('/', reverse(file_path)))
31-
{%- else -%}
32-
left(file_path, len(file_path) - charindex('\', reverse(file_path)))
33-
{%- endif -%}
42+
{%- set on_mac_or_linux = dbt_project_evaluator.is_os_mac_or_linux() -%}
43+
{%- if on_mac_or_linux -%}
44+
left(file_path, len(file_path) - charindex('/', reverse(file_path)))
3445
{%- else -%}
35-
{%- set on_mac_or_linux = dbt_project_evaluator.is_os_mac_or_linux() -%}
36-
{%- if on_mac_or_linux -%}
37-
{{ dbt.replace("file_path", "regexp_replace(file_path,'.*/','')", "''") }}
38-
{% else %}
39-
{{ dbt.replace("file_path", "regexp_replace(file_path,'.*\\\\\\\\','')", "''") }}
40-
{% endif %}
46+
left(file_path, len(file_path) - charindex('\', reverse(file_path)))
4147
{%- endif -%}
4248
{% endif %}
4349
{% endmacro %}

0 commit comments

Comments
 (0)