dbt 0.21.0b1

An initial beta release for v0.21.0.

Breaking changes

• Add full node selection to source freshness command and align selection syntax with other tasks (dbt source freshness --select source_name --> dbt source freshness --select source:souce_name) and rename dbt source snapshot-freshness -> dbt source freshness. (#2987, #3554)
• dbt-snowflake: Turn off transactions and turn on autocommit by default. Explicitly specify begin and commit for DML statements in incremental and snapshot materializations. Note that this may affect user-space code that depends on transactions.

Features

• Add dbt build command to run models, tests, seeds, and snapshots in DAG order. (#2743, #3490, #3608)
• Introduce on_schema_change config to detect and handle schema changes on incremental models (#1132, #3387)

Fixes

• Fix docs generation for cross-db sources in REDSHIFT RA3 node (#3236, #3408)
• Fix type coercion issues when fetching query result sets (#2984, #3499)
• Handle whitespace after a plus sign on the project config (#3526)
• Fix table and view materialization issue when switching from one to the other (#2161), #3547)

Under the hood

• Improve default view and table materialization performance by checking relational cache before attempting to drop temp relations (#3112, #3468)
• Add optional sslcert, sslkey, and sslrootcert profile arguments to the Postgres connector. (#3472, #3473)
• Move the example project used by dbt init into dbt repository, to avoid cloning an external repo (#3005, #3474, #3536)
• Better interaction between dbt init and adapters. Avoid raising errors while initializing a project (#2814, #3483)
• Update create_adapter_plugins script to include latest accessories, and stay up to date with latest dbt-core version (#3002, #3509)
• Scrub environment secrets from logs and console output (#3617)

Dependencies

• Require werkzeug>=1 (#3590)

Contributors:

• @kostek-pl (#3236)
• @matt-winkler (#3387)
• @tconbeer #3468)
• @JLDLaughlin (#3473)
• @jmriego (#3526)
• @danielefrigo (#3547)

dbt 0.20.1rc1

An initial release candidate for v0.20.1.

Features

• Adds install-prerelease parameter to hub packages in packages.yml. When set to True, allows prerelease packages to be installed. By default, this parameter is False unless explicitly set to True.

Fixes

• Fix config merge behavior with experimental parser 3637
• Fix exception on yml files with all comments 3568
• Fix store_failures config when defined as a modifier for unique and not_null tests (#3575, #3577)
• Fix where config with relationships test by refactoring test SQL. Note: The default relationships test now includes CTEs, and may need reimplementing on adapters that don't support CTEs nested inside subqueries. (#3579, #3583)
• Partial parsing: don't reprocess SQL file already scheduled (#3589, #3620)
• Handle interator functions in model config (#3573)
• Partial parsing: fix error after changing empty yaml file (#3567, #3618)
• Partial parsing: handle source tests when changing test macro (#3584, #3620)
• Fix dbt deps version comparison logic which was causing incorrect pre-release package versions to be installed. (#3578, #3609)
• Partial parsing: schedule new macro file for parsing when macro patching (#3627, #3627)
• Use SchemaParser's render context to render test configs in order to support var() configured at the project level and passed in from the cli (#3564. #3646)
• Partial parsing: check unique_ids when recursively removing macros (#3636)

Under the hood

• Add tracking for determine why dbt needs to re-parse entire project when partial parsing is enabled (#3572, #3652)

Docs

• Fix docs site crash if relationships test has one dependency instead of two (docs#207, (docs#208))

Under the hood

• Handle exceptions from anonymous usage tracking for users of dbt-snowflake on Apple M1 chips (#3162, #3661)

Contributors:

• @NiallRees (#3624)

dbt 0.20.0

dbt 0.20.0 - Margaret Mead (July 12, 2021)

• 💬 Discussion: v0.20.0
• 📖 New and changed documentation

Breaking changes

• Add Jinja tag for generic test definitions. Replacement for macros prefixed test_ (#1173, #3261)
• Update schema/generic tests to expect a set of rows instead of a single numeric value, and to use test materialization when executing. (#3192, #3286)
• Plugin maintainers: For adapters that inherit from other adapters (e.g. dbt-postgres → dbt-redshift), adapter.dispatch() will now include parent macro implementations as viable candidates (#2923, #3296)
• Fix adapter.dispatch macro resolution when statically extracting macros. Introduce new project-level dispatch config. The packages argument to dispatch no longer supports macro calls; there is backwards compatibility for existing packages. The argument will no longer be supported in a future release, instead provide the macro_namespace argument. (#3362, #3363, #3383, #3403)
• Bump schema versions for run results and manifest artifacts (#3422, #3421)

Features

Tests

• Add a new materialization for tests, update data tests to use test materialization when executing. (#3154, #3181)
• Support disabling schema tests, and configuring tests from dbt_project.yml (#3252,
  #3253, #3257)
• Less greedy test selection: expand to select unselected tests if and only if all parents are selected (#2891, #3235)
• Store test failures in the database (#517, #903, #2593, #3316)
• Add new test configs: where, limit, warn_if, error_if, fail_calc (#3258, #3321, #3336)

Parsing

• Add the opt-in --use-experimental-parser flag (#3307, #3374)
• Move partial parsing to end of parsing and implement new partial parsing method. (#3217, #3364)
• Save doc file node references and use in partial parsing. (#3425, #3432)

Packages

• Support commit hashes in dbt deps package revision (#3268, #3270)
• Add optional subdirectory key to install dbt packages that are not hosted at the root of a Git repository (#275, #3267)

Core

• Add tags and meta properties to the exposure resource schema. (#3404, #3405)
• Support optional updated_at config parameter with check strategy snapshots. If not supplied, will use current timestamp (default). (#1844, #3376)

Plugins

• Add optional configs for require_partition_filter and partition_expiration_days in BigQuery (#1843, #2928)
• Fix for EOL SQL comments prevent entire line execution (#2731, #2974)
• Add optional merge_update_columns config to specify columns to update for merge statements in BigQuery and Snowflake (#1862, #3100)
• Use query comment JSON as job labels for BigQuery adapter when query-comment.job-label is set to true (#2483), (#3145)
• Set application_name for Postgres connections (#885, #3182)
• Add native support for Postgres index creation (#804, 3106)
• Prevent locks in Redshift during full refresh in incremental materialization. (#2426, #2998)

Docs

• Reversed the rendering direction of relationship tests so that the test renders in the model it is defined in (docs#181, docs#183)
• Support dots in model names: display them in the graphs (docs#184, docs#185)
• Render meta tags for sources (docs#192, docs#193)
• Display tags on exposures (docs#194, docs#195)
• Update dbt logo and links (docs#197)

Fixes

• Fix exit code from dbt debug not returning a failure when one of the tests fail (#3017, #3018)
• Auto-generated CTEs in tests and ephemeral models have lowercase names to comply with dbt coding conventions (#3027, #3028)
• Fix incorrect error message when a selector does not match any node #3036)
• Fix variable _dbt_max_partition declaration and initialization for BigQuery incremental models (#2940, #2976)
• Moving from 'master' to 'HEAD' default branch in git (#3057, #3104, #3117))
• Requirement on dataclasses is relaxed to be between >=0.6,<0.9 allowing dbt to cohabit with other libraries which required higher versions. (#3150, #3151)
• Add feature to add _n alias to same column names in SQL query (#3147, #3158)
• Raise a proper error message if dbt parses a macro twice due to macro duplication or misconfiguration. (#2449, #3165)
• Fix exposures missing in graph context variable. (#3241, #3243)
• Ensure that schema test macros are properly processed (#3229, #3272)
• Use absolute path for profiles directory instead of a path relative to the project directory. Note: If a user supplies a relative path to the profiles directory, the value of args.profiles_dir will still be absolute. (#3133, #3176)
• Fix FQN selector unable to find models whose name contains dots (#3246, #3247)
• Handle quoted values within test configs, such as where (#3458, #3459)
• Avoid slowdown in column-level `pers...

dbt 0.20.0rc2

A second release candidate for v0.20.0.

Fixes

• Handle quoted values within test configs, such as where (#3458, #3459)

Docs

• Display tags on exposures (docs#194, docs#195)

Under the hood

• Swap experimental parser implementation to use Rust #3497
• Dispatch the core SQL statement of the new test materialization, to benefit adapter maintainers (#3465, #3461)
• Minimal validation of yaml dictionaries prior to partial parsing (#3246, #3460)
• Add partial parsing tests and improve partial parsing handling of macros (#3449, #3505)
• Update project loading event data to include experimental parser information. (#3438, #3495)

dbt 0.19.2

Breaking changes

• Fix adapter.dispatch macro resolution when statically extracting macros. Introduce new project-level dispatch config. The packages argument to dispatch no longer supports macro calls; there is backwards compatibility for existing packages. The argument will no longer be supported in a future release, instead provide the macro_namespace argument. (#3362, #3363, #3383, #3403)

Fixes

• Ensure that schema test macros are properly processed (#3229, #3272)
• Fix regression for default project/database for BigQuery connections (#3218, #3305)
• Fix references to macros with package names when rendering schema tests (#3324, #3345)
• Fix infinite recursion when parsing schema tests due to loops in macro calls (#3444, #3454)

dbt 0.20.0rc1

An initial release candidate for v0.20.0.

Breaking changes

• Fix adapter.dispatch macro resolution when statically extracting macros. Introduce new project-level dispatch config. The packages argument to dispatch no longer supports macro calls; there is backwards compatibility for existing packages. The argument will no longer be supported in a future release, instead provide the macro_namespace argument. (#3362, #3363, #3383, #3403)

Features

• Support optional updated_at config parameter with check strategy snapshots. If not supplied, will use current timestamp (default). (#1844, #3376)
• Add the opt-in --use-experimental-parser flag (#3307, #3374)
• Store test failures in the database (#517, #903, #2593, #3316)
• Add new test configs: where, limit, warn_if, error_if, fail_calc (#3258, #3321, #3336)
• Move partial parsing to end of parsing and implement new partial parsing method. (#3217, #3364)
• Save doc file node references and use in partial parsing. (#3425, #3432)

Fixes

• Fix compiled sql for ephemeral models (#3317, #3318)
• Now generating run_results.json even when no nodes are selected (#3313, #3315)
• Add missing packaging dependency (#3312, #3339)
• Fix references to macros with package names when rendering schema tests (#3324, #3345)
• Stop clobbering default keyword arguments for jinja test definitions (#3329, #3340)
• Fix unique_id generation for generic tests so tests with the same FQN but different configuration will run. (#3254, #3335)
• Update the snowflake adapter to only comment on a column if it exists when using the persist_docs config (#3039, #3149)
• Add a better error messages for undefined macros and when there are less packages installed than specified in packages.yml. (#2999)
• Separate compiled_path from build_path, and print the former alongside node error messages (#1985, #3327)
• Fix exception caused when running dbt debug with BigQuery connections (#3314, #3351)
• Raise better error if snapshot is missing required configurations (#3381, #3385)
• Fix dbt run errors caused from receiving non-JSON responses from Snowflake with Oauth (#3350

Docs

• Reversed the rendering direction of relationship tests so that the test renders in the model it is defined in (docs#181, docs#183)
• Support dots in model names: display them in the graphs (docs#184, docs#185)
• Render meta tags for sources (docs#192, docs#193)

Under the hood

• Added logic for registry requests to raise a timeout error after a response hangs out for 30 seconds and 5 attempts have been made to reach the endpoint (#3177, #3275)
• Added support for invoking the list task via the RPC server (#3311, #3384)
• Added unique_id and original_file_path as keys to json responses from the list task (#3356, #3384)
• Use shutil.which so Windows can pick up git.bat as a git executable (#3035, #3134)
• Add ssh-client and update git version (using buster backports) in Docker image (#3337, #3338)
• Add tags and meta properties to the exposure resource schema. (#3404, #3405)
• Update test sub-query alias (#3398, #3414)
• Bump schema versions for run results and manifest artifacts (#3422, #3421)
• Add deprecation warning for using packages argument with adapter.dispatch (#3419, #3420)

dbt 0.19.2rc2

A second release candidate for v0.19.2.

Breaking changes

• Fix adapter.dispatch macro resolution when statically extracting macros. Introduce new project-level dispatch config. The packages argument to dispatch no longer supports macro calls; there is backwards compatibility for existing packages. The argument will no longer be supported in a future release, instead provide the macro_namespace argument. (#3362, #3363, #3383, #3403)

Fixes

• Fix references to macros with package names when rendering schema tests (#3324, #3345)

dbt 0.20.0b1

An initial beta release for v0.20.0.

dbt 0.19.2rc1

An initial release candidate for v0.19.2.

Fixes

• Ensure that schema test macros are properly processed (#3229, #3272)
• Fix regression for default project/database for BigQuery connections (#3218, #3305)

dbt 0.19.1

Fixes

• On BigQuery, fix regressions for insert_overwrite incremental strategy with int64 and timestamp partition columns (#3063, #3095, #3098)
• Fix compiled sql for ephemeral models (#3139, #3056)
• Pass service-account scopes to gcloud-based oauth (#3040, #3041)

Under the hood

• Bump werkzeug upper bound dependency to <v2.0 (#3011)
• Performance fixes for many different things (#2862, #3034)
• Update code to use Mashumaro 2.0 (#3138)
• Pin agate<1.6.2 to avoid installation errors relating to its new dependency PyICU (#3160, #3161)
• Add an event to track resource counts (#3050, #3157)

Contributors:

• @Bl3f (#3011)
• @yu-iskw (#3041)