dbt 0.17.1

dbt 0.17.1 (July 20, 2020)

This is a bugfix release with one breaking change and one enhancement of an existing feature.

Breaking changes

• dbt native rendering is no longer enabled by default, as it was in v0.17.0. It requires the use of opt-in filters: as_native, as_bool, and as_number. (#2612, #2618)

Features

• On BigQuery, persist_docs persists descriptions for nested columns (#2549, #2550)

Fixes

• Hash names of local packages (#2600)
• Handle too-long file paths on Windows (#2591, #2558, #2566, #2603)
• Fix a bug where persist_docs ignored seeds on BigQuery (#2598, #2601)
• Fix a bug where dbt config-version: 2 rendered +pre-hook and +post-hook fields too eagerly (#2583, #2854)
• Fix a bug where dbt compile and dbt ls would create schemas if they didn't already exist (#2525, #2528)
• Fix a bug where dbt deps ignored the --project-dir flag (#2519, #2534)
• Fix a bug where parallel RPC requests confounded each others' arguments ([#2484, #2554])
• Fix a bug where dbt ignored config aliases in config() calls and dbt_project.yml (#2557, #2559, #2575)
• Fix unclickable nodes in the dbt Docs DAG viz (#101)
• Fix relation names with null databases (i.e. Spark) in dbt Docs site (#96)

Contributors:

• @bodschut (#2550)

dbt 0.17.1rc4

This is a bugfix release

dbt 0.17.1rc3

This is a bugfix release

dbt 0.17.1rc2

This is a bugfix release

dbt 0.17.1rc1

This is a bugfix release

dbt 0.17.0

dbt 0.17.0 - Octavius Catto (June 8th, 2020)

Links:

• 📖 Migration guide + docs

Breaking changes

• Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version (#2300, #2312)
• The graph object available in some dbt contexts now has an additional member sources (along side the existing nodes). Sources have been removed from nodes and added to sources instead (#2312)
• The 'location' field has been removed from bigquery catalogs (#2382)
• The list_relations_without_caching, drop_schema, and create_schema macros and methods now accept a single argument of a Relation object with no identifier field. (#2411)

Features

Core

• Add --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. (#1649, #2224)
• Support for appending query comments to SQL queries. (#2138, #2199)
• Add a get-manifest API call. (#2168, #2232)
• Support adapter-specific aliases (like project and dataset on BigQuery) in source definitions. (#2133, #2244)
• Users can now use jinja as arguments to tests. Test arguments are rendered in the native context and injected into the test execution context directly. (#2149, #2220)
• Users can supply paths as arguments to --models and --select, either explicitily by prefixing with path: or implicitly with no prefix. (#454, #2258)
• dbt now builds the relation cache for "dbt compile" and "dbt ls" as well as "dbt run" (#1705, #2319)
• Add a 'depends_on' attribute to the log record extra field (#2316, #2341)
• Add a '--no-browser' argument to "dbt docs serve" so you can serve docs in an environment that only has a CLI browser which would otherwise deadlock dbt (#2004, #2364)
• Sources (and therefore freshness tests) can be enabled and disabled via dbt_project.yml (#2283, #2312, #2357)
• schema.yml files are now fully rendered in a context that is aware of vars declared in from dbt_project.yml files (#2269, #2357)
• Sources from dependencies can be overridden in schema.yml files (#2287, #2357)
• Add a filter named as_text to the native environment rendering code that allows users to mark a value as always being a string (#2384, #2395)
• Add warning to nodes selector if nothing was matched (#2115, #2343)

Snowflake

• Snowflake now uses "show terse objects" to build the relations cache instead of selecting from the information schema (#2174, #2322)
• Snowflake now uses "describe table" to get the columns in a relation (#2260, #2324)
• Relation comments supported for Snowflake tables and views. Column comments supported for tables. (#1722, #2321)

BigQuery

• Suport column descriptions for BigQuery models (#2335, #2402)
• Add BigQuery option maximum_bytes_billed to set an upper limit for query costs (#2346, #2427)

Postgres & Redshift

• Added support for db_groups and autocreate flags in Redshift configurations. (#1995, #2262)
• Implement persist_docs for both relation and comments on postgres and redshift, and extract them when getting the catalog. (#2333, #2378)

Docs

• Do not render hidden models in the search bar (docs#89, docs#90)

Fixes

• When a jinja value is undefined, give a helpful error instead of failing with cryptic "cannot pickle ParserMacroCapture" errors (#2110, #2184)
• Added timeout to registry download call (#2195, #2228)
• When a macro is called with invalid arguments, include the calling model in the output (#2073, #2238)
• When a warn exception is not in a jinja do block, return an empty string instead of None (#2222, #2259)
• Add dbt plugin versions to --version(#2272, #2279)
• When a Redshift table is defined as "auto", don't provide diststyle (#2246, #2298)
• Made file names lookups case-insensitve (.sql, .SQL, .yml, .YML) and if .yaml files are found, raise a warning indicating dbt will parse these files in future releases. (#1681, #2263)
• Return error message when profile is empty in profiles.yml. (#2292, #2297)
• Fix skipped node count in stdout at the end of a run (#2095, #2310)
• Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. (#2188, #2325)
• Fix "dbt deps" command so it respects the "--project-dir" arg if specified. (#2338, #2339)
• On run_cli API calls that are passed --vars differing from the server's --vars, the RPC server rebuilds the manifest for that call. (#2265, #2363)
• Remove the query job SQL from bigquery exceptions (#2383, #2393)
• Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures (#2336, #2348)
• No longer query the information_schema.schemata view on bigquery (#2320, #2382)
• Preserve original subdirectory structure in compiled files. (#2173, #2349)
• Add support for sql_header config in incremental models (#2136, #2200)
• The ambiguous alias check now examines the node's database value as well as the schema/identifier ([#2326](#2326...

dbt 0.16.1

dbt 0.16.1 (April 14, 2020)

This is a bugfix release with no breaking changes and one net-new feature.

Features

• Support for appending query comments to SQL queries. (#2138, #2199) (docs)

Fixes

• dbt now renders the dbt_project.yml name config in the "base" context, giving it access to var and env_var (#2230, #2251)
• Fix an issue with {% raw %} blocks where multiple raw blocks in the same file resulted in an error (#2241, #2252)
• Fix a redshift-only issue that caused an error when dbt seed found a seed with an entirely empty column that was set to a varchar data type. (#2250, #2254)
• Fix a bug where dbt plugins that used the default list_schemas and information_schema_name macros with database quoting enabled double-quoted the database name in their queries (#2267, #2281)
• The BigQuery "partitions" config value can now be used in dbt_project.yml (#2256, #2280)
• dbt deps no longer require a profile, but if profile-specific fields are accessed users will get an error (#2231, #2290)
• Macro name collisions between dbt and plugins now raise an appropriate exception, instead of an AttributeError (#2288, #2293)
• The create_adapter_plugin.py script has been updated to support 0.16.X adapters (#2145, #2294)

Under the hood

• Pin google libraries to higher minimum values, add more dependencies as explicit (#2233, #2249)

Contributors:

• @ilkinulas #2199

dbt 0.16.0

dbt 0.16.0 (March 23, 2020)

This release contains many new features, bug fixes, and performance improvements.

Links:
🔍 Upgrading guide
💬 Discussion: v0.16.0 is released
⏩ Discussion: BigQuery incremental model updates
📈 Discussion: Benchmarking BigQuery incremental strategies

Breaking changes

• Agate type inference is no longer locale-specific. Only a small number of date/datetime formats are supported. If a seed has a specified column type, agate will not perform any type inference (it will instead be cast from a string). (#999, #1639, #1920)
• BigQuery range bucket partitioning must now be configured with the dictionary-style partitioning syntax (#2140) (docs)
• generate_schema_name macros that accept a single argument are no longer supported (#2143)
• Files with a .yml extension found in the data/, macros/, analysis/, tests/, and snapshots/ directories will now be parsed as schema.yml specifications (#2160)
• The syntax of the get_catalog macro has changed (#2037)
• When overriding the snowflake__list_schemas macro, you must now run a result with a column named 'name' instead of the first column (#2171)
• dbt no longer supports building models in Snowflake databases with greater than 10,000 schemas (#2171)
• Arguments to source tests are not parsed in the config-rendering context, and are passed as their literal unparsed values to macros (#2150)
• Update the debug log format (#2099)
• Removed docrefs from output (#2096)

Features

Core

• Compilation contexts have updated to be more consistent and well-defined (#1053, #1981, #1255, #2085) (docs)
• Add support for generate_database_name macro (#1695, #2143) (docs)
• Add meta, docs, and tags, and column quoting options to schema.yml files (docs)
  • Add a meta key to most schema.yml specifications (#1362, #2015)
  • Add tags for sources (like model tags) and columns (tags apply to tests of that column) (#1906, #1586, #2039)
  • Add column-level quoting control for tests (#2106, #2047)
  • Add a "docs" field to models, with a "show" subfield (#1671, #2107)
• Add support for documentation of macros, analyses, snapshots and seeds (docs)
  • Add documentation for macros/analyses (#1041, #2068)
  • Add arguments field to macro schema.yml specifications (#2081, #2083, #2096)
  • Add documentation for snapshots and seeds (#1974, #2051)
• Improve the speed of catalog generation by performing multiple smaller queries instead of one huge query (#2009, #2037)
• Add batching to anonymous usage statistics requests to improve performance (#2008, #2089)
• Add search for docs to include 'data' and 'snapshots' folders, in addition to 'models' (#1832, #2058)
• Add search for docs to include macro-paths and analysis-paths (in addition to source-paths, data-paths, and snapshot-paths) (#2155, #2160)
• Add Column.is_number/Column.is_float methods (#1969, #2046)
• Add detection and error handling for duplicated macro names (#1891, #2045)
• Add support for --select on dbt seed (#1711, #2042) (docs)
• Addtoyaml and fromyaml methods to the base context (#1911, #2036) (docs)
• Add database_schemas to the on-run-end context (#1924, #2031) (docs)
• Add the concept of builtins to the dbt context, make it possible to override functions like ref (#1603, #2028) (docs)
• Add clickable docs URL link in CLI output (#2027, #2131)
• Add parsing model hooks and collect ref statements (#1957, #2025)
• Add the macros every node uses to its depends_on.macros list (#2082, #2103)

Docs website

• Add handling non-array accepted_values test arguments (dbt-docs#70)
• Add support for filtering by resource type (dbt-docs#77)
• Add analyses, macros, and custom data tests (dbt-docs#72, dbt-docs#77, dbt-docs#69)
• Add support for hiding models from the docs (these nodes still render in the DAG view as "hidden") (dbt-docs#71)
• Add meta fields as "details" in node views (dbt-docs#73)
• Add lower-casing for Snowflake columns specified in all-caps (dbt-docs#74)
• Upgrade site dependencies

Postgres/Redshift

• Add role parameter in Postgres target configuration (#1955, #2137) (docs)
• Add sslmode parameter for Postgres target configuration (#2152, #2154) (docs)

Snowflake

• Remove the requirement to have a passphrase when using Snowflake key pair authentication (#1805, #2164) (docs)

BigQuery

• Support a cost-effective approach for incremental models on BigQuery using scri...

dbt 0.15.2

dbt 0.15.2 (February 2, 2020)

This is a bugfix release.

Features

• Add support for Snowflake OAuth authentication (#2050, #2069)
• Add a -t flag as an alias for dbt run --target (#1281, #2057)

Fixes

• Fix for UnicodeDecodeError when installing dbt via pip (#1771, #2076)
• Fix for ability to clean "protected" paths in the dbt clean command and improve logging (#2059, #2060)
• Fix for dbt server error when {% docs %} tags are malformed (#2066, #2067)
• Fix for errant duplicate resource errors when models are disabled and partial parsing is enabled (#2055, #2056)
• Fix for errant duplicate resource errors when a resource is included in multiple source paths (#2064, #2065)

Contributors

• @aescay (#2057)
• @markberger (#2076)
• @emilieschario (#2060)

dbt 0.15.1

dbt 0.15.1 (January 17, 2020)

This is a bugfix release.

Features

• Lazily load database connections (#1584, #1992)
• Support raising warnings in user-space (#1970, #1977) (docs)
• Suppport BigQuery label configuration for models (#1942, #1964) (docs)
• Support retrying when BigQuery models fail with server errors (#1579, #1963) (docs)
• Support sql headers in create table/view statements (#1879, #1967) (docs)
• Support source snapshot-freshness command in the dbt server (#2040, #2041)
• Support the -t shorthand for the --target flag on the CLI (#1281, #2057)

Fixes

• Fix for catalog generation error when datasets are missing on BigQuery (#1984, #2005)
• Fix for invalid SQL generated when "check" strategy is used in Snapshots with changing schemas (#1797, #2001)
• Fix for gaps in valid_from and valid_to timestamps when "check" strategy is used in Snapshots on some databases (#1736, #1994)
• Fix incorrect thread names in dbt server logs (#1905, #2002)
• Fix for ignored catalog data when user schemas begin with pg* on Postgres and Redshift (#1960, #2003)
• Fix for poorly defined materialization resolution logic (#1962, #1976)
• Fix missing drop_schema method in adapter namespace (#1980, #1983)
• Fix incorrect generated_at value in the catalog (#1988)

Under the hood

• Fail more gracefully at install time when setuptools is downlevel (#1975, #1978)
• Make the DBT_TEST_ALT integration test warehouse configurable on Snowflake (#1939, #1979)
• Pin upper bound on google-cloud-bigquery dependency to 1.24.0. (#2007)
• Remove duplicate get_context_modules method (#1996)
• Add type annotations to base adapter code (#1982)

Contributors:

• @Fokko (#1996, #1988, #1982)
• @kconvey (#1967)