dbt 0.12.1

Overview

This is a bugfix release.

Fixes

• Fix for relation caching when views outside of a dbt schema depend on relations inside of a dbt schema (#1119)

dbt 0.12.0

dbt 0.12.0 - Guion Bluford (November 12, 2018)

Overview

This release adds caching for some introspective queries on all adapters. Additionally, custom tags can be supplied for models, along with many other minor improvements and bugfixes.

Breaking Changes

• Support for the repositories: block in dbt_project.yml (deprecated in 0.10.0) was removed. Use the packages.yml file instead (docs)

tl;dr

• Make runs faster by caching introspective queries
• Support model tags
• Add a list of schemas to the on-run-end context
• Set your profiles directory with an environment variable

Features

• Cache the existence of relations to speed up dbt runs (#1025)
• Add support for tag configuration and selection (#1014)
  • Add tags to the model and graph views in the docs UI (#7)
• Add the set of schemas that dbt built models into in the on-run-end hook context (#908)
• Warn for unused resource config paths in dbt_project.yml (#725)
• Add more information to the dbt --help output (#1058)
• Add support for configuring the profiles directory with an env var (#1055)
• Add support for cli and env vars in most dbt_project.yml and profiles.yml fields (#1033)
• Provide a better error message when seed file loading fails on BigQuery (#1079)
• Improved error handling and messaging on Redshift (#997)
• Include datasets with underscores when listing BigQuery datasets (#954)
• Forgo validating the user's profile for dbt deps and dbt clean commands (#947, #1022)
• Don't read/parse CSV files outside of the dbt seed command (#1046)

Fixes

• Fix for incorrect model selection with the --models CLI flag when projects and directories share the same name (#1023)
• Fix for table clustering configuration with multiple columns on BigQuery (#1013)
• Fix for incorrect output when a single row fails validation in dbt test (#1040)
• Fix for unwieldly Jinja errors regarding undefined variables at parse time (#1086, #1080, #935)
• Fix for incremental models that have a line comment on the last line of the file (#1018)
• Fix for error messages when ephemeral models fail to compile (#1053)

Under the hood

• Create adapters as singleton objects instead of classes (#961)
• Combine project and profile into a single, coherent object (#973)
• Investigate approaches for providing more complete compilation output (#588)

Contributors

Thanks for contributing!

• @mikekaminsky (#1049, #1060)
• @joshtemple (#1079)
• @k4y3ff (#954)
• @elexisvenator (#1019)
• @clrcrl (#725)

Guion Bluford

Dr. Guion “Guy” Stewart Bluford Jr. is a true Philadelphian, through and through. He was born in Philadelphia, PA November 22nd, 1942 and attended Overbrook High School. He grew up in a home that encouraged academic success, which inspired him to eventually receive his Ph. D in Aerospace Engineering from the Air Force Institute of Technology in 1978.

Not only an academic, Bluford flew 144 combat missions during the Vietnam War, and earned numerous medals in the process. He became the first African American in space in 1983 as a specialist for mission STS-8 aboard the space shuttle Challenger. He was inducted into the International Space Hall of Fame in 1997 and the United States Astronaut Hall of Fame in 2010.

In addition to his professional success, Bluford has been seen as a role model for many African Americans. He has been awarded the NCAAP Image Award, Ebony Black Achievement Award and the 1991 Black Engineer of the Year award, to name a few.

A family man, a war veteran, an astronaut and a role model; Guion Bluford.

dbt 0.11.1

dbt 0.11.1 - Lucretia Mott (September 18, 2018)

This is a patch release containing a few bugfixes and quality of life changes, especially around dbt docs.

Features

• dbt
  • Add --port parameter to dbt docs serve (#987)
  • Add support for BigQuery clustering (#978)

Fixes

• dbt
  • Fix hooks in model configs not running (#985)
  • Fix integration test on redshift catalog generation (#977)
  • Snowflake: Fix docs generation errors when QUOTED_IDENTIFIER_IGNORE_CASE is set (#998)
  • Translate empty strings to null in seeds (#995)
  • Filter out null schemas during catalog generation (#992)
  • Fix quoting on drop, truncate, and rename (#991)
• dbt-docs
  • Fix for non-existent column in schema.yml (#3)
  • Fixes for missing tests in docs UI when columns are upcased (#2)
  • Fix "copy to clipboard" (#4)

Thanks to @ericalouie for the biography of Lucretia Mott, below.

Lucretia Mott

Lucretia Mott (1793-1880) was a women’s rights activist, abolitionist, and reformer. As a powerful orator, she dedicated her life to achieving equality for all of America’s disadvantaged and disenfranchised, including Indians, women, slaves, and free blacks. During an era where women were to be seen and never heard, Mott’s public speaking certainly defied that. And while we now have the luxury to admire that, her behavior was often denounced as controversial and inappropriate. Nevertheless, she persisted.

To pinpoint Mott’s accomplishments into a single entity is almost impossible as her influence expanded to so many areas: speaking for women’s rights at the Seneca Falls Convention, authoring Discourse on Woman, co-founding the Philadelphia Female Anti-Slavery Society.

Originally born in Nantucket, Massachusetts, she was raised as a Quaker. And although her impact on the feminist discourse and abolitionist movement spread throughout America and American history, her legacy is rooted in Philadelphia. Having moved to Philadelphia at 16, she certainly built her roots here. She married James Mott, also an abolitionist and social reformer, at Front and 2nd in Old City. Together, they fought for the same causes, housed several runaway slaves, and raised 5 children who all became active in the anti-slavery movement. In 1869 Mott and her husband were active in the plans for the opening of Swarthmore College, a Quaker institute for higher learning. When the college had been chartered in 1864, she and James had insisted it be coeducational.

Because she had a strong presence as a fighter against gender and racial injustices, mobs would often target her home in Philadelphia. Mott was often seen waiting in her parlor, willing to face her violent opponents. And although she didn’t live to see the day women won the right to vote, Mott is credited with igniting the women’s rights movement and serving as mentor to Elizabeth Cady Stanton, who continued Mott’s work after she died.

dbt 0.11.0

dbt v0.11.0, Isaac Asimov - tl;dr

This release introduces auto-generated dbt documentation, adds a new syntax for schema.yml files, and fixes a number of minor bugs. With the exception of planned changes to Snowflake's default quoting strategy, this release should not contain any breaking changes. Check out the release blog post for more information about this release.

Breaking Changes

• Change default Snowflake quoting strategy to "unquoted" (docs)

Features

• Add autogenerated dbt project documentation (docs)
• Version 2 of schema.yml, which allows users to create table and column comments that end up in the manifest (docs)
• Remove distinction between this.table and this.schema by refactoring materialization SQL

For a full list of changes, check out the changelog.

Resources for upgrading to 0.11:

• schema.yml upgrade script
• migration guide

Contributors

Thanks for contributing!

• @rsmichaeldunn (#799)
• @lewish (#915)
• @MartinLue (#872)

Isaac Asimov

Isaac Asimov (1920-1992) was a prolific writer of science fiction. He is best known for his "Foundation" series, although he is credited with writing and editing thousands of works of both fiction and nonfiction. Though he was born in New York, he lived in West Philadelphia from 1942 to 1949 with his wife, Gertrude. During this time he wrote a handful of short stories which would later become "Foundation," the first novel in the famous series.

In "Foundation," the mathematician Hari Seldon uses the advanced science of psychohistory to predict the future of human civilization. Concerned by what he foresees, Seldon is allowed to gather the most gifted scientific minds in the galaxy onto a remote planet to create the Encyclopedia Galactica, a compendium of all human knowledge, in the hopes that this knowledge will be preserved for tens of thousands of years and prevent the future downfall of human civilization. The Encyclopedists spend hundreds of years compiling the Encyclopedia, and in the process, they create a rich, new civilization with scientific prowess at the forefront.

Inspired by Hari Seldon's effort, we hope that dbt 0.11 helps you capture and share the important knowledge in your data warehouse.

dbt 0.10.2

This release makes it possible to alias relation names, rounds out support for BigQuery with incremental, archival, and hook support, adds the IAM Auth method for Redshift, and builds the foundation for autogenerated dbt project documentation, to come in the next release.

Additionally, a number of bugs have been fixed including intermittent BigQuery 404 errors, Redshift "table dropped by concurrent query" errors, and a probable fix for Redshift connection timeout issues. For a complete list of changes, check out the Changelog.

tl;dr

• BigQuery
  • Support incremental models (#856) (docs)
  • Support archival (#856) (docs)
  • Add pre/post hook support (#836) (docs)
• Redshift: IAM Auth (#818) (docs)
• Model aliases (#800)(docs)

Contributors

We want to extend a big thank you to our outside contributors for this release! You all are amazing.

• @danielchalef (#818)
• @mjumbewu (#796)
• @abelsonlive (#800)
• @jon-rtr (#800)
• @mturzanska (#797)
• @cpdean (#780)

A word on future releases

Going forwards, we'll be naming planned released after famous Philadelphians. We'll include a brief biography of these historical figures in the dbt release notes. This is our little way of exporting Philly culture across the world. You can read a short bio of Betsy Ross, courtesy of @jtcohen6, below 🔔

Betsy Ross

Betsy Ross (1752-1836) was an American seamstress and upholsterer famously associated with three legends: having sewn the ‘first’ American flag for General George Washington in 1776; having kept Hessian Colonel Carl von Dunup from fighting in the Battle of Trenton, the one for which Washington crossed the Delaware; and having lived in a still-standing, built-in-1740 house near 2nd & Arch that remains one of Philadelphia’s most visited tourist attractions.

The first-flag story dates to the 1876 American centennial (celebrated where else but in Philly), when a grandson invented details of Ross’s fateful, five-pointed-star meeting with Washington—though not out of whole cloth, since Ross did sew naval ensigns and later designs. Who knows about von Dunup. As far as 239 Arch, she most likely lived right next door.

We would be wrong, however, to dismiss mythologies as mere falsehoods. Though Ross’s stature in the pantheon of American revolutionaries is somewhat a fabrication, her place in Philadelphia was very real. She grew up in a strict, Quaker household, eloped with a fellow apprentice (and nephew of a signer of the Declaration of Independence), and they attended Christ Church in Old City. She married three times, her first two husbands both casualties of the revolution against Britain. She lived in the heart of the capital of the brand-new United States until, as all good Philadelphians must, she moved to be with her children in Montgomery County.

The Flag Resolution of 1777, officially replacing the Continental Colors / Grand Union Flag with the Stars and Stripes, is one sentence and 31 words long. Ross’s story wraps Old Glory in a more compelling narrative of it-takes-a-city ingenuity. The historical Betsy Ross was a significant Philadelphian whose life spanned several transformative decades; the mythical Ross helped stitch together the tapestry of a nation.

dbt 0.10.1

This release focuses on achieving functional parity between all of dbt's adapters. With this release, most dbt functionality should work on every adapter except where noted here.

tl;dr

• Configure model schema and name quoting in your dbt_project.yml file (Docs)
• Add a Relation object to the context to simplify model quoting Docs
• Implement BigQuery materializations using new create table as (...) syntax, support partition by clause (Docs)
• Override seed column types (Docs)
• Add get_columns_in_table context function for BigQuery (Docs)

Changes

• Consistent schema and identifier quoting (#727)
  • Configure quoting settings in the dbt_project.yml file (#742)
  • Add a Relation object to the context to make quoting consistent and simple (#742)
• Use the new create table as (...) syntax on BigQuery (#717)
  • Support partition by clause
• CSV Updates:
  • Use floating point as default seed column type to avoid issues with type inference (#694)
  • Provide a mechanism for overriding seed column types in the dbt_project.yml file (#708)
  • Fix seeding for files with more than 16k rows on Snowflake (#694)
  • Implement seeds using a materialization
• Improve get_columns_in_table context function (#709)
  • Support numeric types on Redshift, Postgres
  • Support BigQuery (including nested columns in struct types)
  • Support cross-database information_schema queries for Snowflake
  • Retain column ordinal positions

Bugfixes

• Fix for incorrect var precendence when using --vars on the CLI (#739)
• Fix for closed connections in on-run-end hooks for long-running dbt invocations (#693)
• Fix: don't try to run empty hooks (#620, #693)
• Fix: Prevent seed data from being serialized into graph.gpickle file (#720)
• Fix: Disallow seed and model files with the same name (#737)

dbt 0.10.0

dbt 0.10.0

This release overhauls dbt's package management functionality, makes seeding csv files work across all adapters, and adds date partitioning support for BigQuery.

Upgrading Instructions:

• Check out full installation and upgrading instructions here
  • homebrew: brew upgrade dbt
  • pip: pip install -U dbt
• Transition the repositories: section of your dbt_project.yml file to a packages.yml file as described here
• You may need to clear out your dbt_modules directory if you use packages like dbt-utils. Depending how your project is configured, you can do this by running dbt clean.
• We're using a new CSV parsing library, agate, so be sure to check that all of your seed tables are parsed as you would expect!

Changes

• Support for variables defined on the CLI with --vars (#640) (docs)
• Improvements to dbt seed (docs)
  • Support seeding csv files on all adapters (#618)
  • Make seed csv's ref()-able in models (#668)
  • Support seed file configuration (custom schemas, enabled / disabled) in the dbt_project.yml file (#561)
  • Support --full-refresh instead of --drop-existing (deprecated) for seed files (#515)
  • Add --show argument to dbt seed to display a sample of data in the CLI (#74)
• Improvements to package management (docs)
  • Deprecated repositories: config option in favor of packages: (#542)
  • Deprecated package listing in dbt_project.yml in favor of packages.yml (#681)
  • Support stating local file paths as dependencies (#542)
• Support date partitioning in BigQuery (#641) (docs)
• Move schema creation to after on-run-start hooks (#652)
• Replace csvkit dependency with agate (#598)
• Switch snowplow endpoint to pipe directly to Fishtown Analytics (#682)

Bugfixes

• Throw a compilation exception if a required test macro is not present in the context (#655)
• Make the adapter_macro use the return() function (#635)
• Fix bug for introspective query on late binding views (redshift) (#647)
• Disable any non-dbt log output on the CLI (#663)

Thanks!

Special thanks to @b-ryan for his many contributions to dbt v0.10.0, and to @RealSelf for sponsoring our work on BigQuery date partitioning. Is there a feature on our roadmap that you’d like to sponsor? Let us know in Slack!

dbt 0.9.1

dbt 0.9.1

This release fixes some bugs and adds supports for late binding views on Redshift.

Changes:

• support late binding views (docs)
• make run_started_at timezone-aware (Contributed by @mturzanska) (docs)

Bugfixes:

• Include hook run time in reported model run time (#607)
• Add warning for missing test constraints (#600)
• Fix for schema tests used or defined in packages (#599)
• Run hooks in defined order (#601)
• Skip tests that depend on nonexistent models (#617)
• Fix for adapter_macro called within a package (#630)

And one more thing

Questions? Comments? Suggestions? We want to hear them! If you haven't already, join our vibrant and bustling slack team 📈

dbt 0.9.0

dbt 0.9.0

This release focuses on improvements to macros, materializations, and package management. Check out the blog post to learn more about what's possible in this new version of dbt.

Installation

Full installation instructions for macOS, Windows, and Linux can be found here. If you use Windows or Linux, installation works the same as with previous versions of dbt. If you use macOS and Homebrew to install dbt, note that installation instructions have changed:

macOS Installation Instructions

brew update
brew tap fishtown-analytics/dbt
brew install dbt

Overview

• More powerful macros and materializations
• Custom model schemas
• BigQuery improvements
• Bugfixes
• Documentation (0.9.0 docs can be found here)

Breaking Changes

• adapter functions must be namespaced to the adapter context variable. To fix this error, use adapter.already_exists instead of just already_exists, or similar for other adapter functions.

Bugfixes

• Handle lingering __dbt_tmp relations (#511)
• Run tests defined in an ephemeral directory (#509)

Changes

• use adapter, ref, and var inside of macros (#466)
• Build custom tests and materializations in dbt packages (#466)
• Support pre- and post- hooks that run outside of a transaction (#510)
• Support table materializations for BigQuery (#507)
• Support querying external data sources in BigQuery (#507)
• Override which schema models are materialized in (#522) (docs)
• Make {{ ref(...) }} return the same type of object as {{ this }} (#530)
• Replace schema test CTEs with subqueries to speed them up for Postgres (#536) (@ronnyli)
• Bump Snowflake dependency, remove pyasn1 (#570)

Documentation

• Document how to create a package
• Document how to make a materialization
• Document how to make custom schema tests
• Document how to use hooks to vacuum
• Document all context variables

New Contributors

• @ronnyli (#536)

And one more thing

Questions? Comments? Suggestions? We want to hear them! If you haven't already, join our vibrant and bustling slack team 📈

dbt 0.9.0 alpha 2

dbt 0.9.0 alpha 2

This is the second alpha version of the dbt 0.9.0 release. It may contain bugs or incompatibilities with previous versions of dbt. Please reach out to us on Slack or create an issue if you have any questions or find any such bugs.

Over the next few weeks, we (and hopefully you) will put dbt through its paces, fixing bugs or tweaking features as required. When we're happy with the results, we will release dbt v0.9.0 through the regular channels including pypi, homebrew, and conda. If you're interested in using the alpha preview in the meantime, you can do so using the installation instructions below.

Changelog

Check out the changelog here

Installation

Since this is an alpha release, package managers like homebrew, pip and conda will still point to the last production release, dbt v0.8.3. To install dbt v0.9.0a2, you must either install this version explicitly from pip, or install from source.

Installing from pip

Note: we recommend that you install dbt into a virtualenv. While this isn't necessary, it may save you a headache if you choose to roll back to 0.8.3, or when the time comes to upgrade to the final 0.9.0 release.

1. Uninstall any existing versions of dbt brew uninstall dbt, pip uninstall dbt, etc
2. Run pip install dbt==0.9.0a2

Installing from source

git clone [email protected]:fishtown-analytics/dbt.git
cd dbt
git fetch --tags
git checkout tags/v0.9.0a2
pip install -e .

Verify your installation

Regardless of the method you've chosen, run:

dbt --version

to confirm that your installed version of dbt is 0.9.0a2

One more thing

As always, feel free to reach out to us on Slack with any questions or comments!