Refactor dbt utility for modularity

[austinweisgrau]

These changes uncouple the code for the dbt utility that runs a dbt command from the code that logs dbt commands. This allows for modular use of different loggers and a very flexible ability to customize logging. This addresses the main concerns from the original PR (#841).

Breaking changes

This PR does cause breaking changes with the previous implementation of the dbt utility, but those changes are well worth it for a much cleaner and clearer interface.

Using the new implementation looks like:

import os
from pathlib import Path
from parsons.utilities.dbt import run_dbt_commands, dbtLoggerSlack, dbtLoggerPython

results = run_dbt_commands(
    commands=["dbt run", "dbt test"],
    dbt_project_directory=Path("/path/to/dbt/project"),
    loggers=[dbtLoggerPython, dbtLoggerSlack(webhook=os.environ["SLACK_WEBHOOK"])],
)

The prior implementation was potentially only compatible with Redshift, depending on how the user's dbt profiles.yaml was set up. The new implementation allows the dbt process to inherit the shell environment from the python parent process, and is therefore compatible with running dbt on any database, and the user can configure credential passing using environment variables (recommended) or any other method.

Improved documentation

There is now much more thorough documentation throughout the 4 modules containing the dbt utility code.

TO DO:

• Touch up the dbtLoggerMarkdown text formatting a bit
• dbtLoggerDatabase splits artifacts into a run table and a node table

[elyse-weiss]

I do not feel qualified to approve, but having the requirements.txt file updated as it is here, would be helpful for me separately!

[austinweisgrau]

The build step is failing due to an unrelated dependency conflict, I think it's only triggering on this PR and not others because this one updates the docs folder

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
parsons/utilities/dbt
__init__.py					46-54
dbt.py					3-122
logging.py					3-290
models.py					3-90
Project Total

This report was generated by python-coverage-comment-action

[austinweisgrau]

Finally fixed the build issue

[shaunagm]

These docs are good, but ideally we want to be able to surface them on the docs website. Maybe add a hook into the autodocs on this page? https://move-coop.github.io/parsons/html/stable/utilities.html#

[austinweisgrau]

I thiiink 8d56b8a should do this

[shaunagm]

A couple minor requests/questions but overall looks good. I know @matthewkrausse is looking forward to seeing this merged so hope to get it over the finish line soon and promise not to be a blocker on my end :)

[shaunagm]

Where is raise_error set? Where is it referenced? Is this an underlying dbt thing or is it part of this method/class?

[austinweisgrau]

Good catch, that was a line of obsolete documentation, removed with 10359ba