Test packages against their deployment environment

[mikealfare]

Problem

We currently test packages against main for all dbt- managed packages in all scenarios, whether it's scheduled tests, PR checks, or publishing checks. Packages in this repository are installed locally, and packages like dbt-common and dbt-core are installed from main from their repositories. This makes sense for scheduled tests and potentially for PR checks. However it leaves a gap when deploying. It is too easy to introduce a dependency across packages and not realize it. This leads to a scenario where the latest versions on PyPI are broken until all of the packages are released. Even if the gap is only a day, this could break Core users who are always on latest. We should be aware of when we do this, and if possible, prevent it from happening.

Solution

There are three types of dependencies:

• Packages in this repo (e.g. dbt-adapters, dbt-postgres)
• Packages in this org, but not this repo (e.g. dbt-core, dbt-common)
• Packages on PyPI

There are five scenarios:

• Local development
  • Install repo packages from the current branch
  • Install org packages from main
    • This can be overwritten to a feature branch for cross-package features
• Scheduled testing
  • Install repo packages from main
  • Install org packages from main
• Pull request checks
  • Install repo packages from the current branch
  • Install org packages from main
    • This should not be overwritten to prevent merging a broken state
• Internal publishing checks
  • Install repo packages from main
    • This mimics installing from internal PyPI since we bulk publish all packages at the same time
  • Install org packages from main
• PyPI publishing checks
  • Install repo packages from PyPI
  • Install org packages from PyPI

Open questions:

• Should we add the PyPI environment to scheduled testing so that we know ahead of time that we need to publish in a certain order
• Should we add the PyPI environment to pull request checks as a non-blocking check so that we know that we'll need to update the pin on one or more org packages
• Should we only

To accomplish this, I added two new hatch environments, ci and cd, to each package to install the proper dependencies for testing in these slightly different scenarios. This also allowed the dependencies to shrink a bit in each environment. ddtrace is not needed locally while pytest-dotenv and pre-commit are not needed in GHA.