Merge pull request #16 from brooklyn-data/environment_per_adapter_version

Switch to package-version level environment management

Author: NiallRees

CHANGELOG.md

@@ -5,15 +5,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased](https://github.com/brooklyn-data/dbtenv/compare/v1.3.2...HEAD)
+## [Unreleased](https://github.com/brooklyn-data/dbtenv/compare/v2.0.0a1...HEAD)
 
 ### Added
 
 ### Changed
 
 ### Fixed
 
-## [1.3.2](https://github.com/brooklyn-data/dbtenv/compare/v1.3.1...1.3.2)
+## [2.0.0a1](https://github.com/brooklyn-data/dbtenv/compare/v1.3.2...v2.0.0a1)
+
+### Added
+- dbtenv now operates at the adapter-version level, introduced by dbt in version 1.0.0. The interface is identical to prior versions, dbtenv will automatically detect the needed adapter version from `profiles.yml`, or the `--adapter` argument set in a dbt command passed to `dbtenv --execute`.
+
+### Changed
+- Dropped support for Homebrew.
+- Previously created environments through dbtenv cannot be used, and will be recreated by dbtenv at the adapter-version level.
+- dbtenv's default behaviour is not to install missing dbt adapter versions automatically. It can be disabled by setting the `DBTENV_AUTO_INSTALL` environment variable to `false`.
+
+### Fixed
+
+## [1.3.2](https://github.com/brooklyn-data/dbtenv/compare/v1.3.1...v1.3.2)
 
 ### Added
 

README.md

@@ -1,6 +1,6 @@
 # dbtenv
 
-dbtenv lets you easily install and run multiple versions of [dbt](https://docs.getdbt.com/docs/introduction) using [pip](https://pip.pypa.io/) with [Python virtual environments](https://docs.python.org/3/library/venv.html), or optionally using [Homebrew](https://brew.sh/) on Mac or Linux.
+dbtenv is a version manager for dbt, automatically installing and switching to the needed adapter and version of [dbt](https://docs.getdbt.com/docs/introduction).
 
 
 ## Installation
@@ -13,23 +13,25 @@ dbtenv lets you easily install and run multiple versions of [dbt](https://docs.g
 
 Run `dbtenv --help` to see some overall documentation for dbtenv, including its available sub-commands, and run `dbtenv <sub-command> --help` to see documentation for that sub-command.
 
-### Using pip and/or Homebrew
-By default dbtenv uses [pip](https://pip.pypa.io/) to install dbt versions from the [Python Package Index](https://pypi.org/project/dbt/#history) into Python virtual environments within `~/.dbt/versions`.
-
-However, on Mac or Linux systems dbtenv will automatically detect and use any version-specific dbt installations from [Homebrew](https://brew.sh/) (e.g. `[email protected]` but not plain `dbt`), and you can have dbtenv use Homebrew to install new dbt versions by setting a `DBTENV_DEFAULT_INSTALLER=homebrew` environment variable, or specifying `--installer homebrew` when running `dbtenv install`.
+dbtenv uses [pip](https://pip.pypa.io/) to install dbt versions from the [Python Package Index](https://pypi.org/project/dbt/#history) into Python virtual environments within `~/.dbt/versions`.
 
 ### Installing dbt versions
-You can run `dbtenv versions` to list the versions of dbt available to install, and run `dbtenv install <version>` to install a version.
+You can run `dbtenv versions` to list the versions of dbt available to install, and run `dbtenv install <version>` to install a specific version.
 
-If you don't want to have to run `dbtenv install <version>` manually, you can set a `DBTENV_AUTO_INSTALL=true` environment variable so that as you run commands like `dbtenv version` or `dbtenv execute` any dbt version specified that isn't already installed will be installed automatically.
+dbtenv will automatically install the required version of dbt for the current project by default. To disable this behaviour, set the environment variable `DBTENV_AUTO_INSTALL` to `false`.
 
-Some tips when dbtenv is using pip:
-- You can customize where the dbt version-specific Python virtual environments are created by setting `DBTENV_VENVS_DIRECTORY` and `DBTENV_VENVS_PREFIX` environment variables.
+Some tips:
+- You can customize where the dbt package-version-specific Python virtual environments are created by setting the `DBTENV_VENVS_DIRECTORY` environment variable.
 - You can have dbtenv only install Python packages that were actually available on the date the dbt version was released by setting a `DBTENV_SIMULATE_RELEASE_DATE=true` environment variable, or specifying `--simulate-release-date` when running `dbtenv install`.
   This can help if newer versions of dbt's dependencies are causing installation problems.
 - By default dbtenv uses whichever Python version it was installed with to install dbt, but that can be changed by setting a `DBTENV_PYTHON` environment variable to the path of a different Python executable, or specifying `--python <path>` when running `dbtenv install`.
 
 ### Switching between dbt versions
+#### Adapter type
+If a dbtenv command is invoked from within a dbt project, dbtenv will try to determine the in-use adapter from the default set for the project's profile in `profiles.yml`. If the `--adapter` argument is set in the dbt command passed to `dbtenv execute`, dbtenv will use that adapter's type instead.
+
+#### dbt version
+
 dbtenv determines which dbt version to use by trying to read it from the following sources, in this order, using the first one it finds:
 
 1. The `dbtenv execute` command's optional `--dbt <version>` argument.
@@ -48,7 +50,7 @@ You can:
 - Run `dbtenv version --global <version>` to set the dbt version globally in the `~/.dbt/version` file.
 - Run `dbtenv version --local <version>` to set the dbt version for the current directory in a `.dbt_version` file.
 
-### Running dbt versions
+### Running dbt through dbtenv
 Run `dbtenv execute -- <dbt arguments>` to execute the dbt version determined dynamically based on the current environment, or run `dbtenv execute --dbt <version> -- <dbt arguments>` to execute the specified dbt version.
 
 For example:

dbtenv/__init__.py

@@ -15,7 +15,6 @@
 
 
 DEFAULT_VENVS_DIRECTORY = os.path.normpath('~/.dbt/versions')
-DEFAULT_VENVS_PREFIX    = ''
 
 GLOBAL_VERSION_FILE = os.path.normpath('~/.dbt/version')
 LOCAL_VERSION_FILE  = '.dbt_version'
@@ -28,7 +27,6 @@
 QUIET_VAR                 = 'DBTENV_QUIET'
 SIMULATE_RELEASE_DATE_VAR = 'DBTENV_SIMULATE_RELEASE_DATE'
 VENVS_DIRECTORY_VAR       = 'DBTENV_VENVS_DIRECTORY'
-VENVS_PREFIX_VAR          = 'DBTENV_VENVS_PREFIX'
 
 
 def string_is_true(value: str) -> bool:
@@ -66,71 +64,71 @@ def get(self, name: str, default: Optional[Any] = None) -> Optional[Any]:
 
 class Installer(Enum):
     PIP      = 'pip'
-    HOMEBREW = 'homebrew'
 
     def __str__(self) -> str:
         return self.value
 
 
 class Version(distutils.version.LooseVersion):
-    def __init__(self, version: str, source: Optional[str] = None, source_description: Optional[str] = None) -> None:
-        self.pypi_version = self.homebrew_version = self.raw_version = version.strip()
-        self.source = source
-        if source and not source_description:
-            self.source_description = f"set by {source}"
+    def __init__(self, pip_specifier: str = None, adapter_type: str = None, version: str = None, source: Optional[str] = None, source_description: Optional[str] = None) -> None:
+        if pip_specifier:
+            self.pip_specifier = pip_specifier
+            self.name, self.version = re.match(r"(.*)==(.*)", pip_specifier).groups()
+            self.pypi_version = self.version
         else:
-            self.source_description = source_description
+            self.name = f"dbt-{adapter_type}"
+            self.pypi_version = version
+            self.pip_specifier = f"{self.name}=={self.pypi_version}"
+        if not self.name.startswith('dbt'):
+            raise(Exception)
+        self.source_description = source_description
+        self.source = source
 
-        version_match = re.match(r'(?P<version>\d+\.\d+\.\d+)(-?(?P<prerelease>[a-z].*))?', self.raw_version)
+        version_match = re.match(r'(?P<version>\d+\.\d+\.\d+)(-?(?P<prerelease>[a-z].*))?', self.pypi_version)
         self.is_semantic = version_match is not None
         self.is_stable = version_match is not None and not version_match['prerelease']
         self.major_minor_patch = version_match['version'] if version_match is not None else None
         self.prerelease = version_match['prerelease'] if version_match is not None else None
 
-        # dbt pre-release versions are formatted slightly differently in PyPI and Homebrew.
+        # dbt pre-release versions are formatted slightly differently.
         if version_match and version_match['prerelease']:
             self.pypi_version     = f"{version_match['version']}{version_match['prerelease']}"
-            self.homebrew_version = f"{version_match['version']}-{version_match['prerelease']}"
 
-        # Standardize on the PyPI version for comparison and hashing.
         super().__init__(self.pypi_version)
 
     def __hash__(self) -> int:
         return self.pypi_version.__hash__()
 
     def __str__(self) -> str:
-        return self.raw_version
+        return self.pip_specifier
 
     def __repr__(self) -> str:
-        return f"Version('{self.raw_version}')"
+        return f"Version('{self.pip_specifier}')"
 
-    def _cmp(self, other: Any) -> int:
-        # Comparing standard integer-based versions to non-standard text versions will raise a TypeError.
-        # In such cases we'll fall back to comparing the entire version strings rather than the individual parts.
-        try:
-            return super()._cmp(other)
-        except:
-            if isinstance(other, str):
-                return self._str_cmp(other)
-            if isinstance(other, Version):
-                return self._str_cmp(other.pypi_version)
-            raise
-
-    def _str_cmp(self, other: str) -> int:
-        if self.pypi_version == other:
+    def _cmp(self, other: 'Position') -> int:
+        if self.name < other.name:
+            return -1
+        if self.name > other.name:
+            return 1
+        if self.pypi_version == other.pypi_version:
             return 0
-        if self.pypi_version < other:
+        if self.pypi_version < other.pypi_version:
             return -1
-        if self.pypi_version > other:
+        if self.pypi_version > other.pypi_version:
             return 1
 
+    @property
+    def source(self):
+        return self._source
+
+    @source.setter
+    def source(self, source):
+        if source and not self.source_description:
+            self.source_description = f"set by {source}"
+        self._source = source
+
     def get_installer_version(self, installer: Installer) -> str:
-        if installer == Installer.PIP:
-            return self.pypi_version
-        elif installer == Installer.HOMEBREW:
-            return self.homebrew_version
-        else:
-            return self.raw_version
+        return self.pypi_version
 
 
 class Environment:
@@ -147,20 +145,9 @@ def __init__(self) -> None:
         self.project_directory = os.path.dirname(self.project_file) if self.project_file else None
 
         self.venvs_directory = os.path.expanduser(self.env_vars.get(VENVS_DIRECTORY_VAR) or DEFAULT_VENVS_DIRECTORY)
-        self.venvs_prefix = self.env_vars.get(VENVS_PREFIX_VAR) or DEFAULT_VENVS_PREFIX
 
         self.global_version_file = os.path.expanduser(GLOBAL_VERSION_FILE)
 
-        self.homebrew_installed = False
-        self.homebrew_prefix_directory = self.env_vars.get('HOMEBREW_PREFIX')
-        if not self.homebrew_prefix_directory and self.os != 'Windows':
-            brew_executable = shutil.which('brew')
-            if brew_executable:
-                self.homebrew_prefix_directory = os.path.dirname(os.path.dirname(brew_executable))
-        if self.homebrew_prefix_directory and os.path.isdir(self.homebrew_prefix_directory):
-            self.homebrew_installed = True
-            logger.debug(f"Homebrew is installed with prefix `{self.homebrew_prefix_directory}`.")
-
     _debug: Optional[bool] = None
 
     @property
@@ -207,11 +194,7 @@ def update_logging_level(self) -> None:
 
     @property
     def default_installer(self) -> Installer:
-        if self._default_installer is None:
-            if DEFAULT_INSTALLER_VAR in self.env_vars:
-                self._default_installer = Installer(self.env_vars[DEFAULT_INSTALLER_VAR].lower())
-            else:
-                self._default_installer = Installer.PIP
+        self._default_installer = Installer.PIP
 
         return self._default_installer
 
@@ -233,10 +216,6 @@ def primary_installer(self) -> Installer:
     def use_pip(self) -> bool:
         return not self.installer or self.installer == Installer.PIP
 
-    @property
-    def use_homebrew(self) -> bool:
-        return (not self.installer or self.installer == Installer.HOMEBREW) and self.homebrew_installed
-
     _python: Optional[str] = None
 
     @property
@@ -271,7 +250,7 @@ def auto_install(self) -> bool:
             if AUTO_INSTALL_VAR in self.env_vars:
                 self._auto_install = string_is_true(self.env_vars[AUTO_INSTALL_VAR])
             else:
-                self._auto_install = False
+                self._auto_install = True
 
         return self._auto_install
 

dbtenv/execute.py

@@ -1,5 +1,6 @@
 # Standard library
 import argparse
+import re
 from typing import List
 
 # Local
@@ -34,10 +35,10 @@ def add_args_parser(self, subparsers: argparse._SubParsersAction, parent_parsers
         parser.add_argument(
             '--dbt',
             dest='dbt_version',
-            type=Version,
+            type=str,
             metavar='<dbt_version>',
             help="""
-                Exact version of dbt to execute.
+                dbt version to use (e.g. 1.0.1).
                 If not specified, the dbt version will be automatically detected from the environment.
             """
         )
@@ -53,11 +54,22 @@ def add_args_parser(self, subparsers: argparse._SubParsersAction, parent_parsers
         )
 
     def execute(self, args: Args) -> None:
+        arg_target_name = None
+        for i, arg in enumerate(args.dbt_args):
+            if arg == "--target":
+                arg_target_name = args.dbt_args[i+1]
+                break
+        adapter_type = dbtenv.version.try_get_project_adapter_type(self.env.project_file, target_name=arg_target_name)
+        if not adapter_type:
+            logger.info("Could not determine adapter, either not running inside dbt project or no default target is set for the current project in profiles.yml.")
+            return
+
         if args.dbt_version:
-            version = args.dbt_version
+            version = Version(adapter_type=adapter_type, version=args.dbt_version)
         else:
-            version = dbtenv.version.get_version(self.env)
-            logger.info(f"Using dbt {version} ({version.source_description}).")
+            arg_target_name = None
+            version = dbtenv.version.get_version(self.env, adapter_type=adapter_type)
+            logger.info(f"Using {version} ({version.source_description}).")
 
         dbt = dbtenv.which.try_get_dbt(self.env, version)
         if not dbt: