v0.7.2

• Compatibility fix for Databricks SDK 0.51 (#158). The minimum required version of the Databricks SDK has been updated to 0.51.0 to accommodate breaking changes introduced in this version, including the removal of certain enumeration values. Notably, the ServedModelInputWorkloadSize enumeration has been replaced with well-known string constants, prompting an update to the create function to use the string constant Small instead of the removed ServedModelInputWorkloadSize.SMALL value. This change affects the creation of serving endpoints, which now specify workload size using a string value, and resolves issues reported in related projects, including test failures in downstream projects, by ensuring compatibility with the updated SDK version.

Contributors: @asnare

v0.7.1

• Set upper SDK limit to v1 (#109). In this change, the upper SDK limit is set to version 1.0 in the pyproject.toml file of a library project, allowing projects using this library to set their own appropriate SDK limits. The databricks-sdk dependency is updated from "<0.42" to "<1.0" as a result of this commit. Additionally, the create function in the ml.py file of the pytester library has been modified. Previously, the EndpointCoreConfigInput object was passed as an argument to the serving_endpoints.create method using the keyword argument served_models. Now, EndpointCoreConfigInput is passed as a keyword argument config, which contains the served_models parameter, aligning with the new SDK version 1.0 and ensuring consistent endpoint creation. These changes are intended to ensure compatibility with a variety of SDK versions and improve the library's usability for software engineers.

Contributors: @JCZuurmond

v0.7.0

• Let make_serving_endpoint reference a valid model version (#106). In this release, the make_serving_endpoint fixture has been updated to enhance its functionality and flexibility. The fixture now accepts optional keyword arguments for specifying the endpoint name, model name, and model version, with the model version defaulting to 1 and the latest version automatically retrieved for workspace local models. The served model input is created with the specified model name, model version, and workload size, and a mock endpoint object is returned if obtaining the served model fails. The remove method has been updated to accept a ServingEndpointDetailed object for deletion, and the fixture now maintains a logger instance for warning messages during model version retrieval. Unit tests have been added to ensure the changes work as expected, although integration tests have not been fixed yet. These updates address issues arising from changes in the make_model fixture and improve the robustness and flexibility of the make_serving_endpoint fixture.

Contributors: @mwojtyczka, @JCZuurmond

v0.6.0

What's Changed

• Added serverless support to spark fixture by @mwojtyczka in #91
• Release v0.6.0 by @mwojtyczka in #96
• Fix release pipeline to use known runners by @mwojtyczka in #97

Full Changelog: v0.5.0...v0.6.0

v0.5.0

• Added make_run_as fixture (#82). A new pytest fixture, make_run_as, has been added to create an account service principal via the acc fixture and assign it to a workspace with default permissions, which is removed after the test is complete. The fixture creates a service principal with a random display name and assigns it to the workspace. Users can optionally assign the service principal to account groups for specific actions using the account_groups argument. The returned object contains properties for the workspace client, SQL backend, and SQL execution functions, as well as the display name and application ID of the ephemeral service principal. If desired, the ws fixture can be overridden to make all workspace fixtures provided by the plugin run as the ephemeral service principal, allowing for testing with lower privilege ephemeral service principals and improving security and isolation. This feature is not currently supported with Databricks Metadata Service authentication on Azure Databricks.
• Bump codecov/codecov-action from 4 to 5 (#85). In this release, the Codecov GitHub Action has been updated from version 4 to 5, introducing several new features and changes. The new version uses the Codecov Wrapper to encapsulate the CLI, allowing for quicker updates to the Action. Additionally, version 5 includes an opt-out feature for tokens in public repositories, enabling contributors and other members to upload coverage reports without requiring access to the Codecov token. This can be accomplished by setting the ability for Codecov to receive a coverage report from any source in the Global Upload Token section of the settings page on codecov.io. Furthermore, the updated version introduces several new arguments, including binary, gcov_args, gcov_executable, gcov_ignore, gcov_include, report_type, and skip_validation, and changes the file and plugin arguments to files and plugins, respectively.
• Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#80). In this release, the databrickslabs/sandbox dependency has been updated from version acceptance/v0.3.1 to 0.4.2. This update includes the addition of install instructions, more go-git libraries, and modifications to the README to explain how to use the library with the databricks labs sandbox command. Dependency updates include golang.org/x/crypto from version 0.16.0 to 0.17.0. The Run nightly tests job in the workflow has also been updated to use the new version of the databrickslabs/sandbox/acceptance image. The commit history for this release shows several commits, including the creation of "[TODO] XXX" issues and a full diff comparison. The pull request includes instructions for triggering Dependabot actions through comments. Reviewers are encouraged to thoroughly examine the changelog and commit history for more detailed information on the changes in this release.
• Force keyword argument in make_query fixture (#81). In the latest update, the make_query fixture in the redash.py file has undergone changes to enhance code readability and maintainability. The sql_query parameter is now required to be passed as a keyword argument, preventing the possibility of it being mistakenly passed as a positional argument. Furthermore, the create function's signature has been revised to include an explicit * before the sql_query parameter. It is important to note that these changes have not affected the functionality of the method, but have instead altered the parameter passing style for improved clarity. This minor update is intended to elevate the overall quality of the codebase and promote best practices.
• Renamed internal notebooks module to workspace (#86). In this release, the internal notebooks module has been renamed to workspace to better reflect its current functionality of managing and interacting with notebooks and other resources in a Databricks workspace. This renaming applies to the import statements in plugin.py and test_notebooks.py, which has been renamed to test_workspace.py. Additionally, import statements for making cluster policy and instance pool permissions have been added. The functionality of the imported functions, such as make_directory, make_workspace_file, make_notebook, and make_repo, remains unchanged. This change is part of the fix for issue #59 and aims to improve the clarity and consistency of the codebase. Software engineers adopting this project should update any imports or references to the notebooks module to use the new workspace module instead.

Dependency updates:

• Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#80).
• Bump codecov/codecov-action from 4 to 5 (#85).

Contributors: @dependabot[bot], @nfx, @JCZuurmond

v0.4.0

• Added Volume Fixture (#72). This commit introduces a Managed Volume fixture, make_volume, to the Unity Catalog in the test suite, facilitating the creation and use of a random volume for testing purposes. The fixture, when called without arguments, generates a volume with a random name. Alternatively, specifying the name argument creates a volume with the given name, using the MANAGED volume type, and associating it with a randomly generated catalog and schema. This promotes test isolation and prevents unintended interference. Additionally, this PR resolves issue #70 and includes unit and integration tests that have been manually verified to ensure the fixture's proper functioning. The commit also demonstrates a bug fix related to table retrieval in the test_remove_after_property_table test case.

Contributors: @christophergrant

v0.3.1

• Updated databrickslabs/sandbox requirement to acceptance/v0.3.1 (#64). In this pull request, we are updating the requirement for the databrickslabs/sandbox package to version acceptance/v0.3.1. This update is necessary to resolve any conflicts and ensure compatibility with the latest version of the package. The update includes several bug fixes, dependency updates, and the addition of install instructions in the changelog. The package also includes new git-related libraries and modifications to the README to explain how to use the package with the databricks labs sandbox command. Additionally, there are dependency updates for golang.org/x/crypto in the go-libs and runtime-packages directories. Keeping dependencies up-to-date is important for taking advantage of new features and bug fixes, so it is recommended to merge this pull request and address any conflicts or issues that may arise. This update is generated by Dependabot, a tool used to keep dependencies up-to-date, and it will handle any conflicts as long as the pull request is not modified. The databrickslabs/sandbox package is used for building and testing the project.
• Updating make_schema fixture to include location to create managed schema (#66). In this update, the make_schema fixture has been enhanced to include an optional location parameter, enabling users to specify the location for creating a managed schema. This change addresses issue #6
• [chore] update acceptance.yml and remove circular downstreads. In this release, the acceptance.yml file in the .github/workflows directory has been updated to use the acceptance/v0.3.1 Docker image for the Run integration tests job, replacing the previous acceptance/v0.2.2 version. This change ensures that the latest version of the integration tests are being run, which may include bug fixes, new features, and other improvements. The timeout parameter has been removed from the job, which may result in longer test execution times but ensures that the tests have enough time to complete. Additionally, the ARM_CLIENT_ID and GITHUB_TOKEN secrets are now passed as environment variables to the job. The downstreams.yml file has also been updated to remove circular dependencies and streamline the build process, while maintaining the use of the ubuntu-latest runtime and fail-fast option. The Run nightly tests job has been updated to use the acceptance/v0.3.1 Docker image, which may improve test execution during the nightly build due to potential bug fixes, new functionality, or better handling of timeout scenarios. The create_issues parameter remains set to true, ensuring that any test failures will result in the creation of GitHub issues to track them. These changes aim to improve the reliability and efficiency of the build and test processes.

Dependency updates:

• Updated databrickslabs/sandbox requirement to acceptance/v0.3.1 (#64).

Contributors: @nfx, @dependabot[bot], @HariGS-DB

v0.3.0

• Documentation: fix make_query() parameter name (#61). The make_query() fixture's documentation has been updated to correct the name of the query parameter to sql_query. The sql_query parameter is used to specify the SQL query stored in the fixture, with the default value being "SELECT * FROM ". This change aims to enhance clarity and consistency in the naming of the argument, making it easier for users of the make_query() fixture to comprehend its purpose and usage. By correcting the parameter name, the documentation provides a clearer and more consistent user experience.
• Removed references to UCX (#56). This release includes changes to remove references to UCX in fixture names and descriptions within the testing process. The create function in catalog.py now appends a random string to "dummy_t", "dummy_s", or dummy_c for table, schema, and catalog names, respectively, instead of using "ucx_t", "ucx_S", and "ucx_C". The test_catalog_fixture function has also been updated to replace dummy with dummy_c and dummy_s for catalogs and schemas. Additionally, the description of a test query in redash.py has been updated to remove the reference to UCX. Lastly, fixture names in the unit tests for a catalog have been updated to use dummy instead of "ucx". These changes improve the independence of the testing process by removing technology-specific references, without affecting functionality.
• Store watchdog tags in storage credentials comment (#57). In this release, the watchdog's behavior has been modified to retain properly tagged credentials when deleting them, as previously all credentials were removed without discrimination. This change introduces tagging for preserving specific credentials, and the watchdog_remove_after fixture has been added to the README file for documentation. The make_storage_credential fixture has been updated to include a new parameter, watchdog_remove_after, which specifies the time at which the storage credential should be removed by the watchdog. The create function has been updated to accept this parameter and adds it as a comment to the storage credential. The remove function remains unmodified. The related fixtures section has been updated to include the new watchdog_remove_after fixture. This change was co-authored by Eric Vergnaud, but please note that it has not been tested yet.
• [FEATURE] Extend make_job to run SparkPythonTask (#60). The make_job fixture has been extended to support running SparkPythonTask in addition to notebook tasks. A new make_workspace_file fixture has been added to create and manage Python files in the workspace. The make_job fixture now supports SQL notebooks and files and includes a task_type parameter to specify the type of task to run and an instance_pool_id parameter to reuse an instance pool for faster job execution during integration tests. Additionally, unit and integration tests have been added to ensure the proper functioning of the new and modified fixtures. These changes allow for more flexible and efficient testing of Databricks jobs with different task types and configurations. The make_notebook fixture has also been updated to accept a content parameter for creating notebooks with custom content. The Language enum from the databricks.sdk.service.workspace module is used to specify the language of a notebook or workspace file.

Contributors: @JCZuurmond, @ericvergnaud, @asnare

v0.2.4

• Fixed PyPI metadata (#54). In this commit, the PyPI metadata for the pytester project has been updated with the new repository location at https://github.com/databrickslabs/pytester. The URLs for issues and source have been changed to point to the new repository, with the issues URL now directing to https://github.com/databrickslabs/pytester/issues and the source URL to https://github.com/databrickslabs/pytester. Furthermore, the versioning tool hatch has been configured to manage the version number in the "src/databricks/labs/pytester/about.py" file. This ensures accurate and consistent versioning for the pytester project moving forward.
• Improve make_group/make_acc_group fixture consistency (#50). This PR introduces improvements to the make_group and make_acc_group fixtures, designed for managing Databricks workspace groups. The enhancements include a double-check approach to ensure group visibility by requiring the group to be retrievable via both .get() and .list() calls. This mitigates, but does not entirely eliminate, consistency issues with the APIs used for managing groups. The wait_for_provisioning argument has been removed and replaced with an internal wait mechanism. The argument is still accepted but triggers a deprecation warning. Internal unit-test plumbing has been updated to use mock fixtures tailored for each test, ensuring double-check implementation testability. New and updated unit tests are included in the test_iam.py file, along with the introduction of the _setup_groups_api function, which mocks specific clients to ensure group visibility when created. These changes improve consistency and reliability when working with Databricks workspace groups, making it easier for users to adopt the project.

Contributors: @asnare, @nfx

v0.2.3

• Support providing name in make_catalog fixture (#52). The make_catalog fixture in our open-source library has been updated to allow users to specify a name for the catalog using a new name parameter. Previously, the catalog was given a random name, but now users can have more control and customization over catalog names in their tests. This change includes updates to the docstring and the addition of unit tests to ensure the fixture behaves as expected with the new parameter. Additionally, the underlying call_stateful function was updated to expect a callable that returns a generator of callables, enabling the support for providing a name. The test_make_catalog_creates_catalog_with_name and test_make_catalog tests have been added to verify the behavior of the fixture with the new name parameter.

Contributors: @asnare, @JCZuurmond