v0.7.4

• Increased the number of random characters used when creating resources to minimize collisions (#214). The continuous integration workflow has been updated to install a newer version of the hatch tool, specifically version 1.15.0, replacing the previous version 1.9.4, to avoid compatibility issues with click. Additionally, the code has been modified to increase the number of random characters used when creating resources from 4 to 8, aiming to minimize collisions, particularly in larger projects that rely heavily on PyTester. This change affects various resource creation methods and is intended to reduce the likelihood of name collisions and improve the overall stability of the system. The changes are covered by regression testing and have been manually tested to ensure their correctness and effectiveness.

Contributors: @mwojtyczka

v0.7.3

• Added property id to get service principal id (#170). The service principal functionality has been enhanced with the introduction of a new id property, which returns the service principal ID as a string, simplifying its usage in tests. The create method has been updated to utilize this id property, ensuring consistency throughout the code. Additionally, the id property is now directly used when creating secrets, updating group memberships, and updating workspace assignments, with the exception of the workspace_assignment.update method. The logging of account links has also been updated to use the id property. Furthermore, test cases have been updated to include service principals in group members, and a new function make_run_as has been introduced to create service principals for testing purposes, along with a new test case test_make_run_as_no_args to verify its functionality, addressing issues related to no-cheat in linked pull requests and improving overall testing capabilities.
• Updated call_fixture() to work with pytest 8.4 (#166). The call_fixture function has been enhanced to support pytest version 8.4 and later, while maintaining backward compatibility with older versions. To address the deprecation of direct access to pytest fixtures since version 4.x, the function now dynamically checks the pytest version and utilizes the corresponding mechanism to unwrap fixtures. For pytest 8.4 and later, it leverages the _get_wrapped_function attribute, whereas for older versions, it relies on the __pytest_wrapped__ attribute. Additionally, the function now includes input validation, raising a ValueError if the provided function is not a valid pytest fixture, ensuring more robust error handling. This update provides a temporary solution, with plans for future refinement to potentially utilize pytest.FixtureRequest and request.getfixturevalue() for accessing fixtures in a more supported and sustainable manner.

Contributors: @asnare, @pritishpai

v0.7.2

• Compatibility fix for Databricks SDK 0.51 (#158). The minimum required version of the Databricks SDK has been updated to 0.51.0 to accommodate breaking changes introduced in this version, including the removal of certain enumeration values. Notably, the ServedModelInputWorkloadSize enumeration has been replaced with well-known string constants, prompting an update to the create function to use the string constant Small instead of the removed ServedModelInputWorkloadSize.SMALL value. This change affects the creation of serving endpoints, which now specify workload size using a string value, and resolves issues reported in related projects, including test failures in downstream projects, by ensuring compatibility with the updated SDK version.

Contributors: @asnare

v0.7.1

• Set upper SDK limit to v1 (#109). In this change, the upper SDK limit is set to version 1.0 in the pyproject.toml file of a library project, allowing projects using this library to set their own appropriate SDK limits. The databricks-sdk dependency is updated from "<0.42" to "<1.0" as a result of this commit. Additionally, the create function in the ml.py file of the pytester library has been modified. Previously, the EndpointCoreConfigInput object was passed as an argument to the serving_endpoints.create method using the keyword argument served_models. Now, EndpointCoreConfigInput is passed as a keyword argument config, which contains the served_models parameter, aligning with the new SDK version 1.0 and ensuring consistent endpoint creation. These changes are intended to ensure compatibility with a variety of SDK versions and improve the library's usability for software engineers.

Contributors: @JCZuurmond

v0.7.0

• Let make_serving_endpoint reference a valid model version (#106). In this release, the make_serving_endpoint fixture has been updated to enhance its functionality and flexibility. The fixture now accepts optional keyword arguments for specifying the endpoint name, model name, and model version, with the model version defaulting to 1 and the latest version automatically retrieved for workspace local models. The served model input is created with the specified model name, model version, and workload size, and a mock endpoint object is returned if obtaining the served model fails. The remove method has been updated to accept a ServingEndpointDetailed object for deletion, and the fixture now maintains a logger instance for warning messages during model version retrieval. Unit tests have been added to ensure the changes work as expected, although integration tests have not been fixed yet. These updates address issues arising from changes in the make_model fixture and improve the robustness and flexibility of the make_serving_endpoint fixture.

Contributors: @mwojtyczka, @JCZuurmond

v0.6.0

What's Changed

• Added serverless support to spark fixture by @mwojtyczka in #91
• Release v0.6.0 by @mwojtyczka in #96
• Fix release pipeline to use known runners by @mwojtyczka in #97

Full Changelog: v0.5.0...v0.6.0

v0.5.0

• Added make_run_as fixture (#82). A new pytest fixture, make_run_as, has been added to create an account service principal via the acc fixture and assign it to a workspace with default permissions, which is removed after the test is complete. The fixture creates a service principal with a random display name and assigns it to the workspace. Users can optionally assign the service principal to account groups for specific actions using the account_groups argument. The returned object contains properties for the workspace client, SQL backend, and SQL execution functions, as well as the display name and application ID of the ephemeral service principal. If desired, the ws fixture can be overridden to make all workspace fixtures provided by the plugin run as the ephemeral service principal, allowing for testing with lower privilege ephemeral service principals and improving security and isolation. This feature is not currently supported with Databricks Metadata Service authentication on Azure Databricks.
• Bump codecov/codecov-action from 4 to 5 (#85). In this release, the Codecov GitHub Action has been updated from version 4 to 5, introducing several new features and changes. The new version uses the Codecov Wrapper to encapsulate the CLI, allowing for quicker updates to the Action. Additionally, version 5 includes an opt-out feature for tokens in public repositories, enabling contributors and other members to upload coverage reports without requiring access to the Codecov token. This can be accomplished by setting the ability for Codecov to receive a coverage report from any source in the Global Upload Token section of the settings page on codecov.io. Furthermore, the updated version introduces several new arguments, including binary, gcov_args, gcov_executable, gcov_ignore, gcov_include, report_type, and skip_validation, and changes the file and plugin arguments to files and plugins, respectively.
• Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#80). In this release, the databrickslabs/sandbox dependency has been updated from version acceptance/v0.3.1 to 0.4.2. This update includes the addition of install instructions, more go-git libraries, and modifications to the README to explain how to use the library with the databricks labs sandbox command. Dependency updates include golang.org/x/crypto from version 0.16.0 to 0.17.0. The Run nightly tests job in the workflow has also been updated to use the new version of the databrickslabs/sandbox/acceptance image. The commit history for this release shows several commits, including the creation of "[TODO] XXX" issues and a full diff comparison. The pull request includes instructions for triggering Dependabot actions through comments. Reviewers are encouraged to thoroughly examine the changelog and commit history for more detailed information on the changes in this release.
• Force keyword argument in make_query fixture (#81). In the latest update, the make_query fixture in the redash.py file has undergone changes to enhance code readability and maintainability. The sql_query parameter is now required to be passed as a keyword argument, preventing the possibility of it being mistakenly passed as a positional argument. Furthermore, the create function's signature has been revised to include an explicit * before the sql_query parameter. It is important to note that these changes have not affected the functionality of the method, but have instead altered the parameter passing style for improved clarity. This minor update is intended to elevate the overall quality of the codebase and promote best practices.
• Renamed internal notebooks module to workspace (#86). In this release, the internal notebooks module has been renamed to workspace to better reflect its current functionality of managing and interacting with notebooks and other resources in a Databricks workspace. This renaming applies to the import statements in plugin.py and test_notebooks.py, which has been renamed to test_workspace.py. Additionally, import statements for making cluster policy and instance pool permissions have been added. The functionality of the imported functions, such as make_directory, make_workspace_file, make_notebook, and make_repo, remains unchanged. This change is part of the fix for issue #59 and aims to improve the clarity and consistency of the codebase. Software engineers adopting this project should update any imports or references to the notebooks module to use the new workspace module instead.

Dependency updates:

• Bump databrickslabs/sandbox from acceptance/v0.3.1 to 0.4.2 (#80).
• Bump codecov/codecov-action from 4 to 5 (#85).

Contributors: @dependabot[bot], @nfx, @JCZuurmond

v0.4.0

• Added Volume Fixture (#72). This commit introduces a Managed Volume fixture, make_volume, to the Unity Catalog in the test suite, facilitating the creation and use of a random volume for testing purposes. The fixture, when called without arguments, generates a volume with a random name. Alternatively, specifying the name argument creates a volume with the given name, using the MANAGED volume type, and associating it with a randomly generated catalog and schema. This promotes test isolation and prevents unintended interference. Additionally, this PR resolves issue #70 and includes unit and integration tests that have been manually verified to ensure the fixture's proper functioning. The commit also demonstrates a bug fix related to table retrieval in the test_remove_after_property_table test case.

Contributors: @christophergrant

v0.3.1

• Updated databrickslabs/sandbox requirement to acceptance/v0.3.1 (#64). In this pull request, we are updating the requirement for the databrickslabs/sandbox package to version acceptance/v0.3.1. This update is necessary to resolve any conflicts and ensure compatibility with the latest version of the package. The update includes several bug fixes, dependency updates, and the addition of install instructions in the changelog. The package also includes new git-related libraries and modifications to the README to explain how to use the package with the databricks labs sandbox command. Additionally, there are dependency updates for golang.org/x/crypto in the go-libs and runtime-packages directories. Keeping dependencies up-to-date is important for taking advantage of new features and bug fixes, so it is recommended to merge this pull request and address any conflicts or issues that may arise. This update is generated by Dependabot, a tool used to keep dependencies up-to-date, and it will handle any conflicts as long as the pull request is not modified. The databrickslabs/sandbox package is used for building and testing the project.
• Updating make_schema fixture to include location to create managed schema (#66). In this update, the make_schema fixture has been enhanced to include an optional location parameter, enabling users to specify the location for creating a managed schema. This change addresses issue #6
• [chore] update acceptance.yml and remove circular downstreads. In this release, the acceptance.yml file in the .github/workflows directory has been updated to use the acceptance/v0.3.1 Docker image for the Run integration tests job, replacing the previous acceptance/v0.2.2 version. This change ensures that the latest version of the integration tests are being run, which may include bug fixes, new features, and other improvements. The timeout parameter has been removed from the job, which may result in longer test execution times but ensures that the tests have enough time to complete. Additionally, the ARM_CLIENT_ID and GITHUB_TOKEN secrets are now passed as environment variables to the job. The downstreams.yml file has also been updated to remove circular dependencies and streamline the build process, while maintaining the use of the ubuntu-latest runtime and fail-fast option. The Run nightly tests job has been updated to use the acceptance/v0.3.1 Docker image, which may improve test execution during the nightly build due to potential bug fixes, new functionality, or better handling of timeout scenarios. The create_issues parameter remains set to true, ensuring that any test failures will result in the creation of GitHub issues to track them. These changes aim to improve the reliability and efficiency of the build and test processes.

Dependency updates:

• Updated databrickslabs/sandbox requirement to acceptance/v0.3.1 (#64).

Contributors: @nfx, @dependabot[bot], @HariGS-DB

v0.3.0

• Documentation: fix make_query() parameter name (#61). The make_query() fixture's documentation has been updated to correct the name of the query parameter to sql_query. The sql_query parameter is used to specify the SQL query stored in the fixture, with the default value being "SELECT * FROM ". This change aims to enhance clarity and consistency in the naming of the argument, making it easier for users of the make_query() fixture to comprehend its purpose and usage. By correcting the parameter name, the documentation provides a clearer and more consistent user experience.
• Removed references to UCX (#56). This release includes changes to remove references to UCX in fixture names and descriptions within the testing process. The create function in catalog.py now appends a random string to "dummy_t", "dummy_s", or dummy_c for table, schema, and catalog names, respectively, instead of using "ucx_t", "ucx_S", and "ucx_C". The test_catalog_fixture function has also been updated to replace dummy with dummy_c and dummy_s for catalogs and schemas. Additionally, the description of a test query in redash.py has been updated to remove the reference to UCX. Lastly, fixture names in the unit tests for a catalog have been updated to use dummy instead of "ucx". These changes improve the independence of the testing process by removing technology-specific references, without affecting functionality.
• Store watchdog tags in storage credentials comment (#57). In this release, the watchdog's behavior has been modified to retain properly tagged credentials when deleting them, as previously all credentials were removed without discrimination. This change introduces tagging for preserving specific credentials, and the watchdog_remove_after fixture has been added to the README file for documentation. The make_storage_credential fixture has been updated to include a new parameter, watchdog_remove_after, which specifies the time at which the storage credential should be removed by the watchdog. The create function has been updated to accept this parameter and adds it as a comment to the storage credential. The remove function remains unmodified. The related fixtures section has been updated to include the new watchdog_remove_after fixture. This change was co-authored by Eric Vergnaud, but please note that it has not been tested yet.
• [FEATURE] Extend make_job to run SparkPythonTask (#60). The make_job fixture has been extended to support running SparkPythonTask in addition to notebook tasks. A new make_workspace_file fixture has been added to create and manage Python files in the workspace. The make_job fixture now supports SQL notebooks and files and includes a task_type parameter to specify the type of task to run and an instance_pool_id parameter to reuse an instance pool for faster job execution during integration tests. Additionally, unit and integration tests have been added to ensure the proper functioning of the new and modified fixtures. These changes allow for more flexible and efficient testing of Databricks jobs with different task types and configurations. The make_notebook fixture has also been updated to accept a content parameter for creating notebooks with custom content. The Language enum from the databricks.sdk.service.workspace module is used to specify the language of a notebook or workspace file.

Contributors: @JCZuurmond, @ericvergnaud, @asnare