[FEATURE] Extend make_job to run SparkPythonTask
#60
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JCZuurmond
temporarily deployed
to
account-admin
GitHub Actions
Inactive
|
Needs #58 to let CI pass |
|
This PR breaks backwards compatibility for databrickslabs/blueprint downstream. See build logs for more details. Running from downstreams #61 |
|
This PR breaks backwards compatibility for databrickslabs/lsql downstream. See build logs for more details. Running from downstreams #61 |
|
✅ 37/37 passed, 3 skipped, 8m38s total Running from acceptance #102 |
1 task
1 task
JCZuurmond
had a problem deploying
to
account-admin
GitHub Actions
Error
JCZuurmond
temporarily deployed
to
account-admin
GitHub Actions
Inactive
1 task
nfx
added a commit
that referenced
this pull request
Oct 8, 2024
* Documentation: fix `make_query()` parameter name ([#61](#61)). The `make_query()` fixture's documentation has been updated to correct the name of the `query` parameter to `sql_query`. The `sql_query` parameter is used to specify the SQL query stored in the fixture, with the default value being "SELECT \* FROM <newly created random table>". This change aims to enhance clarity and consistency in the naming of the argument, making it easier for users of the `make_query()` fixture to comprehend its purpose and usage. By correcting the parameter name, the documentation provides a clearer and more consistent user experience. * Removed references to UCX ([#56](#56)). This release includes changes to remove references to UCX in fixture names and descriptions within the testing process. The `create` function in `catalog.py` now appends a random string to "dummy_t", "dummy_s", or `dummy_c` for table, schema, and catalog names, respectively, instead of using "ucx_t", "ucx_S", and "ucx_C". The `test_catalog_fixture` function has also been updated to replace `dummy` with `dummy_c` and `dummy_s` for catalogs and schemas. Additionally, the description of a test query in `redash.py` has been updated to remove the reference to UCX. Lastly, fixture names in the unit tests for a catalog have been updated to use `dummy` instead of "ucx". These changes improve the independence of the testing process by removing technology-specific references, without affecting functionality. * Store watchdog tags in storage credentials comment ([#57](#57)). In this release, the watchdog's behavior has been modified to retain properly tagged credentials when deleting them, as previously all credentials were removed without discrimination. This change introduces tagging for preserving specific credentials, and the `watchdog_remove_after` fixture has been added to the README file for documentation. The `make_storage_credential` fixture has been updated to include a new parameter, `watchdog_remove_after`, which specifies the time at which the storage credential should be removed by the watchdog. The `create` function has been updated to accept this parameter and adds it as a comment to the storage credential. The `remove` function remains unmodified. The related fixtures section has been updated to include the new `watchdog_remove_after` fixture. This change was co-authored by Eric Vergnaud, but please note that it has not been tested yet. * [FEATURE] Extend `make_job` to run `SparkPythonTask` ([#60](#60)). The `make_job` fixture has been extended to support running `SparkPythonTask` in addition to notebook tasks. A new `make_workspace_file` fixture has been added to create and manage Python files in the workspace. The `make_job` fixture now supports SQL notebooks and files and includes a `task_type` parameter to specify the type of task to run and an `instance_pool_id` parameter to reuse an instance pool for faster job execution during integration tests. Additionally, unit and integration tests have been added to ensure the proper functioning of the new and modified fixtures. These changes allow for more flexible and efficient testing of Databricks jobs with different task types and configurations. The `make_notebook` fixture has also been updated to accept a `content` parameter for creating notebooks with custom content. The `Language` enum from the `databricks.sdk.service.workspace` module is used to specify the language of a notebook or workspace file.
Merged
nfx
added a commit
that referenced
this pull request
Oct 8, 2024
* Documentation: fix `make_query()` parameter name ([#61](#61)). The `make_query()` fixture's documentation has been updated to correct the name of the `query` parameter to `sql_query`. The `sql_query` parameter is used to specify the SQL query stored in the fixture, with the default value being "SELECT \* FROM <newly created random table>". This change aims to enhance clarity and consistency in the naming of the argument, making it easier for users of the `make_query()` fixture to comprehend its purpose and usage. By correcting the parameter name, the documentation provides a clearer and more consistent user experience. * Removed references to UCX ([#56](#56)). This release includes changes to remove references to UCX in fixture names and descriptions within the testing process. The `create` function in `catalog.py` now appends a random string to "dummy_t", "dummy_s", or `dummy_c` for table, schema, and catalog names, respectively, instead of using "ucx_t", "ucx_S", and "ucx_C". The `test_catalog_fixture` function has also been updated to replace `dummy` with `dummy_c` and `dummy_s` for catalogs and schemas. Additionally, the description of a test query in `redash.py` has been updated to remove the reference to UCX. Lastly, fixture names in the unit tests for a catalog have been updated to use `dummy` instead of "ucx". These changes improve the independence of the testing process by removing technology-specific references, without affecting functionality. * Store watchdog tags in storage credentials comment ([#57](#57)). In this release, the watchdog's behavior has been modified to retain properly tagged credentials when deleting them, as previously all credentials were removed without discrimination. This change introduces tagging for preserving specific credentials, and the `watchdog_remove_after` fixture has been added to the README file for documentation. The `make_storage_credential` fixture has been updated to include a new parameter, `watchdog_remove_after`, which specifies the time at which the storage credential should be removed by the watchdog. The `create` function has been updated to accept this parameter and adds it as a comment to the storage credential. The `remove` function remains unmodified. The related fixtures section has been updated to include the new `watchdog_remove_after` fixture. This change was co-authored by Eric Vergnaud, but please note that it has not been tested yet. * [FEATURE] Extend `make_job` to run `SparkPythonTask` ([#60](#60)). The `make_job` fixture has been extended to support running `SparkPythonTask` in addition to notebook tasks. A new `make_workspace_file` fixture has been added to create and manage Python files in the workspace. The `make_job` fixture now supports SQL notebooks and files and includes a `task_type` parameter to specify the type of task to run and an `instance_pool_id` parameter to reuse an instance pool for faster job execution during integration tests. Additionally, unit and integration tests have been added to ensure the proper functioning of the new and modified fixtures. These changes allow for more flexible and efficient testing of Databricks jobs with different task types and configurations. The `make_notebook` fixture has also been updated to accept a `content` parameter for creating notebooks with custom content. The `Language` enum from the `databricks.sdk.service.workspace` module is used to specify the language of a notebook or workspace file.
nfx
pushed a commit
to databrickslabs/ucx
that referenced
this pull request
Oct 8, 2024
…nt` workflow (#2815) ## Changes Scope resources to run the assessment workflow for in integration tests to shorten the time it takes for the assessment to run: <img width="1582" alt="Screenshot 2024-10-04 at 17 45 52" src="https://github.com/user-attachments/assets/8ee12e89-1068-45f4-a5de-c2da121a6101"> - Move the populate for linting logic to the context - Set the `include_job_ids` to the job ids created by the populate for linting logic - Set the `include_dashboard_ids` to the dashboard ids created by the populate for linting logic - Move the create job based on Python file fixture to [pytester](databrickslabs/pytester#60) ### Linked issues Resolves #2637 Resolves #2849 ### Tests - [x] modified integration tests: - `test_running_real_assessment_job` - `test_running_real_assessment_job_ext_hms` - `test_running_real_migration_progress_job`
nfx
pushed a commit
to databrickslabs/ucx
that referenced
this pull request
Oct 8, 2024
## Changes This PR updates the minimum required version of `pytester` from 0.2.1 to 0.3.0. As of #2852 our integration tests depends on changes introduced in databrickslabs/pytester#60 (and released with 0.3.0).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Extend
make_jobto runSparkPythonTaskby:make_workspace_filefixture to create the Python file to run.task_typetomake_jobto signal what type of task to run (notebook or python file)instance_pool_idtomake_jobfor speeding up a job in integration tests by reusing an instance poolTests