- Added databricks_volume resource in Unity Catalog (#2324).
- Added databricks_model_serving support to databricks_permissions (#2331).
- Added suppress diff for URL change that only changes
/(remove or add) in UC resources (#2336). - Fixed attributes typo in SCIM API (#2344).
- Fixed updates for databricks_share resource (#2307).
- Updated documentation for databricks_service_principal_secret (#2332).
- Updated documentation for troubleshooting guide with a typical error when creating groups/users on the Account level (#2338).
- Other testing infrastructure improvements (#2350, #2355, #2358).
Updated dependency versions:
- Bump github.com/hashicorp/hcl/v2 from 2.16.2 to 2.17.0 (#2359).
- Bump github.com/stretchr/testify from 1.8.3 to 1.8.4 (#2354).
- Bump github.com/zclconf/go-cty from 1.13.1 to 1.13.2 (#2329).
- Removed support for releasing 32-bit binaries (#2315, #2320).
- Added more information on impact of using a cluster policy in databricks_cluster resource (#2313).
- Added missing
serverlessoption to databricks_pipeline (#2308). - Updated
channelandeditionvalues in databricks_pipeline docs (#2322). - Automatically add
CAN_MANAGEpermission on databricks_instance_pool for calling user (#2298). - Migrated databricks_ip_access_list resource to Go SDK (#2306).
Updated dependency versions:
- Bump github.com/stretchr/testify from 1.8.2 to 1.8.3 (#2317).
- Bump github.com/databricks/databricks-sdk-go from v0.8.1 to v0.9.0 (#2327).
- Added databricks_service_principal_secret documentation (#2296).
- Documentation about Unity Catalog integration in databricks_pipeline (#2289).
- Exporter: don't emit databricks_permissions for
/Shareddirectory (#2288). - Fix dependency when exporting databricks_service_principal_role (#2285).
- Removed reference to deprecated databricks_group_instance_profile from databricks_group_role docs (#2290).
- Updated documentation for databricks_secret_scope resource (#2297).
- Added databricks_workspace_file resource (#2266).
- Added
notification_settingsblock to databricks_job resource (#2276). - Added missing permission to databricks_aws_crossaccount_policy data source (#2283).
- Fixed databricks_sql_table doc (#2281).
- Fixed doc on tag propagation and tag conflict for databricks_instance_pool resource (#2242).
- Document & export databricks_workspace_conf parameters for legacy init scripts (#2280).
- Removed deprecated
CREATE_VIEWfrom code of databricks_grants resource (#2230). - Exporter: relax handling of problematic files/directories (#2258).
- Added experimental databricks_sql_table resource to manage tables in the Unity Catalog (#2213).
- Added databricks_pipelines data source (#2202).
- Added missing permissions to databricks_grants resource (#2260).
- Added support for running databricks_cluster init scripts from workspace files (#2251).
- Added suppress diff for
run_ifin databricks_job (#2229). - Fixed delete permission error for manually removed databricks_job (#2225).
- Documented
fleetselector in the databricks_node_type data source (#2261). - Documented example for databricks_job tags (#2235).
- Documented more Azure Databricks workspaces creation (#2228, #2248).
- Removed note about Azure KeyVault-based databricks_secret_scope and service principals (#2231).
- Migrated databricks_cluster_policy to Go SDK (#2250).
- Updated databricks_sql_endpoint docs for Azure DBSQL Serverless GA (#2249, #2254).
- Updated policy definition in databricks_aws_crossaccount_policy data source (#2262).
- Bumped
databricks-sdk-gofrom 0.7.0 to 0.8.0 (#2259).
- Add exporter support for databricks_sql_alert (#2207).
- Add support for notifications in databricks_pipeline resource (#2218).
- Add suppress diff for
pathargument of databricks_directory resource when it ends with/(#2204). - Exporter: don't export
storagewhen databricks_pipeline was created with default value (#2203). - Exporter: phase 1 of support for Account-level exports (#2205).
- Fixed regression in databricks_group data not to require workspace admin privileges (#2210).
- Suppress diff for
editionattribute in databricks_pipeline (#2219).
- Explicitly include SCIM attributes for databricks_group, databricks_user, databricks_user_role, databricks_group_role, databricks_group_member, databricks_group_instance_profile, databricks_user data, databricks_group data, and databricks_entitlement resources (#2200).
- Added SQL File task to databricks_job resource (#2199).
- Fix diff detection for databricks_share resource (#2197).
- Guide to deploy Databricks on Azure with PrivateLink - Standard deployment (#2066).
- Remove
CREATE_VIEWfrom databricks_grants doc (#2198).
Updated dependency versions:
- Bump github.com/databricks/databricks-sdk-go from 0.6.0 to 0.7.0 (#2195).
- Bump golang.org/x/mod from 0.9.0 to 0.10.0 (#2192).
- Allow rotating
tokenblock in databricks_mws_workspaces resource by only changingcommentfield (#2114). - Exclude roles in SCIM API list calls to reduce load on Databricks SCIM service (#2181).
- Fixed databricks_service_principals data source issue with empty filter (#2185).
- Update Go SDK to v0.6.0 (#2186).
- Added caching of SCIM Me API call to databricks_current_user data source and databricks_permissions resource. (#2170).
- Added example for automatically rotating a token (#2135).
- Added
run_ifcondition in databricks_job resource (#2125). - Added File Arrival trigger to databricks_job resource (#2142).
- Added
sourceparameter forspark_python_taskin databricks_job (#2157). - Automatically add
CAN_MANAGEpermission on databricks_sql_endpoint for calling user (#2168). - Deprecated
enable_serverless_computeon databricks_sql_global_config resource (#2139). - Document
enable_serverless_computeAPI changes in databricks_sql_endpoint resource (#2137). - Fix edge cases for databricks_permissions resource (#2158).
- Migrate databricks_cluster_policy data source to Go SDK (#2155).
- Update the description of
ip_addressesparameter in databricks_ip_access_list docs (#2116).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.25.0 to 2.26.1 (#2140).
- Bump github.com/zclconf/go-cty from 1.13.0 to 1.13.1 (#2124).
- Added databricks_sql_alert resource (#2047).
- Added experimental GCP support for more resources (#2088, #2091, #2090, #2089, #2080).
- Added suppress diff to
default_data_access_config_idin databricks_metastore (#2111). - Export
initialize_file_systemfor ADLS Gen2 mounts (#2107). - Enforce formatting on pull requests (#2099).
- Added
MatchRegexpmatch type for partial matches (#2102). - Skip model serving integration test on GCP (#2098).
- More doc fixes (#2108, #2106).
Updated dependency versions:
- Bump github.com/databricks/databricks-sdk-go from 0.4.0 to 0.5.0 (#2096, #2112).
- Bump github.com/hashicorp/hcl/v2 from 2.16.1 to 2.16.2 (#2097).
- Added databricks_model_serving resource (#2054).
- Added Unity Catalog on GCP support (#2000).
- Deprecate
scheduleblock in databricks_sql_query resource (#2078). - Fixed InitScripts Type to work with GCS and ABFS in databricks_cluster resource (#2067).
- Added more testing for databricks_tables data source (#2075).
- Added more testing for databricks_schemas data source (#2074).
- Migrated databricks_node_type data source to Go SDK (#2070).
- Migrated databricks_schemas data to Go SDK (#2065).
- Migrated databricks_tables data source to SDK (#2068).
- Migrated databricks_views data source to Go SDK (#2073).
- Migrated databricks_git_credential to Go SDK (#2069).
- Migrated databricks_shares data source to Go SDK (#2072).
Updated dependency versions:
- Bump github.com/databricks/databricks-sdk-go from v0.3.3 to v0.4.0 (#2086).
- Bump golang.org/x/mod from 0.8.0 to 0.9.0 (#2076).
- Databricks on Azure with PrivateLink - Simplified deployment (#1977).
- Deterministic diff for databricks_permissions (#2059).
- Added
force_delete_home_dirandforce_delete_reposattributes to databricks_user and databricks_service_principal resources (#2032). - Added docs for
continuousblock in the databricks_job resource (#2048). - Exporter:
databricks_permissionsfor databricks_notebook & databricks_directory (#1908). - Improve error messages for databricks_permissions (#2055).
- Make reflect resource utility friendly with Go SDK (#2051).
Updated dependency versions:
- Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 (#2049).
- Migrated databricks_catalogs data to Go SDK (#2038).
- Migrated databricks_current_user data source to Go SDK (#2037).
- Migrated databricks_workspace_conf to Go SDK (#2035).
- Added new feature (#2015).
- Minor documentation improvements (#2039).
Updated dependency versions:
- Bump github.com/databricks/databricks-sdk-go from v0.3.2 to v0.3.3 (#2044).
- Bump github.com/zclconf/go-cty from 1.12.1 to 1.13.0 (#2043).
- Migrated client and configuration layer to
github.com/databricks/databricks-sdk-go(#1848). - Added new attribute to databricks_job resource (#1988).
- Added
history_data_sharing_status,partitionandstatusfields to databricks_share (#2013). - Added
start_versionandcdf_enabledsupport for databricks_share (#2007). - Fixed databricks_share update logic (#2022).
- Updated GCP guide with new permissions required to deploy databricks_mws_networks (#1999).
- Minor doc changes to address #1916, #1937, #2010 (#2012).
- Various documentation improvements (#1981, #1998, #2002, #2016, #2019).
- Integration testing improvements (#1935, #2024).
Updated dependency versions:
- Removed unused
github.com/Azure/go-autorest/autorestandgithub.com/mitchellh/go-homedir(#2026). - Bump Go from 1.18 to 1.19 (#2029).
- Bump CodeQL to v2 (#2027).
- Bump github.com/golang-jwt/jwt/v4 from 4.4.3 to 4.5.0 (#2030).
- Bump github.com/hashicorp/hcl/v2 from 2.15.0 to 2.16.1 (#1971, #1997).
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.24.1 to 2.25.0 (#2031).
- Bump golang.org/x/mod from 0.7.0 to 0.8.0 (#1996).
- Bump golang.org/x/net from 0.6.0 to 0.7.0 (#2028).
- Bump google.golang.org/api from 0.108.0 to 0.110.0 (#1984, #1995).
- Added
filelibrary type to databricks_pipeline resource (#1975). - Added new node type selector to databricks_node_type data source (#1957).
- Removed check for creation of Azure Key Vault-based databricks_secret_scope using Service Principals (#1965).
- Reworked Graviton selection logic in databricks_node_type data source (#1974).
- Don't consider GPU-based nodes in databricks_node_type data source until explicitly requested (#1978).
Documentation:
- Added SQL example for databricks_job documentation (#1967).
- Added description to
max_clusters_per_userin databricks_cluster_policy (#1959). - Clarify SQL warehouse types for
sql_taskin databricks_job (#1964). - Moved tags to
default_tagsin AWS provider examples (#1969). - Added documentation around
webhooks_notificationsfor databricks_job (#1968). - Removed deprecated params from bucket doc examples (#1970).
- Added
max_clusters_per_userto databricks_cluster_policy resource (#1955). - Added support for
cluster_mount_infosin databricks_cluster (#1945). - Added
parentattribute to databricks_sql_query and databricks_sql_dashboard (#1949). - Added support for
iam_role_arnin databricks_instance_profile (#1943). - Use
zone_id = "auto"for databricks_mount on S3 (#1932). - Fixed databricks_cluster resource missing library definitions issues (#1925).
- Remove
computedannotation foraws_attributes(#1933).
Updated dependency versions:
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.21 to 0.9.22 (#1953).
- Bump github.com/Azure/go-autorest/autorest/azure/auth (#1952).
- Bump google.golang.org/api from 0.106.0 to 0.107.0 (#1929).
- Bump google.golang.org/api from 0.107.0 to 0.108.0 (#1954).
- Added databricks_mws_credentials data source (#1909).
- Fixed incorrect diff for
num_clustersandmin_num_clustersfor stopped databricks_sql_endpoint (#1927). - Fixed
privilege_assignmentin databricks_sql_permissions (#1872). - Documentation improvements (#1921, #1923).
Updated dependency versions:
- Bump github.com/hashicorp/go-retryablehttp from 0.7.1 to 0.7.2 (#1913).
- Bump google.golang.org/api from 0.105.0 to 0.106.0 (#1914).
- Added support for GCP (Public Preview) with state upgraders for databricks_mws_workspaces (#1871, #1886, #1748, #1879).
- Added documentation for databricks_permission_assignment (#1880).
- Added
azure_attributesto clusters in databricks_pipeline resource (#1854). - Added databricks_instance_pool data source (#1907).
- Added databricks_provider resource for Delta Sharing (#1572).
- Added acceptance tests for databricks_cluster data and databricks_sql_endpoint data (#1882).
- Added search by name for databricks_cluster data source (#1901).
- Added support for sparse checkouts in the databricks_repo (#1869).
- Added
config_referenceattribute to databricks_secret for easier reference from Spark configuration (#1898). - Added
datarbicks_directorydata source (#1902). - Extended databricks_catalog resource to support Delta Sharing Catalog (#1887).
- Improved databricks_metastore_assignment resource (#1900).
- Improved databricks_grants error messages (#1888).
- Improved databricks_group documentation (#1816).
- Improved
data_security_modedocumentation for databricks_cluster (#1830). - Replace the incorrect
idattribute with the correctsp_idin service principal data source docs example (#1873). - Use NVMe disk size for selection in the databricks_node_type data source (#1856).
- Allowed to override resource ID for specific data source (#1885).
Experimental Resource Exporter
- Added command-line flag to export all users and service principals (#1893).
- Added emit for secret scopes from Spark Conf and Env Vars (#1897).
- Fixed issue with exporting of resources with simple name (#1891).
- Generate references that are matching only to a prefix (#1890).
- Updated list of supported services and associated resources (#1894).
- Use databricks_group_role instead of deprecated databricks_group_instance_profile (#1905).
- Use
dbfs_pathas reference for init scripts in clusters (#1892). - Logging improvements (#1895).
Updated dependency versions:
- Bump google.golang.org/api from 0.104.0 to 0.105.0 (#1863).
- Added support for Unity Catalog on GCP through
gcp_service_account_key(#1836). - Added support for Jupyter (
.ipynb) files in databricks_notebook resource (#1801). - Added databricks_cluster_policy data source (#1792).
- Added
catalogparameter todbt_taskin databricks_job resource (#1774). - Added
homeandreposattributes to databricks_user and databricks_service_principal (#1791). - Added
storage_rootparameter in databricks_catalog and databricks_schema resources (#1784). - Added exporting of databricks_service_principal resource (#1811).
- Added notebook
sourceparameter for databricks_jobs data (#1837). - Added support for init scripts on ADLS in databricks_cluster and related resources (#1845).
- Added
definitonattribute to databricks_cluster_policy data (#1834). - Added validation for preview account API paths (#1795).
- Made
lifetime_seconds&commentoptional for databricks_obo_token (#1844). - Force replace when changing
display_nameof databricks_service_principal (#1783). - We can now export
adminandusersgroups as databricks_group data, not resource (#1721). - We can now export predefined databricks_cluster_policy as data sources (#1815).
- We can now export databricks_sql_global_config resource (#1806).
- Fixed reference to databricks_cluster_policy in databricks_job during export (#1813).
- We can now explicitly normalize variable names during export (#1835).
- We can now fail fast in
import.shwhen can't import resource (#1814). - Fixed multidomain databricks_user in exporter (#1722).
- Fixed missing attributes from databricks_job data source (#1798).
- Suppressed
networkdiff for GCP workspaces with Databricks-managed VPC (#1847). - Improved export of DBSQL objects (#1788).
- Documented
webhook_notificationsfor databricks_job resource (#1842). - Clarified databricks_job,
databricks_mws_*, databricks_cluster docs (#1796).
Updated dependency versions:
- Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.4.3 (#1817).
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.24.0 to 2.24.1 (#1761).
- Bump golang.org/x/mod from 0.6.0 to 0.7.0 (#1760).
- Bump google.golang.org/api from 0.102.0 to 0.104.0 (#1758, #1839).
- Added
query_planattribute todatabricks_sql_visualization(#1752). - Fixed
member_idreference of nested groups and databricks_group_member for exporter (#1723). - Fixed auto-purged cluster behaviour for databricks_library (#1745).
- Use Jobs API 2.1 with name filter for databricks_job data source (#1744).
- Reverted
PROas defaultwarehouse_typein databricks_sql_endpoint (#1741). - Moved testing of clusters, instance pools, and SQL Warehouses to internally configured test environment variables (#1739).
Updated dependency versions:
- Bump google.golang.org/api from 0.101.0 to 0.102.0 (#1736).
- Added
warehouse_typeparameter to databricks_sql_endpoint to support PRO SKU (#1728). - Correct exporting of the computed attributes for databricks_cluster (#1711).
- Escape database name in databricks_sql_permissions resource (#1730).
- Added
runtime_engineto databricks_cluster (#1686). - Added validation for
pathin databricks_repo (#1702). - Added auto-detection of AWS CodeCommit URLs in databricks_repo (#1704).
- Restricting access to S3 bucket by custom tag on the IAM identity according to security email in databricks_aws_bucket_policy data resource (#1694).
- Update Azure Unity Catalog guide to use
azurerm_databricks_access_connector(#1685). - Clarify that databricks_mws_permission_assignment should be used for assigning account-level users/groups (#1706).
- Other documentation fixes (#1696, #1692).
Updated dependency versions:
- Bump github.com/stretchr/testify from 1.8.0 to 1.8.1 (#1689).
- Bump github.com/zclconf/go-cty from 1.11.1 to 1.12.0 (#1714).
- Bump golang.org/x/mod from 0.5.1 to 0.6.0 (#1690).
- Bump google.golang.org/api from 0.99.0 to 0.101.0 (#1713, #1691).
- Added
CAN_VIEWpermissions for databricks_sql_dashboard and databricks_sql_query (#1682). - Added
webhook_notificationssupport to databricks_job (#1674). - Allow updating
private_access_settings_idfor databricks_mws_workspaces (#1668). - Changed databricks_pipeline edition default value from
advancedtoADVANCED(#1683). - Fixed reference to
display_name_containsin docs example (#1677).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.23.0 to 2.24.0 (#1671).
- Bump github.com/zclconf/go-cty from 1.11.0 to 1.11.1 (#1672).
- Bump google.golang.org/api from 0.98.0 to 0.99.0 (#1673).
- Added databricks_share resource, databricks_shares data source, and databricks_share data source (#1664).
- Documentation updates (#1666, #1669).
- Added
enable_serverless_computein the documentation for the databricks_sql_global_config (#1655). - Update databricks_grants with new permissions model (#1657).
- Exporter now adds the
forceflag to users and groups (#1661).
Updated dependency versions:
- Bump google.golang.org/api from 0.97.0 to 0.98.0 (#1652).
- Added databricks_job data resource (#1509).
- Fixed handling of references in databricks_permissions resource exporter (#1631).
- Fixed stripping of
CAN_MANAGEpermission from caller of databricks_sql_query resource (#1644). - Fixed provider debug mode (#1560).
Documentation improvements:
- Expanded documentation for databricks_sql_visualization resource (#1630).
Updated dependency versions:
- Bump github.com/hashicorp/hcl/v2 from 2.14.0 to 2.14.1 (#1634).
- Bump google.golang.org/api from 0.96.0 to 0.97.0 (#1633).
- Added autoscale
modeconfiguration to databricks_pipeline resource (#1600). - Fixed
gcp_availabilityfield in databricks_insance_pool resource (#1610). - Fixed
secretfield name in databricks_mlflow_webhook resource (#1625). - Further improvements of exporter (#1602).
Documentation improvements:
- Updates for databricks_table and databricks_mws_vpc_endpoint (#1616).
- Document
http_timeout_secondsfromcommon/client.go(#1599). - Expand
parameterdocumentation for databricks_sql_query resource (#1603). - Updated Unity Catalog docs (#1605, #1626, #1620).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.21.0 to 2.23.0 (#1597, #1611).
- Bump google.golang.org/api from 0.94.0 to 0.96.0 (#1598, #1612).
- Added
force_destroyflag to databricks_schema & databricks_catalog (#1578). - Added databricks_entitlements resource (#1583).
- Added databricks_group_role resource (#1575).
- Added databricks_recipient resource for Delta Sharing (#1571).
- Added
dbt_taskfield to databricks_job (#1537). - Fixed drift in
storagefor databricks_pipeline (#1574). - Fixed databricks_mws_private_access_settings defaults (#1567).
- Fixed databricks_user creation with
forceon account (#1577). - Fixed databricks_permissions for calling user to
CAN_MANAGEon databricks_cluster (#1579).
Documentation improvements:
- Added
sql_taskconfiguration block in databricks_job resource (#1589). - Added supported languages in Unity Shared clusters (#1587).
- Removed "Public Preview" marker from Unity Catalog resources docs (#1570).
- Fixed instance pool docs (#1581).
Updated dependency versions:
- Bump github.com/hashicorp/hcl/v2 from 2.13.0 to 2.14.0 (#1582).
- Bump google.golang.org/api from 0.93.0 to 0.94.0 (#1569).
- Use resize API to scale
databricks_clusterwhile it's running (#1541). - Updated to latest Unity Catalog privileges model (#1556).
- Added cluster policy support for
databricks_pipeline(#1554). - Fixed
databricks_node_typeby skipping nodes that aren't available in subscription/region (#1534). - Fixed sending of
activeflag in thedatabricks_user(#1536). - Fixed Azure CLI tests on Go 1.19 (#1538).
- Various doc updates (#1553, #1552, #1544, #1543).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.20.0 to 2.21.0 (#1540).
- Bump github.com/zclconf/go-cty from 1.10.0 to 1.11.0 (#1558).
- Bump google.golang.org/api from 0.90.0 to 0.93.0 (#1525, #1545).
- Bump gopkg.in/ini.v1 from 1.66.6 to 1.67.0 (#1526).
- Added databricks_mws_permission_assignment resource (#1491).
- Added databricks_mws_workspaces data resource (#1497).
- Added exporter for databricks_pipeline and databricks_job (#1484).
- Fixed databricks_permissions for calling user to
CAN_MANAGEon databricks_mlflow_model (#1507). - Fixed phantom
clusterblocks in databricks_pipeline (#1508). - Improve test coverage of multitask databricks_job and databricks_pipeline (#1493).
- Minor stability improvements (#1505, #1487, #1496, #1503).
Updated dependency versions:
- Bump github.com/Azure/go-autorest/autorest from 0.11.27 to 0.11.28 (#1513).
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.20 to 0.9.21 (#1512).
- Bump github.com/Azure/go-autorest/autorest/azure/cli from 0.4.5 to 0.4.6 (#1515).
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.19.0 to 2.20.0 (#1516).
- Bump google.golang.org/api from 0.87.0 to 0.90.0 (#1499, #1514).
- Added
databricks_sql_warehousesdata resource to list all warehouses in a workspace with a name filter (#1460). - Added
databricks_sql_warehousedata resource to list SQL warehouse attributes based on single id (#1460). - Added
databricks_clusterdata resource to list cluster attributes based on single id (#1460). - Added Azure Managed Identity documentation examples (#1471).
- Added more attributes to
databricks_cluster(#1459). - Added more attributes to
databricks_instance_pool(#1463). - Added feature request GitHub issue template (#1482).
- Added
tf:optionaltag to simplify the code (#1395). - Fixed
databricks_pipelineincorrect generation ofclusterblocks (#1416). - Fixed
databricks_tableupdate forcolumnblock (#1468). - Fixed reads for
any_fileandanonymous_functionindatabricks_sql_permissions(#1477). - Tuned integration tests for
databricks_mws_*(#1483). - Removed integration tests for
databricks_azure_adls_gen1_mount(#1461).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.18.0 to 2.19.0 (#1475).
- Bump google.golang.org/api from 0.86.0 to 0.87.0 (#1476).
- Added
metastoretodatabricks_grants(#1447). - Added update support for
databricks_mws_log_delivery(#1439). - Fixed name generation for exported
databricks_notebook(#1435). - Fixed
databricks_jobrecreation on changeddocker_imageURL (#1437). - Fixed replace
databricks_mws_networksonvpc_endpointschange (#1453). - Fixed diff suppress in
databricks_external_locationonskip_validation(#1421). - Switched to SQL warehouses API for
databricks_sql_endpoint(#1414). - Improve docs for
databricks_permissionsanddatabricks_obo_token(#1462). - Improve docs for
abfsblock indatabricks_mount(#1446). - Improve dev docs for
tf:"suppress_diff",tf:"force_new"andtf:"sensitive"tags (#1465). - Moved couple of acceptance tests out of preview (#1433).
- Migrated to Go 1.18:
interface{}->any(#1466).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.17.0 to 2.18.0 (#1450).
- Added better handling of missing objects during import (#1417).
- Fix problem in the
databricks_node_typedata source (#1400). - Fix update for
databricks_storage_credentialresource (#1403). - Improve
databricks_jobresource (#1428). - Upgraded UC API from 2.0 to 2.1 (#1418).
- Sanitized
hostprovider field prior to auth flow (#1385). - Use
delta_sharing_scopeinstead ofdelta_sharing_enabled(#1398). - Added instructions to create missing
.terraform.lock.hcl(#1397). - Clarified
databricks_permissionsdoc for service principals (#1426). - Clarified
databricks_clusterautotermination_minutesdefault value (#1419). - Fully moved codebase from
databrickslabstodatabricksnamespace (#1429). - Various integration testing improvements (#1425, #1427, #1420).
Updated dependency versions:
- Bump Go from 1.16.x to 1.18.x (#1413).
- Bump github.com/golang-jwt/jwt/v4 from 4.4.1 to 4.4.2 (#1407).
- Bump github.com/hashicorp/hcl/v2 from 2.12.0 to 2.13.0 (#1406).
- Bump github.com/stretchr/testify from 1.7.3 to 1.8.0 (#1387, #1408, #1422).
- Bump google.golang.org/api from 0.84.0 to 0.86.0 (#1386, #1423).
To make Databricks Terraform Provider generally available, we've moved it from https://github.com/databrickslabs to https://github.com/databricks. We've worked closely with the Terraform Registry team at Hashicorp to ensure a smooth migration. Existing terraform deployments continue to work as expected without any action from your side. We ask you to replace databrickslabs/databricks with databricks/databricks in all your .tf files.
You should have .terraform.lock.hcl file in your state directory that is checked into source control. terraform init will give you the following warning.
Warning: Additional provider information from registry
The remote registry returned warnings for registry.terraform.io/databrickslabs/databricks:
- For users on Terraform 0.13 or greater, this provider has moved to databricks/databricks. Please update your source in required_providers.
After you replace databrickslabs/databricks with databricks/databricks in the required_providers block, the warning will disappear. Do a global "search and replace" in *.tf files. Alternatively you can run python3 -c "$(curl -Ls https://dbricks.co/updtfns)" from the command-line, that would do all the boring work for you.
If you didn't check-in .terraform.lock.hcl to the source code version control, you may you may see Failed to install provider error. Please follow the simple steps described in the troubleshooting guide.
- Add a warning in
databricks_permissionstoken usage docs (#1380).
Updated dependency versions:
- Bump google.golang.org/api from 0.83.0 to 0.84.0
- Bump github.com/stretchr/testify from 1.7.2 to 1.7.3
- Added
databricks_service_principalanddatabricks_service_principalsdata resources (#1370). - Updated
databricks_pipelineresource to match latest APIs (#1368). - Made
gcp_managed_network_configoptional indatabricks_mws_workspaces(#1365). - Enforced consistent naming for resource files (#1366, #1369).
- Updated resources diagram (#1373).
Updated dependency versions:
- Bump google.golang.org/api from 0.82.0 to 0.83.0
- Added
delta_sharing_*support todatabricks_metastore(#1334). - Added
databricks_git_credentialspat discovery from common environment variables (#1353). - Added
databricks_permissionsfordatabricks_pipeline(#1361). - Added
network_idtonetworkblock indatabricks_mws_workspacesfor GCP (#1360). - Added
azure_managed_identityblock todatabricks_storage_credentialanddatabricks_metastore_data_accessresources (#1354). - Update docs regarding importing of
databricks_sql_*resources (#1349). - Apply ownership for UC objects during creation (#1338).
- Re-create purged cluster for
databricks_mountfor Google Storage (#1333). - Various documentation fixes (#1350).
Updated dependency versions:
- Bump google.golang.org/api from 0.80.0 to 0.81.0
- Bump gopkg.in/ini.v1 from 1.66.4 to 1.66.6
- Bump google.golang.org/api from 0.81.0 to 0.82.0
- Bump github.com/stretchr/testify from 1.7.1 to 1.7.2
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.16.0 to 2.17.0
- Added warning section for debug mode (#1325).
- Added ability to specify tags for
databricks_job(#1337). - Upgraded AWS provider for AWS guides. Added examples for account-level identities (#1332).
- Updated docs to use
application_idas privilege fordatabricks_service_principal(#1336). - Added
databricks_service_principal_roleresource (#1340). - Fixed itegration testing image (#1342, #1343).
- Added
skip_validationfordatabricks_external_location(#1330). - Added
alert_on_last_attempttodatabricks_job(#1341). - Skip
make teston doc-only changes (#1339). - Improve common package test coverage (#1344).
- Re-create purged cluster for
databricks_mountfor AWS S3 (#1345).
Updated dependency versions:
- Bump google.golang.org/api from 0.79.0 to 0.80.0
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.19 to 0.9.20
- Update
aws_iam_policy_documentindatabricks_mws_customer_managed_keysdocs to restrict KMS policy to caller AWS account (#1309). - Added
gcsdestination toinit_scriptsindatabricks_cluster(#1308). - Clarify optionality of
databricks_mws_workspaces.deployment_namein docs and examples (#1315). - Update
databricks_mws_log_deliverydocs (#1320). - Fix updating
databricks_service_principalon Azure (#1322). - Added
tf:suppress_diffonartifact_locationfordatabricks_mlflow_experiment(#1323).
Updated dependency versions:
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.18 to 0.9.19
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.14.0 to 2.16.0
- Bump google.golang.org/api from 0.77.0 to 0.79.0
- Added
external_idandforceattributes todatabricks_service_principalresource (#1293). - Added documentation for
databicks_git_credentialresource (#1295). - Added documentation for
git_sourceindatabricks_job(#1297). - Fix
job_cluster.num_workersindatabricks_job(#1284). - Various documentation improvements (#1292), (#1296), (#1298).
Updated dependency versions:
- Bump google.golang.org/api from 0.75.0 to 0.77.0
- Removed github.com/pkg/errors dependency
- Added
databricks_viewsdata resource, makingdatabricks_tablesreturn only managed or external tables in Unity Catalog (#1274). - Added default timeout of 20m to
databricks_mount(#1280). - Made
common.DataResourcedeterministic (#1279). - Fixed exporting text-only widgets (#1278).
- Updated devcontainer to support ARM (#1256).
- Various documentation fixes (#1285, #1282, #1281, #1276).
Updated dependency versions:
- Bump github.com/hashicorp/hcl/v2 from 2.11.1 to 2.12.0
- Bump github.com/Azure/go-autorest/autorest from 0.11.26 to 0.11.27
- Added configuration generators for
databricks_sql_*resources in experimental Resource Exporter (#1199). - Added
google_credentialsprovider agument that has the same semantics ascredentialsargument in officialgoogleprovider (#1214). - Fixed
databricks_grantson UC external location empty list error (#1202). - Fixed errors in
databricks_permissionsresource for auto-purgeddatabricks_cluster(#1252). - Various documentation fixes (#1231, #1239, #1254, #1240).
Updated dependency versions:
- Bump google.golang.org/api from 0.71.0 to 0.75.0
- Bump github.com/golang-jwt/jwt/v4 from 4.3.0 to 4.4.1
- Bump github.com/stretchr/testify from 1.7.0 to 1.7.1
- Bump github.com/hashicorp/go-retryablehttp from 0.7.0 to 0.7.1
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.11.0 to 2.14.0
- Bump github.com/Azure/go-autorest/autorest from 0.11.24 to 0.11.26
- Completely removed custom client-side validation in
databricks_service_principal(#1193). - Added export functionality for Databricks SQL objects - endpoints, queries, dashboards, widgets, visualizations (#1199).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.10.1 to 2.11.0
- Failures in exporter resource listing no longer halt the entire command run (#1166).
- Removed client-side validation in
databricks_service_principalforapplication_id, that may not always be available in the planning stage (#1165). - Use correct HTTP verb for modifying
databricks_permissionsondatabricks_sql_endpointentities. Authorized user, assumingly part ofadminsgroup, is no longer sendingCAN_MANAGEpermission in the HTTP PUT request (#1163). - Added diff suppression for
min_num_clustersfield indatabricks_sql_endpoint(#1172). - Added special case for handling
Cannot access cluster that was terminated or unpinned more than 30 days agoerror indatabricks_clusteras an indication of resource removed on the platform side (#1177). - Fixed updating of
databricks_tableresources (#1175). - Fixed configuration drift in
databricks_grantby reading existing permissions and removing them in subsequent update calls (#1164).
Updated dependency versions:
- Bump google.golang.org/api from 0.70.0 to 0.71.0
- Added
databricks_catalogs,databricks_schemas, anddatabricks_tablesdata resources (#1155). - Fixed
databricks_metastore_assignmentconfiguration drift by properly deleting metastore assignment and detecting manual changes from Account console. This also means that de-assigned metastore from a workspace would mark it as remotely removed. Manual assignment of different metastore would also trigger resource updates (#1146). - Fixed
databricks_tablecreation in managed mode (#1151). - Fixed
databricks_sql_endpointtimeout propagation (#1142). - Multiple documentation fixes.
Updated dependency versions:
- Bump google.golang.org/api from 0.69.0 to 0.70.0
- Added an extended documentation from provisioning AWS PrivateLink workspace (#1084).
- Added
databricks_jobsdata resource to get a map of all job names and their ids (#1138).
Updated dependency versions:
- Bump google.golang.org/api from 0.68.0 to 0.69.0
- Added
workspace_urlattribute to thedatabricks_current_userdata source (#1107). - Fixed issue at
databricks_mountwhere new cluster was created for S3 mount even whencluster_idwas specified (#1064). - Allow to disable auto-termination for Databricks SQL endpoints (#900).
- Added new
gcp_attributestodatabricks_clusteranddatabricks_instance_pool(#1126). - Added exporter functionality for
databricks_ip_access_listanddatabricks_workspace_conf(#1125). - Added
gravitonselector fordatabricks_node_typeanddatabricks_spark_versiondata sources (#1127). - Added interactive mode to resource exporter (#1010).
- Added preview support for
git_sourceindatabricks_job(#1090). - Multiple other fixes and documentation improvements.
Updated dependency versions:
- Bump github.com/golang-jwt/jwt/v4 from 4.2.0 to 4.3.0
- Bump google.golang.org/api from 0.67.0 to 0.68.0
- Bump gopkg.in/ini.v1 from 1.66.3 to 1.66.4
- Prevent creation of
databricks_groupwithusersandadminsreserved names (#1089). - Added support for shared clusters in multi-task
databricks_job(#1082). - Added diff suppression for
external_idindatabricks_group(#1099). - Added diff suppression for
external_idindatabricks_user(#1097). - Added
users,service_principals, andchild_groupsexported properties todatabricks_groupdata resource (#1085). - Added various documentation improvements.
Deprecations
databricks_group.membersis deprecated in favor ofusers,service_principals, andchild_groupsexported properties. Please do slight modifications of your configuration.
Updated dependency versions:
- Bump google.golang.org/api from 0.66.0 to 0.67.0
- Added support for
tf:suppress_diffon primitive types (#984). - Fixed issue arises when destroying
databricks_sql_global_configwith instance profile set (#1076). - Added setting of SQL configuration parameters in
databricks_sql_global_config(#1080). - Added support for release channels in
databricks_sql_endpointconfiguration (#1078). - Added documentation for
databricks_mlflow_webhookresource (#1086).
Updated dependency versions:
- Bump google.golang.org/api from 0.65.0 to 0.66.0
- Added optional
forceargument todatabricks_groupresource to ignorecannot create group: Group with name X already exists.errors and implicitly import the specific group into Terraform state, enforcing entitlements defined in the instance of resource (#1066). - Added support to configure permissions for all MLflow models (#1044).
- Fixed
databricks_service_principaldisplay_nameupdate (#1065). - Added documentation for Unity Catalog resources.
Updated dependency versions:
- Bump gopkg.in/ini.v1 from 1.66.2 to 1.66.3
- Clarified error messages around
azure_workspace_resource_idprovider configuration (#1049). - Added optional
forceargument todatabricks_userresource to ignorecannot create user: User with username X already existserrors and implicitly import the specific user into Terraform state, enforcing entitlements defined in the instance of resource (#1048). - Added
databricks_user_roleresource, that can assign roles on Databricks Account ordatabricks_instance_profilefor data access (#1047).
Deprecations
databricks_user_instance_profileis deprecated in favor ofdatabricks_user_role. Please do slight modifications of your configuration.
Updated dependency versions:
- Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.10 to 0.5.11
- Bump github.com/Azure/go-autorest/autorest/azure/cli from 0.4.3 to 0.4.5
- Bump github.com/Azure/go-autorest/autorest from 0.11.23 to 0.11.24
Updated dependency versions:
- Bump google.golang.org/api from 0.63.0 to 0.65.0
- Added support for running provider in a debug mode from Visual Studio Code through
Debug Providerrun configuration in order to troubleshoot more complicated issues. - Allowed managing of libraries on
databricks_clusteroutside of Terraform state for resources without anylibraryconfiguration blocks. This should simplify PaaS-like CI/CD workflows (#1024). - Added experimental resources.
Behavior changes
- Whenever library is installed on
databricks_clusterwithout anylibraryconfiguration blocks, it won't be removed anymore.
- Added support for
databricks_permissionsfordatabricks_mlflow_experimentanddatabricks_mlflow_model(#1013). - Added
Using XXX authexplanation to HTTP 403 errors, which should help troubleshooting misconfigured authentication or provider aliasing. Example error message now looks like: cannot create group: /2.0/preview/scim/v2/Groups is only accessible by admins. Using databricks-cli auth: host=https://XXX.cloud.databricks.com/, token=***REDACTED***, profile=demo. All sensitive configuration parameters (token,password, andazure_client_secret) are redacted and replaced with***REDACTED***(#821). - Improved documentation with regards to public subnets in AWS quick start (#1005).
- Added
databricks_mountcode genration for exporter tooling (#1006). - Increase dependency check frequency (#1007).
- Added experimental resources.
- Added optional
auth_typeprovider conf to enforce specific auth type to be used in very rare cases, where a single Terraform state manages Databricks workspaces on more than one cloud andMore than one authorization method configurederror is a false positive. Valid values arepat,basic,azure-client-secret,azure-msi,azure-cli, anddatabricks-cli(#1000). - Added
DBCformat support fordatabricks_notebook(#989). - Fixed creating new
databricks_mws_workspaceswithtoken {}block (#994). - Added automated documentation formatting with
make fmt-docs, so that all HCL examples look consistent (#999). - Increased codebase unit test coverage to 91% to improve stability (#996, #992, #991, #990).
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.10.0 to 2.10.1
- Added
databricks_libraryresource to install library ondatabricks_cluster(#904). - Added
databricks_clustersdata resource to list all clusters in the workspace, which might be used to installdatabricks_libraryon all clusters (#955). - Fixed refresh of
libraryblocks on a stoppeddatabricks_cluster(#952). - Whenever a library fails to get installed on a running
databricks_cluster, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts ofdatabricks_clusterresource. (#599). - Added
tokenblock todatabricks_mws_workspacesto avoid unnecessary provider aliasing (#957). - Fixed disabling
databricks_global_init_script(#958). - Fixed configuration drift issues with
aws_attributes,azure_attributes,gcp_attributes, andemail_notificationsconfiguration blocks indatabricks_cluster,databricks_job, anddatabricks_instance_poolresources (#981). - Improved Databricks CLI auth by eagerly resolving
host,username,password, andtokenfrom the specifiedprofile. Added explicit logging of auth parameters in debug logs (#965). - TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy (#966).
debug_headersprovider conf is also logging theHostheader to help troubleshooting auth issues (#964).- Added new experimental resources and increased test coverage.
Updated dependency versions:
- Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
- Bump google.golang.org/api from 0.60.0 to 0.63.0
- Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
- Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
- Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
- Added
databricks_mlflow_modelanddatabricks_mlflow_experimentresources (#931) - Added support for
repo_pathtodatabricks_permissionsresource (#875). - Added
external_idtodatabricks_useranddatabricks_group(#927). - Fixed
databricks_repocreation corner cases on MS Windows OS (#911). - Fixed configuration drift for
databricks_cluster.aws_attributes.zone_idwithauto, which resulted in unwanted cluster restarts (#937). - Added new experimental resources, increased test coverage, and automated integration testing infrastructure.
- Multiple documentation improvements and new guides.
Behavior changes
- Renamed
allow_sql_analytics_accesstodatabricks_sql_accessindatabricks_user,databricks_group, anddatabricks_service_principalresources. - Removed deprecated
azure_use_pat_for_spn,azure_use_pat_for_cli,azure_pat_token_duration_secondsprovider attributes. - Removed deprecated
azure_workspace_name,azure_resource_group,azure_subscription_idin favor of just usingazure_workspace_resource_id. - Renamed
DATABRICKS_AZURE_WORKSPACE_RESOURCE_IDenvironment variable toDATABRICKS_AZURE_RESOURCE_ID. DATABRICKS_AZURE_CLIENT_SECRETenvironment variable is no longer having any effect in favor of just usingARM_CLIENT_SECRET.DATABRICKS_AZURE_CLIENT_IDenvironment variable is no longer having any effect in favor of just usingARM_CLIENT_ID.DATABRICKS_AZURE_TENANT_IDenvironment variable is no longer having any effect in favor of just usingARM_TENANT_ID.
Updated dependency versions:
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.7.1 to 2.9.0
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.16 to 0.9.17
- Bump github.com/golang-jwt/jwt/v4 from 4.0.0 to 4.1.0
- Bump github.com/zclconf/go-cty from 1.9.1 to 1.10.0
- Bump github.com/Azure/go-autorest/autorest from 0.11.21 to 0.11.22
- Added
databricks_sql_global_configresource to provide global configuration for SQL Endpoints (#855) - Added
databricks_mountresource to mount arbitrary cloud storage (#497) - Improved implementation of
databricks_repoby creating the parent folder structure (#895) - Fixed
databricks_joberror related to randomized job IDs (#901) - Replace
databricks_groupon name change (#890) - Names of scopes in
databricks_secret_scopecan have/characters in them (#892)
Deprecations
databricks_aws_s3_mount,databricks_azure_adls_gen1_mount,databricks_azure_adls_gen2_mount, anddatabricks_azure_blob_mountare deprecated in favor ofdatabricks_mount.
Updated dependency versions:
- Bump google.golang.org/api from 0.59.0 to 0.60.0
- Added
private_access_levelandallowed_vpc_endpoint_idstodatabricks_mws_private_access_settingsresource, which is also now updatable (#867). - Fixed missing diff skip for
skip_validationindatabricks_instance_profile(#860). - Added support for
pipeline_task(871) andpython_wheel_task(#872) todatabricks_job. - Improved enterprise HTTPS proxy support for creating workspaces in PrivateLink environments (#882).
- Added
hostnameattribute toodbc_paramsindatabricks_sql_endpoint(#868). - Improved documentation (#858, #870).
Updated dependency versions:
- Bumped google.golang.org/api from 0.58.0 to 0.59.0
- Added initial support for multiple task orchestration in
databricks_job#853 - Fixed provider crash for new terraform states related to bug #813 in Terraform SDK v2.8.0 (#854)
- Re-added
skip_validationtodatabricks_instance_profileresource #762 - Removed direct dependency on
aws-sdk-go.
Updated dependency versions:
- Reverted github.com/hashicorp/terraform-plugin-sdk/v2 from 2.8.0 to 2.7.0
- Added
databricks_reporesource to manage Databricks Repos (#771) - Added support for Azure MSI authentication (#743)
- Added support to create
databricks_useron the account level (#818) - Already deleted
databricks_tokendon't fail the apply (#808) - Default
terraform-mountclusters created for mounting fordatabricks_aws_s3_mount,databricks_azure_adls_gen1_mount,databricks_azure_adls_gen2_mount, anddatabricks_azure_blob_mounthave nowspark.scheduler.modeasFIFO(#828) - Fixed crash when using non-Azure authentication to mount Azure resources (#831)
- Fixed replacement of
instance_pool_idindatabricks_cluster, whendriver_instance_pool_idwas not explicitly specified (#824) - Ingorning diff customization for permissions resource, so that new workspace deployments won't fail without explicit dependency on a workspace resource
- Multiple documentation improvements
Deprecations
azure_workspace_name,azure_resource_group,azure_subscription_id, andazure_workspace_resource_idare deprecated and would be removed in v0.4.0. Please rewrite provider configuration withhost = data.azurerm_databricks_workspace.example.workspace_urlto achieve the same effect. Please check azurerm_databricks_workspace resource documentation for details.azure_use_pat_for_spn,azure_use_pat_for_cli, andazure_pat_token_duration_secondsare deprecated to fully switch to AAD token authentication in the near future.DATABRICKS_AZURE_CLIENT_SECRETenvironment variable is deprecated in favor of just usingARM_CLIENT_SECRET.DATABRICKS_AZURE_CLIENT_IDenvironment variable is deprecated in favor of just usingARM_CLIENT_ID.DATABRICKS_AZURE_TENANT_IDenvironment variable is deprecated in favor of just usingARM_TENANT_ID.
Updated dependency versions:
- Bump github.com/aws/aws-sdk-go from 1.40.19 to 1.40.54
- Bump github.com/Azure/go-autorest/autorest from 0.11.19 to 0.11.21
- Bump github.com/Azure/go-autorest/autorest/azure/cli from 0.4.2 to 0.4.3
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.14 to 0.9.16
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.7.0 to 2.8.0
- Bump github.com/zclconf/go-cty from 1.9.0 to 1.9.1
- Bump golang.org/x/mod from 0.4.2 to 0.5.1
- Bump google.golang.org/api from 0.52.0 to 0.58.0
- Bump gopkg.in/ini.v1 from 1.62.0 to 1.63.2
- Added
databricks_obo_tokenresource to create On-Behalf-Of tokens for a Service Principal in Databricks workspaces on AWS. It is very useful, when you want to provision resources within a workspace through narrowly-scoped service principal, that has no access to other workspaces within the same Databricks Account (#736) - Added support for IAM credential passthrough with
is_meta_instance_profileproperty fordatabricks_instance_profile(#745) - Fixed incorrect workspace update bug and added more validation error messaging (#649)
- Clarify network modification procedure on active workspaces (#732)
- Updated AWS IAM policy templates version to
2012-10-17(databricks_aws_bucket_policy,databricks_aws_assume_role_policy, anddatabricks_aws_crossaccount_policy) (#688) - Various bug fixes in Databricks SQL resources
Updated dependency versions:
- Bump github.com/aws/aws-sdk-go to v1.40.12
- Bump github.com/hashicorp/hcl/v2 to v2.10.1
- Bump github.com/zclconf/go-cty to v1.9.0
- Bump golang.org/x/time to v0.0.0-20210723032227-1f47c861a9ac
- Bump golang.org/x/tools to v0.1.5
- Added support for hybrid pools (#689)
- Added support for
always_runningjobs, which are restarted on resource updates (#715) - Azure CLI auth is now forcing JSON output (#717)
databricks_permissionsare getting validation onterraform planstage (#706)- Added
databricks_directoryresource (#690) - Added
run_as_rolefield todatabricks_sql_query(#684) - Added
user_idattribute fordatabricks_userdata resource, so that it's possible to dynamically create resources based on members of the group (#714) - Added more selectors to
databricks_node_typedata source (#723) - Azure auth with SPN now uses AAD token by default instead of PAT. Previous behavior (using PAT) could be restored by setting
azure_use_pat_for_spntotrue(#721) deployment_namefordatabricks_mws_workspacesis now optional, how it should have been. This enables creation of Databricks workspaces without an account prefix.- To replicate default behavior of Databricks SQL UI,
enable_photonis nowtrueby default fordatabricks_sql_endpoint. - Various documentation and bugfixes
Updated dependency versions:
- Bump github.com/aws/aws-sdk-go from 1.38.51 to 1.38.71
- Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.7 to 0.5.8
- Bump github.com/Azure/go-autorest/autorest from 0.11.18 to 0.11.19
- Bump github.com/Azure/go-autorest/autorest/adal from 0.9.13 to 0.9.14
- Bump github.com/zclconf/go-cty from 1.8.3 to 1.8.4
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.6.1 to 2.7.0
- Fixed setting of permissions for SQLA endpoints (#661)
- Added support for preloading of Docker images into instance pools (#663)
- Added the
databricks_userdata source (#648) - Fixed support for
spot_instance_policyin SQLA Endpoints (#665) - Added documentation for
databricks_pipelineresource (#673) - Fixed mapping for
databricks_service_principalon AWS (#656) - Made preview environment tests to run on a release basis
Updated dependency versions:
- Bump github.com/zclconf/go-cty from 1.8.2 to 1.8.3
- Bump github.com/aws/aws-sdk-go from 1.38.30 to 1.38.51
- Fixed state refresh bugs in
databricks_sql_permissions(#620, #619) - Fixed
workspace_ids_filtermapping fordatabricks_mws_log_delivery(#635) - Multiple documentation improvements (#597, eb60d10, edcd4b1, 404bdab, #615, f14b825, e615c3a, #612)
- Mounting clusters are recreated now, even when they are deleted (#637)
- Fixed handling of empty blocks for clusters/jobs/instance pools (22cdf2f)
- Mark instance pool attributes as ForceNew when it's requited (#629)
- Switched to use https://staticcheck.io/ for static code analysis (#602)
Behavior changes
- The
customer_managed_key_idfield indatabricks_mws_workspacesresource is deprecated and should be replaced withmanaged_services_customer_managed_key_id(and optionallystorage_customer_managed_key_id).databricks_mws_customer_managed_keysnow requires the parameteruse_cases(#642). If you've used the resource before, please adduse_cases = ["MANAGED_SERVICES"]to keep the behaviour.
Updated dependency versions:
- Bump github.com/aws/aws-sdk-go to v1.38.30
- Bump github.com/hashicorp/go-retryablehttp to v0.7.0
- Bump github.com/hashicorp/hcl/v2 to v2.10.0
- Bump github.com/hashicorp/terraform-plugin-sdk/v2 to v2.6.1
- Bump github.com/zclconf/go-cty to v1.8.2
- Added resources for SQL Analytics queries and dashboards:
databricks_sql_query,databricks_sql_visualization,databricks_sql_dashboard,databricks_sql_widget(#553) - Added
databricks_sql_permissionsresource (#545) - Fixed documentation bugs (#603)
- Improved resource exporter (#593)
- Added missing properties to
databricks_mws_private_access_settings(#590) - Include SQLA data source ID in
databricks_sql_endpointstate (#601) - Apply
debug_truncate_bytesalso for response dumps (#589) - More verbose logging of
databricks_clustertermination reason (#588) - Move non-auth provider config documentation into separate section (#587)
- Fixed minor issues to add support for GCP (#558)
- Fixed
databricks_permissionsfor SQL Analytics Entities (#535) - Fixed incorrect HTTP 404 handling on create (#564, #576)
- Fixed incorrect escaping of notebook names (#566)
- Fixed entitlements for databricks_group (#549)
- Fixed rate limiting to perform more than 1 request per second (#577)
- Added support for spot instances on Azure (#571)
- Added job schedules support for
pause_statusas a optional field. (#575) - Fixed minor documentation issues.
Updated dependency versions:
- Bump github.com/aws/aws-sdk-go from 1.37.20 to 1.38.10
- Bump github.com/hashicorp/hcl/v2 from 2.9.0 to 2.9.1
- Bump github.com/zclconf/go-cty from 1.8.0 to 1.8.1
- Bump github.com/google/go-querystring from 1.0.0 to 1.1.0
- Added
databricks_global_init_scriptresource to configure global init scripts (#487). - Added
databricks_sql_endpointresource (#498) - Added experimental resource exporter to generate configuration for entire workspace.
- Improved user-facing documentaiton (#508, #516, #511, #504, [#492](Update docs in various places))
- Simplified authentication issues debugging (#490)
- Made cleaner error message for no config profile (#491)
- Allow tokens without comment or expiration (#495)
- Ensured consistent slashes in notebook paths for different OSes (#500)
- Fix error message panic in command result parsing (#502)
- Updated
databricks_groupdata resource to allow non-alphanumeric characters in group name filter (#507)
Behavior changes
- Assigning any permission to
adminswould result in an error, so that behavior is consistent (#486).
Updated dependency versions:
- github.com/zclconf/go-cty from 1.2.1 to 1.7.1
- github.com/Azure/go-autorest/autorest/azure/auth from 0.5.6 to 0.5.7
- github.com/hashicorp/hcl/v2 from 2.3.0 to 2.8.2
- github.com/aws/aws-sdk-go from 1.37.1 to 1.37.11
- github.com/Azure/go-autorest/autorest from 0.11.17 to 0.11.18
- Added configurable provisioning timeout for
databricks_mws_workspaces, so that local DNS cache issues would be more tolerated. - Added [databricks_current_user] to simplify applying the same Terraform configuration by different users in the shared workspace for testing purposes.
- Added client-side rate limiting to release the pressure on backend APIs and prevent client blocking (#465)
- Member usernames, group names and instance profile names in
databricks_groupdata source are now sorted and providing consistent behavior between runs (#449) - Fixed redundant multiple mounting clusters (#445)
- Added optional parameter azure_environment to provider config which defaults to public (#437).
- Added databricks_service_principal resource.
skip_validationfromdatabricks_instance_profilewas removed and is always set totrue.- Added propagation of terraform version to
User-Agentheader, along with type of resource used. databricks_notebook&databricks_dbfs_filegot newsourcefield to specify location of a local file.databricks_notebookcan havelanguagefield optional, as long assourceis set to a file with.py,.scala,.sql, or.rextension.databricks_medata source was added to representuser_name,home&idof the caller user (or service principal).- Added validation for secret scope name in
databricks_secret,databricks_secret_scopeanddatabricks_secret_acl. Non-compliant names may cause errors. - Added databricks_spark_version data source.
- Fixed support for single node clusters support by allowing
num_workersto be0. - Fixed bug in destruction of IP access lists (#426).
- All resource imports are now making call to corresponding Databricks API by default (#471).
Behavior changes
- Removed deprecated
library_jar,library_egg,library_whl,library_pypi,library_cran, andlibrary_mavenfromdatabricks_clusteranddatabricks_jobin favor of more API-transparent library configuration block. - Removed deprecated
notebook_pathandnotebook_base_parametersfromdatabricks_jobin favor of notebook_task configuration block. - Removed deprecated
jar_uri,jar_main_class_name, andjar_parametersfromdatabricks_jobin favor of spark_jar_task configuration block. - Removed deprecated
python_fileandpython_parametersfromdatabricks_jobin favor of spark_python_task configuration block. - Removed deprecated
spark_submit_parametersfromdatabricks_jobin favor of spark_submit_task configuration block. - Removed deprecated
databricks_scim_userresource in favor of databricks_user. - Removed deprecated
databricks_scim_groupresource in favor of databricks_group. - Removed deprecated
databricks_default_user_rolesdata source in favor of databricks_group data source. - Removed deprecated
basic_authandazure_authprovider configuration blocks in favor of documented authentication methods. format,overwrite, andmkdirswere removed fromdatabricks_notebook. To follow expected behavior of Terraform, notebooks are always overwritten.skip_validationfromdatabricks_instance_profilewas removed and is always set totruefor subsequent requests.databricks_mws_workspacegotverify_workspace_runnningremoved and now validates all every deployment. In case deployment failed, it removes workspace that failed and returns error message with explanation.default_tagswere removed fromdatabricks_instance_pool.disk_specgot new attributedisk_type, that containsazure_disk_volume_typeandebs_volume_type. This change is made to closer reflect API structure.databricks_notebook&databricks_dbfs_filegotcontentattribute renamed tocontent_base64and now share the same logic to work with local files.
- Fixed documentation issues.
- Added missing resource importers and test to cover it.
- Migrated build from TravisCI to GitHub Actions.
- Fixed custom
config_fileissue configuration handling (#420).
Deprecations
databricks_notebookhas gotoverwrite,mkdirsandformatparameters, that always have to be set to certain values in order to follow expected behavior of terraform. These fields would be removed in 0.3 and always set to proper values.databricks_notebook&databricks_dbfs_filefieldcontentis deprecated and would be renamed tocontent_base64to further increase clarity.databricks_dbfs_filehas gotcontent,content_b64_md5,overwrite,mkdirs,validate_remote_filefields deprecated and they will be removed in the next version, where critical code path will be shared withdatabricks_notebook.network_error_messagesandverify_workspace_runnningfromdatabricks_mws_workspacesis deprecated and going to be removed in 0.3.error_messagesfromdatabricks_mws_networksare deprecated and would be removed in 0.3.ebs_volume_typeandazure_disk_volume_typefromdatabricks_instance_poolis going to be moved todisk_typesub-block in 0.3, which means you'll slightly have to modify configuration while migrating to 0.3. Computed fielddefault_tagsis going to be removed from resource. This is done to further increase maintainability of provider in the future.
Updated dependency versions:
- github.com/aws/aws-sdk-go 35.36
- github.com/hashicorp/go-retryablehttp 0.6.8
- github.com/Azure/go-autorest/autorest 0.11.12
Behavior changes
min_idle_instancesfordatabricks_instance_poolis now optional.skip_validationfordatabricks_instance_profileis going to be removed in 0.3.
- Added databricks_workspace_conf resource.
- Added databricks_mws_log_delivery resource for billable usage & audit logs consumption.
- Added databricks_node_type data source for simpler selection of node types across AWS & Azure.
- Added Azure Key Vault support for
databricks_secret_scopefor Azure CLI authenticated users. - Added is_pinned support for
databricks_clusterresource. - Fixed restarting cluster on changes in cluster configuration aren't related to the cluster configuration (issue #379)
- Fixed issue #383 by cleaning up clusters that fail to start.
- Fixed issue #382 by ignoring any incoming changes to deployment name of
databricks_mws_workspaces, as well as propagating the right error messages. - Internal: API for retrieval of the cluster events.
- Increased code coverage to 71%.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.9
- github.com/Azure/go-autorest/autorest/adal v0.9.5
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.3
- github.com/Azure/go-autorest/autorest/azure/cli v0.4.2
- gopkg.in/ini.v1 1.62.0
Deprecations
network_error_messagesfromdatabricks_mws_workspacesis deprecated and going to be removed in 0.3
- Small fixes
- Added support for customer managed keys for Accounts API.
- Added
databricks_userresource. - Added
databricks_user_instance_profileresource. - Added
databricks_groupdata source.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.6
- github.com/Azure/go-autorest/autorest/adal v0.9.4
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.2
- github.com/Azure/go-autorest/autorest/azure/cli v0.4.1
- gopkg.in/ini.v1 v1.61.0
Deprecations
databricks_scim_useris no longer receiving fixes and will be removed in0.3, please rewrite using thedatabricks_userresource, which has more consistent semantics withdatabricks_groupand works better with identity provider SCIM sync.databricks_scim_groupis no longer receiving fixes and will be removed in0.3. Please rewrite using thedatabricks_groupresource.databricks_default_user_rolesis no longer receiving fixes and will be removed in0.3, please rewrite usingdatabricks_user&databricks_groupresources.
Behavior changes
- State changes to legacy
spark.databricks.delta.preview.enabledconfig option are now ignored bydatabricks_job&databricks_cluster - Libraries, which are installed on all clusters and are not part of cluster resource definition, won't be waited for INSTALLED status
- Fixed "Secret scope ACL is MANAGE for all users by default" (issue 322). If you were relying on setting
MANAGEpermission to all users by default, you need to addinitial_manage_principal = "users"to yourresource "databricks_secret_scope"declaration.
- Added support for local disk encryption
- Added more reliable indication about Azure environment and fixed azure auth issue for Terraform 0.13
- Updated databricks_aws_crossaccount_policy to latest rules
- Fixed missing importers for databricks_scim_* resources
- Updated Terraform Plugin SDK to latest version along with transitive dependencies.
- Added support disclaimers
- Increased code coverage to 65%
- Added Azure CLI authentication to bridge the gap of local development workflows and let more people use the provider.
- All authentication is completely lazy-initialized, which makes it provider overall more stable.
- Significantly increased unit test coverage, which runs before every merge of a pull request.
- Introduced constantly running integration test suite environments: azsp, azcli & awsmt
- Numerous stability improvements for clusters, mounts, libraries, notebooks, files, authentication and TLS connectivity.
- Added ability to mount storage without explicitly defining a cluster, though it will still launch auto-terminating
terraform-mountcluster to perform the mount. databricks_cluster&databricks_jobnow share significant portion of configuration wiring code, therefore increasing the stability of codebase.- Added support for Terraform 0.13 local builds for those who develop or cannot wait for next release.
- Added AWS IAM Policy data helpers to simplify new deployments.
- Migrated all documentation to Terraform Registry format, therefore having a single always-accurate place for end-user guides.
- Internally, codebase has been split into multiple packages, which should make further contributions simpler.
Updated dependency versions:
- github.com/Azure/go-autorest/autorest v0.11.4
- github.com/Azure/go-autorest/autorest/adal v0.9.2
- github.com/Azure/go-autorest/autorest/azure/auth v0.5.1
- github.com/aws/aws-sdk-go v1.34.13
- gopkg.in/ini.v1 v1.60.2
Deprecations
library_*is no longer receiving fixes and will be removed in0.3, please rewrite cluster & job resources to uselibraryconfiguration block.basic_authprovider block is no longer receiving fixesand will be removed in0.3, please useusernameandpasswordoptionsazure_authprovider block is no longer receiving fixesand will be removed in0.3, please useazure_*options
Behavior changes
- Previously, mounts code paths were different functions. This release unifies them to be a single testable codebase with different configuration options & re-use of the critical code paths. For maintainability reasons, there's no longer check performed on container & storage account names, but rather on high-level mount source uri.