Race condition in Go's SDK

[ItamarYuran]

Race condition between DeleteByFullName and ExecuteAndWait CREATE EXTERNAL TABLE

Description

There is a race condition in Unity Catalog when using Tables.DeleteByFullName() followed immediately by StatementExecution.ExecuteAndWait() to recreate an external table. The DELETE returns successfully, but the subsequent CREATE fails with TABLE_OR_VIEW_ALREADY_EXISTS or LOCATION_OVERLAP errors, indicating the deletion hasn't propagated through Unity Catalog's metadata system.

Expected Behavior

After Tables.DeleteByFullName(ctx, fullTableName) returns without error, the table metadata should be fully removed from Unity Catalog, and a subsequent CREATE EXTERNAL TABLE via StatementExecution.ExecuteAndWait() should succeed immediately.

Actual Behavior

DeleteByFullName() returns success (no error), but an immediate CREATE EXTERNAL TABLE via StatementExecution.ExecuteAndWait() fails intermittently (occasionally in production, ~40-50% of the time in stress testing) with:

1. TABLE_OR_VIEW_ALREADY_EXISTS: Cannot create table or view because it already exists
2. LOCATION_OVERLAP: Input path url overlaps with other external tables or volumes. Conflicting tables/volumes: [the table we just deleted]

This indicates eventual consistency between Unity Catalog API metadata and the SQL execution engine.

Steps to Reproduce

func main() {
	w, _ := databricks.NewWorkspaceClient(&databricks.Config{
		Host:        "https://your-instance.azuredatabricks.net",
		Token:       "your-token",
		Credentials: config.PatCredentials{},
	})

	ctx := context.Background()
	fullTableName := "main.test_schema.my_external_table"
	
	for i := 0; i < 100; i++ {
		// Delete via Unity Catalog API
		if i > 0 {
			w.Tables.DeleteByFullName(ctx, fullTableName) // Returns success
		}

		// Immediately CREATE via SQL - fails ~50% of iterations
		_, err := w.StatementExecution.ExecuteAndWait(ctx, sql.ExecuteStatementRequest{
			WarehouseId: "your-warehouse-id",
			Statement:   "CREATE EXTERNAL TABLE my_external_table LOCATION 'abfss://...'",
			Catalog:     "main",
			Schema:      "test_schema",
		})
		
		if err != nil {
			fmt.Printf("Iteration %d FAILED: %v\n", i+1, err)
		}
	}
}

Actual Output

Iteration | Drop | Create | Statement Status | StatementID Columns
1 N/A OK SUCCEEDED 01f0ef0d-07cf-159b-a64b-287ac005b073 0

2 FAILED FAILED ERROR FAILED: BAD_REQUEST [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view main.test_schema.my_external_table because it already exists.
Choose a different name, drop the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, add the OR REPLACE clause to replace the existing materialized view, or add the OR REFRESH clause to refresh the existing streaming table. SQLSTATE: 42P07

3 FAILED FAILED ERROR FAILED: BAD_REQUEST [RequestId=2d4ad42e-c9a9-4d97-a4f5-1c64855bc743 ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://...core.windows.net/asdasd/test-data-20260101_132258' overlaps with other external tables or volumes within 'CreateTable' call. Conflicting tables/volumes: main.test_schema.my_external_table.

4 FAILED FAILED ERROR FAILED: BAD_REQUEST [RequestId=7b616746-ca32-4d51-8c69-166c784727b4 ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://...core.windows.net/asdasd/test-data-20260101_132258' overlaps with other external tables or volumes within 'CreateTable' call. Conflicting tables/volumes: main.test_schema.my_external_table.

5 OK OK SUCCEEDED 01f0ef0d-0a29-1c09-8a84-59ec9978093b 0

6 FAILED FAILED ERROR FAILED: BAD_REQUEST [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view main.test_schema.my_external_table because it already exists.
Choose a different name, drop the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, add the OR REPLACE clause to replace the existing materialized view, or add the OR REFRESH clause to refresh the existing streaming table. SQLSTATE: 42P07

7 FAILED FAILED ERROR FAILED: BAD_REQUEST [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view main.test_schema.my_external_ta

Environment

SDK Version: github.com/databricks/databricks-sdk-go v0.26.2
Databricks Platform: Azure Databricks with Unity Catalog
Use Case: Automated external table registration from lakeFS export hooks

Additional Context

This issue is particularly problematic for automated workflows (CI/CD, export hooks) that need to reliably recreate tables. The intermittent failures require complex retry logic and significantly slow down processing.
Related Issues
This may be related to the broader Unity Catalog eventual consistency model, but the SDK should abstract this away from users by not returning success until operations are truly complete.