Bug: tables written to Databricks are unusable if any columns contain only `NA_character_`

[wurli]

This is a weird one which took a while to get to the root cause of!

Reproducible like this:

sc <- spark_connect(method = "databricks_connect", serverless = TRUE, version = "16.1")

df <- tibble(x = NA_character_)
tbl_name <- "x.y.z"
spark_df <- sparklyr::copy_to(sc, df, tbl_name)
spark_write_table(spark_df, tbl_name, "overwrite")

Strangely, the data shows up in the Databricks GUI with Type 'void':

But when I try and check the Sample Data I get an error, and I'm also unable to access the data through normal pyspark methods:

I don't think this issue manifests with other datatypes, e.g. plain old NA (logical).

[edgararuiz]

Hi @wurli, does you table contain NA_characters, or is this a one row table with only NA_character? Because I tested with more than one row that is not NA and it worked. I think bottom line, NA_character is translated as NA, so there is no way for the database to know what the actual field type it is if there is only one row.

library(sparklyr)
sc <- spark_connect(method = "databricks_connect", serverless = TRUE, version = "16.1")
#> ℹ
#> ✔ Python environment: 'Managed `uv` environment' [1.1s]
#> 
#> ℹ Connecting to serverless
#> ✔ Connected to: 'serverless' [8ms]
#> 
df <- tibble::tibble(x = c(NA_character_, "b")) # Added a non NA 
tbl_name <- "test" # Changed to test
spark_df <- copy_to(sc, df, tbl_name)
spark_write_table(spark_df, tbl_name, "overwrite")
dplyr::tbl(sc, tbl_name)
#> # Source:   table<`test`> [?? x 1]
#> # Database: spark_connection
#>   x    
#>   <chr>
#> 1 <NA> 
#> 2 b
packageVersion("sparklyr")
#> [1] '1.9.1'
packageVersion("pysparklyr")
#> [1] '0.1.8'
spark_disconnect(sc)

[wurli]

Hi, thanks for checking this out!

I agree with your assessment, it seems to be a failure to determine the column's datatype when all values are missing, as I also found that a single non-missing value fixes the issue.

I guess this may be a deeper limitation of databricks/spark, so perhaps it would be just running a check on the R side and emitting a warning/error so other users don't get bitten by this?

Many thanks!

Jacob