Hi team, thank you for the development!
When I persist Lance dataset from Spark, I lose a small portion of data. I found out that the options below cause the loss.
.option("max_bytes_per_file", str(512 * 1024 * 1024))
.option("use_queued_write_buffer", "true")
.option("queue_depth", "4")When I removed them as below, the loss was fixed.
I am not fully sure if this is expected or not, but I hope this helps others.
Hi team, thank you for the development!
When I persist Lance dataset from Spark, I lose a small portion of data. I found out that the options below cause the loss.
When I removed them as below, the loss was fixed.
I am not fully sure if this is expected or not, but I hope this helps others.