Skip to content

Spark test coupling #2906

@ADBond

Description

@ADBond

tests/test_full_example_spark.py::test_full_example_spark[delta_lake_table] fails when i run all the tests locally, due to a path already existing. This doesn't happen if I run this test module in isolation, which suggests it's down to some coupling - I haven't investigated why.

Error is:

pyspark.errors.exceptions.captured.SparkRuntimeException: [LOCATION_ALREADY_EXISTS] Cannot name the managed table as `spark_catalog`.`1111`.`__splink__df_tf_city_cde02bf80`, as its associated location 'file:/path/to/repo/splink/spark-warehouse/1111.db/__splink__df_tf_city_cde02bf80' already exists. Please pick a different table name, or remove the existing location first. SQLSTATE: 42710

.venv/lib/python3.13/site-packages/pyspark/errors/exceptions/captured.py:269: SparkRuntimeException

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions