dbt pyspark model in wsl local environment

[rarpal]

I have setup pyspark in wsl (windows 11) with hive support and metastore enabled and working with spark-sql which is working as expected. I have now installed dbt-spark with dbt-spark[session] adaptor. What I want is to be able to create python dbt models with the following configuration in my profiles.yml:

local_spark:
  target: dev
  outputs:
    dev:
      type: spark
      host: local
      method: session
      schema: default
      file_format: delta

As you can see I am trying to run my python model using an active local spark session (method:session).
I run my python model via a entry point python script which creates a PySpark session before calling dbtRunner dbt.invoke method as shown below:

    # Initialize the dbtRunner
    dbt = dbtRunner()

    # Define the dbt command to run, for example: `dbt run --models my_model`
    cli_args = ["run", "--select", "stage", "--project-dir", "<my project folder>", "--profiles-dir", "<my profile folder>"]

    # Use the invoke method to run the command
    result = dbt.invoke(cli_args)

However when I run this I get the following error:
Databricks cluster_id is required for all_purpose_cluster submission method with running with notebook.

Even though I am not using the dbt-databricks adaptor and have requested the method:session

Does anyone know what the issue is here and If I am missing some config ?

[apoorva-01]

Short version: dbt-spark can't run Python models against a local method: session Spark. The Python-model submission path is Databricks-only, and has been since it shipped.

When dbt-spark first got Python models it leaned on a Databricks-specific API to push the Python out to a cluster, and nobody's built the open-source Spark equivalent since. The docs line up with that: the only platforms that list Python model support are Snowflake, BigQuery, and Databricks. So your SQL models will run fine over method: session (that's exactly what the session connection is for), but a .py model won't, no matter how the profile is set up.

If you actually need the PySpark transform, two ways out that work today:

• Run it as a plain PySpark script outside dbt against your local session, write the result to a table, and register that table as a source your downstream dbt models read from. On a local WSL setup this is usually the least painful.
• Or point dbt at a real Databricks connection, where the Python path is wired up.

Not the answer you were hoping for, but it saves you chasing a profile combo that can't exist yet.