[Feature] Use Livy high concurrency (HC) to support parallel model execution

[cheyney-w]

Is this your first time submitting a feature request?

• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

High concurrency support for Fabric Livy was announced last week. It would be great if this adapter could take advantage of it!

In the current version of the adapter (v1.11.0), if you set your profile threads greater than 1, dbt will send multiple model execution statements to the Livy session concurrently, but the session will execute them serially. This means that throughput when threads = 1 is not much different than when threads > 1, even if your models don't ref each other.

Describe alternatives you've considered

No response

Who will this benefit?

Anyone hoping to improve the throughput of their dbt runs by adjusting their profile threads setting.

Are you interested in contributing this feature?

No response

Anything else?

No response

[mdrakiburrahman]

Hey @cheyney-w,

dbt will send multiple model execution statements to the Livy session concurrently, but the session will execute them serially

Are you sure this is the case? Did you test it?

My understanding was, when I run 12 spark sql commands via 12 dbt threads, Livy goes to the spark session and fires that Spark starts evaluating, if your Spark driver has > 12 cores.

I'm happy to share REST API commands that shows this, the API shows you can fire many queries against the SAME non-HC Livy session at the same time, I was not aware that these are serially executed.

My understanding was, the High Concurrency feature allows multiple isolated dbt models to take advantage of the same livy session, it's a completely different use case.

The way this adapter as of today achieves the same outcome between dbt stages is by using the .txt file so we can reuse Livy Sessions across dbt command boundaries so we don't pay for a new Livy every time.

(I'm happy to add support to this adapter for new things, writing code is cheap, but I want to be crystal clear on what problem we're trying to solve first, if there's an existing stable way to do something then we should not adopt something new that could cause instability)

Please note, I read through the HC docs, but I'm not convinced I understand how it applies to this dbt adapter if we already get parallel SQL executions without it in the same livy session.

[mdrakiburrahman]

E.g. here on the existing adapter.

Firing SQL models concurrently:

Spark running them concurrently too:

---

So we see above 6 heavy queries running in parallel.

So in this scenario, what would HC enable exactly?

[cheyney-w]

Interesting, I see different behaviour!

Here's what I did:

1. Created 4 expensive models that take a long time to run but don't depend on each other.
   • For example, 4 copies of this:

     {{ config(materialized='table') }}
     select 
         a.id as a_id
         , b.id as b_id
     from range(50000) as a
     cross join range(1,50000) as b

2. Run dbt with 4 threads
3. Observe that all four models are started at the same time (expected)

19:36:54  Concurrency: 4 threads (target='fabricspark-dev')
19:36:54  
19:36:54  Microsoft Fabric-Spark adapter: Using CLI auth
19:39:57  1 of 4 START sql table model dbt.high_volume_1 ................................. [RUN]
19:39:57  2 of 4 START sql table model dbt.high_volume_2 ................................. [RUN]
19:39:57  3 of 4 START sql table model dbt.high_volume_3 ................................. [RUN]
19:39:57  4 of 4 START sql table model dbt.high_volume_4 ................................. [RUN]

4. Monitor the sessions statements using the REST endpoint. See that one statement is running and 3 are waiting:

5. After the first model is done, check the list of statements again. Now first one is done, second one is running, and third and fourth are still waiting:

6. After all four models are done, observe that the stdout runtimes of each are very different, even though they are identical. Second model takes approx twice as long as the first, third model takes approx. three times as long as the first, fourth model takes approx. four times as long as the first.

19:36:54  Concurrency: 4 threads (target='fabricspark-dev')
19:36:54  
19:36:54  Microsoft Fabric-Spark adapter: Using CLI auth
19:39:57  1 of 4 START sql table model dbt.high_volume_1 ................................. [RUN]
19:39:57  2 of 4 START sql table model dbt.high_volume_2 ................................. [RUN]
19:39:57  3 of 4 START sql table model dbt.high_volume_3 ................................. [RUN]
19:39:57  4 of 4 START sql table model dbt.high_volume_4 ................................. [RUN]
19:42:30  1 of 4 OK created sql table model dbt.high_volume_1 ............................ [OK in 153.22s]
19:44:34  2 of 4 OK created sql table model dbt.high_volume_2 ............................ [OK in 277.70s]
19:46:23  4 of 4 OK created sql table model dbt.high_volume_4 ............................ [OK in 386.73s]
19:48:24  3 of 4 OK created sql table model dbt.high_volume_3 ............................ [OK in 507.10s]

7. Looking at Spark Jobs UI, I see they all seem to run in sequence, not in parallel:

This was using a Fabric Spark session with this config:

"driverMemory": "56g",
"driverCores": 8,
"executorMemory": "56g",
"executorCores": 8,
"numExecutors": 1

[mdrakiburrahman]

Can you bump up your min Executors to 8 (so it starts with 8 - I understand it's wasteful, but just try for this experiment)?

I think you're saturated by the # cores your Spark Cluster has, so the Driver isn't scheduling work because the existing executor has all threads pre-occupied from the other query.

This is not related to Livy, regular old Spark JAR you'll see this behavior

[cheyney-w]

OK, I've ran my test again with a session with this configuration:

"livyInfo": {
    "startingAt": "2026-05-16T20:15:06.957328+00:00",
    "idleAt": "2026-05-16T20:15:47.2658547+00:00",
    "busyAt": "2026-05-16T20:15:52.3284091+00:00",
    "currentState": "idle",
    "jobCreationRequest": {
      "name": "dbt_project",
      "conf": {
        "spark.livy.synapse.sql.displayFormatter.enabled": "true",
        "spark.onesecurity.systemcontext.access.token.table.impl": "<redacted>",
        "spark.onesecurity.systemcontext.access.token.query.impl": "<redacted>",
        "spark.onelake.security.enabled": "true",
        "spark.gluten.olcsdk.enabled": "true",
        "spark.dynamicAllocation.enabled": "true",
        "spark.dynamicAllocation.minExecutors": "8",
        "spark.dynamicAllocation.maxExecutors": "9",
        "spark.livy.session.idle.timeout": "30m",
        "spark.openlineage.transport.sparkcore_ingestion_enable": "false",
        "spark.hadoop.fs.defaultFS": "abfss://*******************@onelake.dfs.fabric.microsoft.com/",
        "spark.hadoop.fs.homeDir": "/*******************",
        "spark.sql.externalCatalogImplementation": "com.microsoft.fabric.spark.catalog.InstrumentedExternalCatalog",
        "spark.livy.synapse.session-warmup.enabled": "false"
      },
      "driverMemory": "56g",
      "driverCores": 8,
      "executorMemory": "56g",
      "executorCores": 8,
      "numExecutors": 8
    }
  }

And the Spark job monitor still shows sequential execution (albeit faster):

[mdrakiburrahman]

This is interesting, I'm using the latest released version of the adapter, I wonder what I'm doing differently.

I think figuring out this discrepancy is more important than adding a new API dependency. I am hesitant to regress stability of this adapter by depending on something new, unless we get tangible benefits and perf improvements.

@cheyney-w -

Similar to last time, are you able to share a dummy model/spark-env repro?

To make the models run long, if you share some regular PySpark code that inserts a bunch of random data into the source, the model should run slow.

After that, I'll give your model a try, and see if there's a setting to add to your model to unblock parallelism.

[cheyney-w]

Similar to last time, are you able to share a dummy model/spark-env repro?

@mdrakiburrahman
I've committed some changes to this repo to support reproduction of this issue: https://github.com/cheyney-w/dbt-fabricspark-cross-workspace-demo

1. Clone the repo
2. Create the following in your Fabric tenant:
   • Workspace named MainWorkspace
   • Schema-enabled lakehouse in MainWorkspace called MainLakehouse
3. pip install -r requirements.txt
4. Set the following environment variables:
   • DBT_WORKSPACE_ID=<ID of MainWorkspace>
   • DBT_LAKEHOUSE_ID=<ID of of MainLakehouse>
5. Run dbt run -s high_volume_1 high_volume_2 high_volume_3 high_volume_4

[mdrakiburrahman]

Got a repro:

[mdrakiburrahman]

Ah @cheyney-w I think you're right. In my above screenshot's case, a specific SQL model caused multiple Spark...job (the job API in Spark) to fire, but I mistakenly thought this was concurrent queries.

Excellent work, thank you for identifying this feature 🙂

I'll have a PR out ASAP after testing and release it.
This is going to make our production models run SIGNIFICANTLY faster, I hope!

[mdrakiburrahman]

Yup - HC API works!

---

Going to test my PR branch in our production model - if green, will release soon!

This is a very happy day for me, I've been trying to reduce our dbt model runtime but I did not realize Livy was the bottleneck 🙂