fix: move livy session cleanup into dbt connection lifecycle

[Copilot]

Note

Thank you for making change! Please fill this template for your pull request to improve quality of check-in message.

Tip

This repo uses Conventional Commit conventions - please try to rename your PR headline to match it.

Warning

Please ensure to read through this whole set of instructions, specially the Test section.

Why this change is needed

Livy session cleanup was implemented via module-level atexit handlers, bypassing dbt’s adapter connection lifecycle (open/close).
That made cleanup execution non-observable to dbt and decoupled failure handling from normal adapter control flow.

Repro context: importing adapter modules registered process-exit cleanup handlers in both singleton and high-concurrency Livy paths, including a global HC active-session registry used only for atexit cleanup.

How

• Lifecycle ownership moved to connection manager

  • Cleanup now runs through FabricSparkConnectionManager.close() / cleanup_all() via manager.disconnect().
  • Connection wrappers now carry manager metadata so close can deterministically disconnect the owning manager.

• Out-of-band cleanup removed

  • Removed module-level atexit.register(...) cleanup from:
    • singleton_livy.py
    • concurrent_livy.py
  • Removed HC global _active_sessions state used exclusively for process-exit cleanup.

• Behavioral intent

  • Session teardown is now aligned with dbt-managed connection shutdown paths instead of Python runtime exit hooks.

# connections.py (shape of change)
handle.manager = manager
handle.thread_id = thread_id

connection = super().close(connection)
if connection.handle.manager is not None:
    connection.handle.manager.disconnect()

Test

Important: Non-Microsoft Employee contributors

If you are not a Microsoft employee with a foo@microsoft.com email, you will not be able to run CI as it runs in the @microsoft.com Fabric Tenant where you do not have access.

In order for your PR to be considered for review, you must attach a clear screenshot of the output of you running the following command successfully:

npx nx run dbt-fabricspark:test --output-style=stream

Here's an example of a successful run:

⚠️ Delete the above image and attach your own screenshot

To keep the quality of the repo high, if you do not attach a screenshot of successful local testing, your PR will be promptly closed.

Microsoft Employee contributors

Your PR will be subjected to full regression suite via GitHub Action.
It's highly recommended to run the tests locally so your contributions are promptly merged rather than failing in CI.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• 1.0.0.127.in-addr.arpa
  • Triggering command: /opt/mssql/bin/sqlservr (dns block)
• api.fabric.microsoft.com
  • Triggering command: /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/.venv/bin/pytest pytest tests/unit -vv (dns block)
  • Triggering command: /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/.venv/bin/pytest pytest tests/unit -vv git conf�� it} --global 150e59783fe3f3304c4f938035e26f5ff2f52381c9b7e88ef47/log.json 4c4f938035e26f5ff2f52381c9b7e88ef47/81c35b081dec6a64bb9dd1071fc171165c6a1ba4abb99ea305abd42dc71f/var/run/docker/runtime-runc/moby (dns block)
  • Triggering command: /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/.venv/bin/pytest pytest tests/unit -vv --detach --pid-file /run/containerd/io.containerd.runtime.v2.task/moby/82e1acda59d9c150e59783fe3f3304c4f938035e26f5ff2f52381c9b7e88ef47/7a9a7c3336ee730ea72bc2b2487f434ef57b1898f550741eefee7ddd5a07dbt-fabricspark:test:unit 82e1acda59d9c150e59783fe3f3304c4f938035e26f5ff2f52381c9b7e88ef47 150e59783fe3f3304c4f938035e26f5ff2f52381c9b7e88ef47/log.json (dns block)
• metadata.google.internal
  • Triggering command: /opt/mssql/bin/sqlservr (dns block)
• settings-win.data.microsoft.com
  • Triggering command: /opt/mssql/bin/sqlservr (dns block)
• vortex.data.microsoft.com
  • Triggering command: /opt/mssql/bin/sqlservr (dns block)
• www.google-analytics.com
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node node /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/node_modules/.bin/nx run dbt-fabricspark:lint (dns block)
  • Triggering command: /opt/hostedtoolcache/node/24.14.1/x64/bin/node /opt/hostedtoolcache/node/24.14.1/x64/bin/node /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker /tmp/plugin5593-2-361.370401.sock /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/node_modules/nx/src/plugins/project-json/build-nodes/project-json (dns block)
  • Triggering command: /opt/hostedtoolcache/node/24.14.1/x64/bin/node /opt/hostedtoolcache/node/24.14.1/x64/bin/node /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker /tmp/plugin5593-1-358.056406.sock /home/REDACTED/work/dbt-fabricspark/dbt-fabricspark/node_modules/nx/src/plugins/package-json (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)