Migrate integration tests to use Databricks clusters instead of Spark local mode

[m-abulazm]

Originally posted by @sundarshankar89 in #2223 (comment)

Motivation

Currently, most integration tests execute using Spark local mode, which adds overhead of running spark and limits our ability to validate Databricks-specific features and creates environment parity gaps between test and production.

Solution

• Get rid of local spark in code and infra
• Replace with spark fixture from pytester. this should be a drop-in replacement