Gate Databricks CPU and GPU tests independently

## Summary
Classify changed paths against the actual Databricks CPU and GPU runtime surfaces, emit separate fail-open decisions, and gate each matrix leg independently.

## Prompting Intent
The engineer asked to determine exactly when Databricks tests should run, lock down the path rules, and deliver the work as a stacked pull request above PR #2581.

## Linked Sources
- Base CI hardening PR: #2581
- GitHub stacked PR documentation: https://docs.github.com/en/pull-requests/how-tos/create-pull-requests/creating-stacked-pull-requests
- ADO timing audit: build 229176406

## Rationale
CPU and GPU decisions are separated because most module changes cannot affect the expensive GPU notebooks. Unknown paths and shared build or test infrastructure remain fail-open, while explicit test-only and unrelated tooling paths skip safely. This preserves coverage while avoiding unrelated GPU capacity waits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: ranadeepsingh

pipeline.yaml

@@ -106,7 +106,8 @@ jobs:
       fetchDepth: 1
     - bash: |
         set -uo pipefail
-        run_databricks=true
+        run_databricks_cpu=true
+        run_databricks_gpu=true
 
         if [ "$(Build.Reason)" = "PullRequest" ]; then
           target_ref="${SYSTEM_PULLREQUEST_TARGETBRANCH:-}"
@@ -119,15 +120,27 @@ jobs:
             # effective PR diff. A source-only checkout can only over-report
             # target changes, which safely keeps Databricks enabled.
             echo "Changed paths used for Databricks E2E impact detection:"
-            git diff --name-only --diff-filter=ACMRD "$target_commit" HEAD | sed 's/^/  /'
-            decision="$(
-              git diff --name-only -z --diff-filter=ACMRD "$target_commit" HEAD |
-                python3 tools/ci/databricks_impact.py --null
+            changed_paths_file="$(mktemp)"
+            trap 'rm -f "$changed_paths_file"' EXIT
+            git diff --name-only -z --diff-filter=ACMRD "$target_commit" HEAD > "$changed_paths_file"
+            tr '\0' '\n' < "$changed_paths_file" | sed 's/^/  /'
+
+            cpu_decision="$(
+              python3 tools/ci/databricks_impact.py --null --suite cpu < "$changed_paths_file"
+            )"
+            gpu_decision="$(
+              python3 tools/ci/databricks_impact.py --null --suite gpu < "$changed_paths_file"
             )"
-            case "$decision" in
-              true|false) run_databricks="$decision" ;;
+            case "$cpu_decision" in
+              true|false) run_databricks_cpu="$cpu_decision" ;;
+              *)
+                echo "##vso[task.logissue type=warning]Invalid Databricks CPU impact result; running CPU E2E"
+                ;;
+            esac
+            case "$gpu_decision" in
+              true|false) run_databricks_gpu="$gpu_decision" ;;
               *)
-                echo "##vso[task.logissue type=warning]Invalid Databricks impact result; running E2E"
+                echo "##vso[task.logissue type=warning]Invalid Databricks GPU impact result; running GPU E2E"
                 ;;
             esac
           else
@@ -137,8 +150,10 @@ jobs:
           echo "Non-PR build; Databricks E2E remains enabled"
         fi
 
-        echo "Databricks E2E enabled: $run_databricks"
-        echo "##vso[task.setvariable variable=runDatabricksE2E;isOutput=true]$run_databricks"
+        echo "Databricks CPU E2E enabled: $run_databricks_cpu"
+        echo "Databricks GPU E2E enabled: $run_databricks_gpu"
+        echo "##vso[task.setvariable variable=runDatabricksCpuE2E;isOutput=true]$run_databricks_cpu"
+        echo "##vso[task.setvariable variable=runDatabricksGpuE2E;isOutput=true]$run_databricks_gpu"
       name: detectDatabricksImpact
       displayName: 'Detect Databricks E2E impact'
     - template: templates/update_cli.yml
@@ -226,14 +241,15 @@ jobs:
         condition: and(succeeded(), eq(variables.isMaster, true))
         displayName: Publish Badges
 
-- job: DatabricksE2E
+- job: DatabricksCPUE2E
   dependsOn: BuildAndCacheSbt
-  displayName: 'Databricks E2E'
+  displayName: 'Databricks CPU E2E'
   condition: >-
     and(
       succeeded(),
+      eq(variables.runTests, 'True'),
       eq('${{ parameters.testDatabricksE2E }}', true),
-      eq(dependencies.BuildAndCacheSbt.outputs['detectDatabricksImpact.runDatabricksE2E'], 'true')
+      eq(dependencies.BuildAndCacheSbt.outputs['detectDatabricksImpact.runDatabricksCpuE2E'], 'true')
     )
   timeoutInMinutes: 300
   cancelTimeoutInMinutes: 0
@@ -255,35 +271,31 @@ jobs:
         TEST-CLASS: "com.microsoft.azure.synapse.ml.nbtest.DatabricksCPUTests4"
       databricks-cpu-5:
         TEST-CLASS: "com.microsoft.azure.synapse.ml.nbtest.DatabricksCPUTests5"
-      databricks-gpu:
-        TEST-CLASS: "com.microsoft.azure.synapse.ml.nbtest.DatabricksGPUTests"
-#      databricks-rapids tests have been disabled because these tests are failing.
-#      This test will be re-enabled once the issue is fixed.
-#      databricks-rapids:
-#        TEST-CLASS: "com.microsoft.azure.synapse.ml.nbtest.DatabricksRapidsTests"
   steps:
-    - template: templates/sbt_cache.yml
-    - template: templates/update_cli.yml
-    - template: templates/conda.yml
-    - template: templates/kv.yml
-    - template: templates/publish.yml
-    - task: AzureCLI@2
-      displayName: 'E2E'
-      inputs:
-        azureSubscription:  'SynapseML Build'
-        scriptLocation: inlineScript
-        scriptType: bash
-        inlineScript: |
-          set -e
-          source activate synapseml
-          sbt "testOnly $(TEST-CLASS)"
-      condition: and(succeeded(), eq(variables.runTests, 'True'))
-    - task: PublishTestResults@2
-      displayName: 'Publish Test Results'
-      inputs:
-        testResultsFiles: '**/test-reports/TEST-*.xml'
-        failTaskOnFailedTests: true
-      condition: and(eq(variables.runTests, 'True'), succeededOrFailed())
+    - template: templates/databricks_e2e_steps.yml
+
+- job: DatabricksGPUE2E
+  dependsOn: BuildAndCacheSbt
+  displayName: 'Databricks GPU E2E'
+  condition: >-
+    and(
+      succeeded(),
+      eq(variables.runTests, 'True'),
+      eq('${{ parameters.testDatabricksE2E }}', true),
+      eq(dependencies.BuildAndCacheSbt.outputs['detectDatabricksImpact.runDatabricksGpuE2E'], 'true')
+    )
+  timeoutInMinutes: 300
+  cancelTimeoutInMinutes: 0
+  pool:
+    vmImage: $(UBUNTU_VERSION)
+  variables:
+    TEST-CLASS: "com.microsoft.azure.synapse.ml.nbtest.DatabricksGPUTests"
+    MML_ADB_AUTH_TYPE: "aad"
+    MML_ADB_WORKSPACE_HOST: "adb-1885762835647850.10.azuredatabricks.net"
+    MML_ADB_WORKSPACE_RESOURCE_ID: "/subscriptions/e342c2c0-f844-4b18-9208-52c8c234c30e/resourceGroups/marhamil-mmlspark/providers/Microsoft.Databricks/workspaces/synapseml-build-adb"
+# The DatabricksRapidsTests suite remains disabled until its existing failures are fixed.
+  steps:
+    - template: templates/databricks_e2e_steps.yml
 
 - job: FabricE2E
   dependsOn: BuildAndCacheSbt

templates/databricks_e2e_steps.yml

@@ -0,0 +1,23 @@
+steps:
+  - template: sbt_cache.yml
+  - template: update_cli.yml
+  - template: conda.yml
+  - template: kv.yml
+  - template: publish.yml
+  - task: AzureCLI@2
+    displayName: 'E2E'
+    inputs:
+      azureSubscription: 'SynapseML Build'
+      scriptLocation: inlineScript
+      scriptType: bash
+      inlineScript: |
+        set -e
+        source activate synapseml
+        sbt "testOnly $(TEST-CLASS)"
+    condition: and(succeeded(), eq(variables.runTests, 'True'))
+  - task: PublishTestResults@2
+    displayName: 'Publish Test Results'
+    inputs:
+      testResultsFiles: '**/test-reports/TEST-*.xml'
+      failTaskOnFailedTests: true
+    condition: and(eq(variables.runTests, 'True'), succeededOrFailed())

tools/ci/README.md

@@ -44,19 +44,29 @@ sbt-running job is wired to the shared cache template + prewarm job.
 ## `databricks_impact.py` — conservative PR E2E gating
 
 The `BuildAndCacheSbt` job compares a pull request with its target branch and
-uses `databricks_impact.py` to decide whether the six Databricks matrix jobs can
-be skipped. Scheduled, master, tag, and manual builds always run Databricks.
+uses `databricks_impact.py` to decide independently whether the five CPU matrix
+jobs and the GPU matrix job can be skipped. Scheduled, master, tag, and manual
+builds always run both suites.
 
-The detector is intentionally fail-open. It skips Databricks only when every
-changed path is clearly unrelated to runtime artifacts and notebook execution:
+The detector mirrors the enabled test suites:
+
+- CPU runs for runtime changes in any module and non-GPU notebooks.
+- GPU runs for shared core/deep-learning runtime changes and the three
+  `Fine-tune`/`Phi Model` notebooks selected by `DatabricksGPUTests`.
+- Databricks utility changes are assigned to CPU, GPU, or both according to
+  which suite imports them.
+
+The detector is fail-open. Unknown paths, build definitions, templates,
+environment files, shared test infrastructure, missing diffs, and detection
+errors run both suites. It skips both suites only for paths known not to affect
+runtime artifacts or notebook execution:
 
 - GitHub metadata and workflows
+- unrelated pipelines and ACR/Docker/Helm tooling
 - CI helper code under `tools/ci/`
 - website files
 - Markdown/reStructuredText documentation
-- non-notebook files under `docs/`
 - module test source outside the Databricks notebook and shared test infrastructure
 
-Runtime source, `.ipynb` notebooks, build definitions, pipeline/templates,
-Databricks test utilities, unknown paths, an empty diff, or a failed target
-branch fetch all keep Databricks E2E enabled.
+Unknown non-notebook assets under `docs/` remain fail-open because notebooks may
+load adjacent data or configuration files.