PR Pipeline - Build/Pack and Local Tests (dotnet#4326)

* Move install-dotnet to common/steps, first bits of pipeline

# Conflicts:
#	eng/pipelines/common/steps/restore-dotnet-tools.yml

* Adding build and pack steps and writing up the build/pack stage.
Leaving the unused build step for now just in case we need it later

* Move common variables to common folder. Move OneBranch specific variables to OneBranch variables file

* Rename prci-pipeline to pr-pipeline

* First round of tests

* Run build/pack regardless of previous package.

* Run tree and publish artifact regardless of previous success

* Completely untested, let's see what it does.

* Rename build-and-pack-stage.yml

* tree on linux

* Fix test framework

* Switch images to ones that have sql server on it
Move a couple things around
WIP for manual test configuration generation

* 🤖 Implement config file generation for manual test

* Fix build name

* WIP

* WIP

* Add secret generation stage

* Run pack-stage and generate-secrets-stage in parallel

* More WIP to wire up manual test config

* 🤖 Finish wiring up what exists today. Time to try it out and see what's broken.

* Fine, 🤖, you were right. Adding back the operating system parameters

* Fix link to linux config step

* 🤖 remove dependency on extutilities

* 🤖 Get connection string for setting up northwind from config

* 🤖 Only swap out the initial catalog field.

* 🤖 vs 🤖 :fight:

* Sync config.cs and config.json

* Cleanup powershells

* Config steps

* 🔨

* Diagnostic

* Stupid powershell.

* Making demands.

* Fixing the first round of manual tests.

* Increase timeout on sql notification test... not ideal but if it helps it helps.

* Add retries to dotnet install steps

* Cleanup and add explicit disposal logic to AlwaysEncrypted.ApiShould.TestExecuteReaderAsyncWithLargeQuery

* Temporarily mark SqlNotificationTest.Test_SingleDependency_AllDefaults_SqlAuth as flay

* Comments for pipeline, stages, jobs, variables.
Moving the SA password retrieval deeper into the manual test job

* Fix SA password issue.

* 🔨

* Exclude AE ApiShould tests on debug builds.

* Address copilot feedback round 1 of `n`

* Config.json changes applied to pipelines.

* Restore tools in all instances before running jobs.

* Getting really tired of these changes

* wouldn't it be nice if we had something to verify this before pushing a bunch of tiny little changes?

* Hope it can handle extra spaces.

* Fix path to dotnet arm64 installation script

* Remove reference to ADO Test Configuration Properties variable group.

* Parameterize the pool

* It's not a pipeline parameter you stupid 🤖

* 😡

* INVERSION

* Fix kerberos pipeline install-dotnet.yml path

Author: benrr101

…pipelines/steps/install-dotnet-arm64.ps1 …es/common/steps/install-dotnet-arm64.ps1eng/pipelines/steps/install-dotnet-arm64.ps1 renamed to eng/pipelines/common/steps/install-dotnet-arm64.ps1 eng/pipelines/steps/install-dotnet-arm64.ps1 renamed to eng/pipelines/common/steps/install-dotnet-arm64.ps1

@@ -69,6 +69,7 @@ steps:
         env:
           PROCESSOR_ARCHITECTURE: x86
 
+
     # Install the desired Runtimes, if any.
     - ${{ each version in parameters.runtimes }}:
       - task: UseDotNet@2
@@ -96,7 +97,7 @@ steps:
       inputs:
         targetType: filePath
         pwsh: true
-        filePath: $(Build.SourcesDirectory)/eng/pipelines/steps/install-dotnet-arm64.ps1
+        filePath: $(Build.SourcesDirectory)/eng/pipelines/common/steps/install-dotnet-arm64.ps1
         # Arguments:
         #
         # -Debug:

eng/pipelines/steps/install-dotnet.yml …ipelines/common/steps/install-dotnet.ymleng/pipelines/steps/install-dotnet.yml renamed to eng/pipelines/common/steps/install-dotnet.yml eng/pipelines/steps/install-dotnet.yml renamed to eng/pipelines/common/steps/install-dotnet.yml

@@ -132,10 +132,10 @@ jobs:
         targetPath: $(localFeedPath)
 
   # Install the .NET SDK.
-  - template: /eng/pipelines/steps/install-dotnet.yml@self
+  - template: /eng/pipelines/common/steps/install-dotnet.yml@self
 
   # Restore dotnet CLI tools (e.g. pwsh, apicompat) before building.
-  - template: /eng/pipelines/steps/restore-dotnet-tools.yml@self
+  - template: /eng/pipelines/common/steps/restore-dotnet-tools.yml@self
 
   # When we're performing a Debug build, we still want to try _compiling_ the
   # code in Release mode to ensure downstream pipelines don't encounter

…pipelines/steps/restore-dotnet-tools.yml …es/common/steps/restore-dotnet-tools.ymleng/pipelines/steps/restore-dotnet-tools.yml renamed to eng/pipelines/common/steps/restore-dotnet-tools.yml eng/pipelines/steps/restore-dotnet-tools.yml renamed to eng/pipelines/common/steps/restore-dotnet-tools.yml

@@ -44,7 +44,7 @@ jobs:
           displayName: '[Debug] List Environment Variables'
 
       # Install the .NET SDK.
-      - template: /eng/pipelines/steps/install-dotnet.yml@self
+      - template: /eng/pipelines/common/steps/install-dotnet.yml@self
         parameters:
           debug: ${{ parameters.debug }}
 

eng/pipelines/common/templates/jobs/ci-build-nugets-job.yml

@@ -30,11 +30,9 @@ parameters:
   # template expansion.
   #
   #   EnclaveEnabled
-  #   IsAzureSynapse
   #   IsDNSCachingSupportedCR
   #   IsDNSCachingSupportedTR
   #   ManagedIdentitySupported
-  #   SupportsFileStream
   #   SupportsIntegratedSecurity
   #   TracingEnabled
   #
@@ -211,7 +209,7 @@ jobs:
         targetPath: $(Build.SourcesDirectory)/packages
 
   # Install the .NET SDK and Runtimes.
-  - template: /eng/pipelines/steps/install-dotnet.yml@self
+  - template: /eng/pipelines/common/steps/install-dotnet.yml@self
     parameters:
       debug: ${{ parameters.debug }}
       ${{ if parameters.isArm64 }}:
@@ -223,7 +221,7 @@ jobs:
         runtimes: [8.x, 9.x]
 
   # Restore dotnet CLI tools (e.g. pwsh, apicompat) before building.
-  - template: /eng/pipelines/steps/restore-dotnet-tools.yml@self
+  - template: /eng/pipelines/common/steps/restore-dotnet-tools.yml@self
 
   - ${{ if ne(parameters.prebuildSteps, '') }}:
     - ${{ parameters.prebuildSteps }} # extra steps to run before the build like downloading sni and the required configuration
@@ -279,8 +277,6 @@ jobs:
           AliasName: ${{ parameters.configProperties.AliasName }}
         ${{ if parameters.configProperties.SupportsIntegratedSecurity }}:
           SupportsIntegratedSecurity: ${{ eq(parameters.configProperties.SupportsIntegratedSecurity, 'true') }}
-        ${{ if parameters.configProperties.SupportsFileStream }}:
-          SupportsFileStream: ${{ eq(parameters.configProperties.SupportsFileStream, 'true') }}
         ${{ if parameters.configProperties.DNSCachingConnString }}:
           DNSCachingConnString: ${{ parameters.configProperties.DNSCachingConnString }}
         ${{ if parameters.configProperties.DNSCachingServerCR }}:
@@ -293,8 +289,6 @@ jobs:
           IsDNSCachingSupportedCR: ${{ eq(parameters.configProperties.IsDNSCachingSupportedCR, 'true') }}
         ${{ if parameters.configProperties.IsDNSCachingSupportedTR }}:
           IsDNSCachingSupportedTR: ${{ eq(parameters.configProperties.IsDNSCachingSupportedTR, 'true') }}
-        ${{ if parameters.configProperties.IsAzureSynapse }}:
-          IsAzureSynapse: ${{ eq(parameters.configProperties.IsAzureSynapse, 'true') }}
         ${{ if parameters.configProperties.ManagedIdentitySupported }}:
           ManagedIdentitySupported: ${{ eq(parameters.configProperties.ManagedIdentitySupported, 'true') }}
 
@@ -352,7 +346,7 @@ jobs:
   - ${{ if and(eq(parameters.enableX86Test, true), eq(parameters.operatingSystem, 'Windows')) }}:
     - ${{ if ne(variables['dotnetx86RootPath'], '') }}:
       # Install the .NET SDK and Runtimes for x86.
-      - template: /eng/pipelines/steps/install-dotnet.yml@self
+      - template: /eng/pipelines/common/steps/install-dotnet.yml@self
         parameters:
           architecture: x86
           debug: ${{ parameters.debug }}

eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml

@@ -5,7 +5,7 @@
 #################################################################################
 steps:
   # Install the .NET SDK and Runtimes.
-  - template: /eng/pipelines/steps/install-dotnet.yml@self
+  - template: /eng/pipelines/common/steps/install-dotnet.yml@self
     parameters:
       runtimes: [8.x, 9.x]
 

eng/pipelines/common/templates/jobs/ci-run-tests-job.yml

@@ -93,10 +93,6 @@ parameters:
     type: boolean
     default: false
 
-  - name: SupportsFileStream
-    type: boolean
-    default: false
-
   - name: DNSCachingConnString
     type: string
     default: ''
@@ -121,10 +117,6 @@ parameters:
     type: boolean
     default: false
 
-  - name: IsAzureSynapse
-    type: boolean
-    default: false
-
   - name: ManagedIdentitySupported
     type: boolean
     default: true
@@ -188,8 +180,6 @@ steps:
 
           $p.DNSCachingConnString="${{parameters.DNSCachingConnString }}"
 
-          $p.SupportsFileStream="${{parameters.SupportsFileStream }}"
-
           $p.LocalDbAppName="${{parameters.LocalDbAppName }}"
 
           $p.TCPConnectionStringAASSGX="${{parameters.TCPConnectionStringAASSGX }}"
@@ -201,7 +191,6 @@ steps:
           $p.UseManagedSNIOnWindows=[System.Convert]::ToBoolean("${{parameters.UseManagedSNIOnWindows }}")
           $p.SupportsIntegratedSecurity=[System.Convert]::ToBoolean("${{parameters.SupportsIntegratedSecurity }}")
           $p.ManagedIdentitySupported=[System.Convert]::ToBoolean("${{parameters.ManagedIdentitySupported }}")
-          $p.IsAzureSynapse=[System.Convert]::ToBoolean("${{parameters.IsAzureSynapse }}")
           $p.IsDNSCachingSupportedTR=[System.Convert]::ToBoolean("${{parameters.IsDNSCachingSupportedTR }}")
           $p.IsDNSCachingSupportedCR=[System.Convert]::ToBoolean("${{parameters.IsDNSCachingSupportedCR }}")
           $p.TracingEnabled=[System.Convert]::ToBoolean("${{parameters.TracingEnabled }}")

eng/pipelines/common/templates/steps/pre-build-step.yml

@@ -34,7 +34,7 @@ parameters:
 
 steps:
   # Install the .NET SDK, required by the PowerShell script.
-  - template: /eng/pipelines/steps/install-dotnet.yml@self
+  - template: /eng/pipelines/common/steps/install-dotnet.yml@self
 
   # Invoke the script with the path to the packages to verify.
   - task: PowerShell@2

eng/pipelines/common/templates/steps/update-config-file-step.yml

@@ -4,11 +4,9 @@
 # See the LICENSE file in the project root for more information.                #
 #################################################################################
 
-# Common variables shared across all OneBranch Official pipelines.
+# Common variables shared across all pipelines.
 
 variables:
-  - name: CommitHead
-    value: '' # the value will be extracted from the repo's head
 
   ##########################################################################
   # "Well-Known" Variables that are ok to use directly, anywhere in the pipeline.
@@ -21,15 +19,3 @@ variables:
   # <ArtifactsPath>).
   - name: BUILD_OUTPUT
     value: $(REPO_ROOT)/artifacts
-
-  # Directory where downloaded pipeline artifacts (NuGet packages from earlier
-  # stages) are placed. Build jobs use this as a local NuGet package source so
-  # that downstream packages can resolve dependencies on packages built by
-  # upstream stages.
-  - name: JOB_INPUT
-    value: $(REPO_ROOT)/packages
-
-  # Root directory for all job output artifacts (NuGet packages, symbols, etc.). OneBranch auto
-  # publishes everything under this directory as the artifact for the job.
-  - name: JOB_OUTPUT
-    value: $(REPO_ROOT)/output