Move symbol publishing into a dedicated stage and refactor build artifacts (dotnet#4175)

Author: paulmedynski

.github/instructions/onebranch-pipeline-design.instructions.md

@@ -36,6 +36,8 @@ Defined in `stages/build-stages.yml`. Four build stages plus validation, ordered
 - **`build_addons`** (Stage 4) — AKV Provider; `dependsOn: build_dependent`; downloads SqlClient + Abstractions + Logging artifacts
 - **`mds_package_validation`** — Validates signed SqlClient package; `dependsOn: build_dependent`; runs in parallel with Stage 4
 
+Each build job copies PDB files into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published pipeline artifact alongside the NuGet packages in `$(JOB_OUTPUT)/packages/`.
+
 Stage conditional rules:
 - Wrap stages/jobs in `${{ if }}` compile-time conditionals based on build parameters
 - `buildSqlClient` controls Stages 2, 3, validation, and Logging (when AKV is disabled)
@@ -45,17 +47,30 @@ Stage conditional rules:
 
 ## Job Templates
 
-- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing
-- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing
+- **`build-signed-csproj-package-job.yml`** — Generic job for csproj-based packages (Logging, SqlServer.Server, Abstractions, Azure, AKV Provider). Flow: Build DLLs → ESRP DLL signing → NuGet pack (`NoBuild=true`) → ESRP NuGet signing → Copy PDBs to artifact
+- **`build-signed-sqlclient-package-job.yml`** — SqlClient-specific job (nuspec-based). Flow: Build all configurations → ESRP DLL signing (main + resource DLLs) → NuGet pack via nuspec → ESRP NuGet signing → Copy PDBs to artifact
 - **`validate-signed-package-job.yml`** — Validates signed MDS package (signature, strong names, folder structure, target frameworks)
 - **`publish-nuget-package-job.yml`** — Reusable release job using OneBranch `templateContext.type: releaseJob` with `inputs` for artifact download; pushes via `NuGetCommand@2`
+- **`publish-symbols-job.yml`** — Reusable symbols job: downloads a build artifact, locates PDBs under `symbols/`, and invokes `publish-symbols-step.yml`
 
 When adding a new csproj-based package:
 - Use `build-signed-csproj-package-job.yml` with appropriate `packageName`, `packageFullName`, `versionProperties`, and `downloadArtifacts`
 - Add build and pack targets to `build.proj`
 - Add version variables to `variables/common-variables.yml`
 - Add artifact name variable to `variables/onebranch-variables.yml`
 
+## Symbols Publishing Stage
+
+- Defined in `stages/publish-symbols-stage.yml`; produces stage `publish_symbols`
+- Entire stage excluded at compile time when `publishSymbols` is false
+- `dependsOn` is conditional based on which `build*` parameters are set, mirroring the build stage dependency graph
+- One job per package (`publish-symbols-job.yml`), each downloading its build artifact and publishing PDBs from `symbols/`
+- Each package's PDBs are published separately with unique artifact names and version information
+- Build jobs copy PDBs into `$(JOB_OUTPUT)/symbols/` so they are included in the auto-published artifact
+- The `publish-symbols-step.yml` accepts a `symbolsFolder` parameter to point at the downloaded PDB location
+- The publish step calls an extracted `publish-symbols.ps1` script with structured error handling and diagnostic logging
+- Symbols publishing credentials come from the `Symbols Publishing` variable group
+
 ## Release Stage
 
 - Defined in `stages/release-stages.yml`; produces stage `release_production` (official) or `release_test` (non-official) via `stageNameSuffix` parameter
@@ -98,8 +113,7 @@ When `isPreview` is true, pipeline resolves `effective*Version` variables to pre
 - When adding a new package, add GA version, preview version, and assembly file version entries
 
 Variable groups:
-- `Release Variables` — release configuration (in `common-variables.yml`)
-- `Symbols publishing` — symbol publishing credentials (in `common-variables.yml`)
+- `Symbols Publishing` — symbol publishing credentials (in `onebranch-variables.yml`)
 - `ESRP Federated Creds (AME)` — ESRP signing credentials (in `common-variables.yml`)
 
 ## Code Signing (ESRP)
@@ -115,15 +129,19 @@ Variable groups:
 
 - TSA: enabled only in official pipeline; disabled in non-official to avoid spurious alerts
 - ApiScan: enabled in both; currently `break: false` pending package registration
-- Each build job sets `ob_sdl_apiscan_*` variables pointing to `$(Build.SourcesDirectory)/apiScan/<PackageName>/`
+- Each build job sets `ob_sdl_apiscan_softwareFolder` to `$(JOB_OUTPUT)/assemblies` and `ob_sdl_apiscan_symbolsFolder` to `$(JOB_OUTPUT)/symbols`
 - CodeQL, SBOM, Policheck (`break: true`): enabled in both pipelines
 - asyncSdl `enabled: false` in both; individual sub-tools (CredScan, BinSkim, Armory, Roslyn) configured underneath
 - Policheck exclusions: `$(REPO_ROOT)\.config\PolicheckExclusions.xml`
 - CredScan suppressions: `$(REPO_ROOT)/.config/CredScanSuppressions.json`
 
 ## Artifact Conventions
 
-- `ob_outputDirectory` set to `$(PACK_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
+- `ob_outputDirectory` set to `$(JOB_OUTPUT)` (= `$(REPO_ROOT)/output`) — OneBranch auto-publishes this directory
+- Each published artifact uses subdirectories to separate file types:
+  - `assemblies/` — DLL assemblies for APIScan (preserving TFM folder structure)
+  - `packages/` — NuGet packages (`.nupkg`, `.snupkg`)
+  - `symbols/` — PDB symbol files (preserving TFM folder structure, shared by APIScan and symbol publishing)
 - Artifact names follow `drop_<stageName>_<jobName>` — defined in `variables/onebranch-variables.yml`
 - Downstream jobs download artifacts via `DownloadPipelineArtifact@2` into `$(Build.SourcesDirectory)/packages`
 - Downloaded packages serve as a local NuGet source for `dotnet restore`

eng/pipelines/onebranch/jobs/build-signed-csproj-package-job.yml

@@ -36,26 +36,6 @@ parameters:
   - name: signingEsrpConnectedServiceName
     type: string
 
-  # Symbols Publishing Parameters ------------------------------------------
-
-  - name: symbolsAzureSubscription
-    type: string
-    default: 'Symbols publishing Workload Identity federation service-ADO.Net'
-
-  - name: symbolsPublishProjectName
-    type: string
-    default: 'Microsoft.Data.SqlClient.SNI'
-
-  - name: symbolsPublishServer
-    type: string
-
-  - name: symbolsPublishTokenUri
-    type: string
-
-  - name: symbolsUploadAccount
-    type: string
-    default: 'SqlClientDrivers'
-
   # OTHERS +=====================================
 
   # Short package name used in the job name, display strings, filesystem paths, and as a suffix for
@@ -74,11 +54,6 @@ parameters:
   - name: packageFullName
     type: string
 
-  # The version of the package.  This is used for symbol publishing.  It is not used for the DLL or
-  # NuGet package versions since those are supplied via the versionProperties parameter.
-  - name: packageVersion
-    type: string
-
   # The MSBuild build target in build.proj (e.g. BuildLogging).  If not specified, defaults to
   # Build<packageName>.
   - name: buildTarget
@@ -106,10 +81,6 @@ parameters:
   - name: assemblyFileVersion
     type: string
 
-  # True to publish symbols to private and public servers.
-  - name: publishSymbols
-    type: boolean
-
   # Optional list of pipeline artifacts to download before building.  Each entry is an object
   # with 'artifactName' (the pipeline artifact name) and 'displayName' (used in the task label).
   # This replaces hard-coded packageName conditionals so callers declare their own dependencies.
@@ -125,12 +96,12 @@ jobs:
 
     variables:
       # Inform OneBranch that files put in this directory should be uploaded as artifacts.
-      ob_outputDirectory: $(PACK_OUTPUT)
+      ob_outputDirectory: $(JOB_OUTPUT)
 
       # APIScan configuration for this Extension package
       ob_sdl_apiscan_enabled: true
-      ob_sdl_apiscan_softwareFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/dlls
-      ob_sdl_apiscan_symbolsFolder: $(REPO_ROOT)/apiScan/${{ parameters.packageName }}/pdbs
+      ob_sdl_apiscan_softwareFolder: $(JOB_OUTPUT)/assemblies
+      ob_sdl_apiscan_symbolsFolder: $(JOB_OUTPUT)/symbols
       ob_sdl_apiscan_softwarename: ${{ parameters.packageFullName }}
       ob_sdl_apiscan_versionNumber: ${{ parameters.assemblyFileVersion }}
 
@@ -149,7 +120,7 @@ jobs:
             displayName: Download ${{ artifact.displayName }}
             inputs:
               artifactName: ${{ artifact.artifactName }}
-              targetPath: $(REPO_ROOT)/packages
+              targetPath: $(JOB_INPUT)
 
       # Install the .NET SDK.
       - template: /eng/pipelines/steps/install-dotnet.yml@self
@@ -179,26 +150,42 @@ jobs:
               authSignCertName: '${{ parameters.signingAuthSignCertName }}'
               esrpClientId: '${{ parameters.signingEsrpClientId }}'
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
-              pattern: '${{ parameters.packageFullName }}.dll'
-
-      # Copy signed/unsigned DLLs and PDBs to APIScan folders.
+              # Minimatch pattern (multi-line, one pattern per line).
+              # **/ matches any nested directory (e.g. net8.0/, netstandard2.0/fr/).
+              #
+              # Matches:  net8.0/<packageFullName>.dll
+              #           netstandard2.0/fr/<packageFullName>.resources.dll
+              pattern: |
+                **/${{ parameters.packageFullName }}.dll
+                **/${{ parameters.packageFullName }}.resources.dll
+
+      # Copy DLLs to the assemblies/ subdirectory for APIScan.
       - task: CopyFiles@2
         displayName: Copy DLLs for APIScan
         inputs:
           SourceFolder: $(BUILD_OUTPUT)/Package/bin
-          Contents: "**/${{ parameters.packageFullName }}.dll"
-          TargetFolder: ${{ variables.ob_sdl_apiscan_softwareFolder }}
+          # Matches: <packageFullName>.dll (main assembly) across all TFM subdirs
+          #          <packageFullName>.resources.dll (localized satellite assemblies in locale subdirs, if any)
+          Contents: |
+            **/${{ parameters.packageFullName }}.dll
+            **/${{ parameters.packageFullName }}.resources.dll
+          TargetFolder: $(JOB_OUTPUT)/assemblies
           # We must preserve the folder structure since our C# projects may produce multiple
           # identically named DLLs for different target frameworks (e.g. netstandard2.0, net5.0,
           # etc.), and we need to keep those separate for APIScan to work correctly.
           flattenFolders: false
 
+      # Copy PDBs into the output directory so they are included in the published pipeline
+      # artifact. The symbols publishing stage will download this artifact and publish PDBs
+      # for this package using the files under symbols/.
       - task: CopyFiles@2
-        displayName: Copy PDBs for APIScan
+        displayName: Copy PDBs for APIScan and Symbols Publishing
         inputs:
           SourceFolder: $(BUILD_OUTPUT)/Package/bin
-          Contents: "**/${{ parameters.packageFullName }}.pdb"
-          TargetFolder: ${{ variables.ob_sdl_apiscan_symbolsFolder }}
+          # Matches: <packageFullName>.pdb across all TFM subdirs.
+          # Note: Resource DLLs are resource-only satellite assemblies and do not produce PDBs.
+          Contents: '**/${{ parameters.packageFullName }}.pdb'
+          TargetFolder: $(JOB_OUTPUT)/symbols
           flattenFolders: false
 
       # Pack the signed DLLs into NuGet package (NoBuild=true).
@@ -217,22 +204,13 @@ jobs:
               authSignCertName: '${{ parameters.signingAuthSignCertName }}'
               esrpClientId: '${{ parameters.signingEsrpClientId }}'
               esrpConnectedServiceName: '${{ parameters.signingEsrpConnectedServiceName }}'
-              searchPath: $(PACK_OUTPUT)
-              searchPattern: '${{ parameters.packageFullName}}.*nupkg'
-
-      # Publish symbols to servers
-      # @TODO: Get these parameters from variables/libraries
-      - ${{ if eq(parameters.publishSymbols, true) }}:
-          - template: /eng/pipelines/onebranch/steps/publish-symbols-step.yml@self
-            parameters:
-              artifactName: '${{ parameters.packageFullName }}_symbols_$(System.TeamProject)_$(Build.Repository.Name)_$(Build.SourceBranchName)_${{ parameters.packageVersion }}_$(System.TimelineId)'
-              azureSubscription: '${{ parameters.symbolsAzureSubscription }}'
-              packageName: '${{ parameters.packageFullName }}'
-              publishProjectName: '${{ parameters.symbolsPublishProjectName }}'
-              publishServer: '${{ parameters.symbolsPublishServer }}'
-              publishToInternal: 'true'
-              publishToPublic: 'true'
-              publishTokenUri: '${{ parameters.symbolsPublishTokenUri }}'
-              searchPattern: '**/${{ parameters.packageFullName }}*.pdb'
-              uploadAccount: '${{ parameters.symbolsUploadAccount }}'
-              version: '${{ parameters.packageVersion }}'
+              searchPath: $(JOB_OUTPUT)/packages
+              # Minimatch pattern with extglob.
+              # [0-9] matches a single digit, anchoring to the version segment so
+              # similarly-prefixed package names are not matched.
+              # ?(s) is an extglob that optionally matches 's'.
+              #
+              # Matches:  <packageFullName>.1.0.0.nupkg
+              #           <packageFullName>.1.0.0-preview.1.snupkg
+              # Excludes: <packageFullName>.SomeOther.1.0.0.nupkg
+              searchPattern: '${{ parameters.packageFullName }}.[0-9]*.?(s)nupkg'