Azure · m-redding · May 15, 2025 · May 15, 2025 · May 15, 2025 · May 16, 2025
diff --git a/doc/dev/AotRegressionChecks.md b/doc/dev/AotRegressionChecks.md
@@ -1,68 +1,24 @@
-# Enable AOT compatibility regression testing in CI pipelines
+# AOT compatibility regression test in CI pipelines
 
-An increasing number of .NET Azure SDK libraries are committed to being compatible with native AOT. For more information about native AOT deployment see [this article](https://learn.microsoft.com/dotnet/core/deploying/native-aot/). To support this work, there is now an opt-in pipeline step called "Check for AOT compatibility regressions in \[Package Name\]". This pipeline creates a small sample app that uses a project reference to collect the set of trimming warnings reported for the specified library. This approach for collecting warnings is described in [this article](https://learn.microsoft.com/dotnet/core/deploying/trimming/prepare-libraries-for-trimming?pivots=dotnet-8-0#show-all-warnings-with-test-app).
+An increasing number of .NET Azure SDK libraries are committed to being fully compatible with native AOT. For more information about native AOT deployment see [this article](https://learn.microsoft.com/dotnet/core/deploying/native-aot/).
 
-## How to enable the pipeline for your package
+In order to align with this priority, a pipeline step called "Check for AOT compatibility regressions in \[Package Name\]" was added in all CI pipelines to prevent AOT regressions in all SDK libraries. This is true even if the package is not yet fully compatible with native AOT. This pipeline creates a small sample app that uses a project reference to collect the set of trimming warnings reported for the specified library. This approach for collecting warnings is described in [this article](https://learn.microsoft.com/dotnet/core/deploying/trimming/prepare-libraries-for-trimming?pivots=dotnet-8-0#show-all-warnings-with-test-app).
 
-### Collect any expected trimming warnings
+## See AOT warnings locally
 
-You can use any of the approaches described in the articles linked at the bottom of this document to find the warnings reported from your library. In an ideal scenario, this would be zero. However, there are cases where a library needs to baseline an expected set of warnings. Sometimes warnings are not straightforward \(or are even impossible\) to resolve, but are not expected to impact the majority of customer use cases. In other cases, warnings may be dependent on other work finishing first (for example, adding a net6.0 target, or upgrading a package dependency version).
+Run the [Check AOT Compatibility script](https://github.com/Azure/azure-sdk-for-net/blob/main/eng/scripts/compatibility/Check-AOT-Compatibility.ps1).
 
-The text file should be formatted with each warning on its own line. The pipeline uses pattern matching to validate warnings. This means that warnings will need to be edited to avoid using special characters incorrectly. Even though it seems easier to just do simple string matching, the errors are formatted differently depending on the environment, so using correctly formatted pattern matching makes it easier.
-
-**Example**:
-
-Actual warning:
-> C:\Users\mredding\source\repos\azure-sdk-for-net\sdk\core\Azure.Core\src\JsonPatchDocument.cs(44): Trim analysis warning IL2026: Azure.JsonPatchDocument.JsonPatchDocument(ReadOnlyMemory`1<Byte>): Using member 'Azure.Core.Serialization.JsonObjectSerializer.JsonObjectSerializer()' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. This class uses reflection-based JSON serialization and deserialization that is not compatible with trimming. [C:\Users\mredding\source\repos\ResolveAOT\ResolveAOT\ResolveAOT.csproj]
-
-Line in the text file:
-> Azure\\.Core.src.JsonPatchDocument\\.cs\\(\\d*\\): Trim analysis warning IL2026: Azure\\.JsonPatchDocument\\.JsonPatchDocument\\(ReadOnlyMemory`1<Byte>\\): Using member 'Azure\\.Core\\.Serialization\\.JsonObjectSerializer\\.JsonObjectSerializer\\(\\)' which has 'RequiresUnreferencedCodeAttribute'
-
-**Note**: In my case, my local environment uses forward slashes in filepaths, while the pipeline uses back slashes, so using a wildcard was the easiest way to reconcile this.
-
-### Update ci.yml file
-
-The ci.yml file needs to be updated to include the following information:
-```yml
-extends:
-  template: /eng/pipelines/templates/stages/archetype-sdk-client.yml
-  parameters:
-    ServiceDirectory: [service directory]
-    # [... other inputs]
-    CheckAOTCompat: true
-    AOTTestInputs:
-    - ArtifactName: [Name of package]
-      ExpectedWarningsFilePath: None [or filepath of errors relative to the service directory]
+To see all warnings:
+```
+PS C:\...\azure-sdk-for-net\eng\scripts\compatibility> .\Check-AOT-Compatibility.ps1 core Azure.Core
 ```
 
-**Example**:
-```yml
-extends:
-  template: /eng/pipelines/templates/stages/archetype-sdk-client.yml
-  parameters:
-    ServiceDirectory: core
-    # [... other inputs]
-    CheckAOTCompat: true
-    AOTTestInputs:
-    - ArtifactName: Azure.Core
-      ExpectedWarningsFilepath: /Azure.Core/tests/aotcompatibility/ExpectedAotWarnings.txt
+To see only the warnings that are not baselined:
 ```
-**Example 2**:
-```yml
-extends:
-  template: /eng/pipelines/templates/stages/archetype-sdk-client.yml
-  parameters:
-    ServiceDirectory: core
-    # [... other inputs]
-    CheckAOTCompat: true
-    AOTTestInputs:
-    - ArtifactName: Azure.Core
-      ExpectedWarningsFilepath: /Azure.Core/tests/aotcompatibility/ExpectedAotWarnings.txt
-    - ArtifactName: Azure.Core.Experimental # For illustration only
-      ExpectedWarningsFilepath: None
+PS C:\...\azure-sdk-for-net\eng\scripts\compatibility> .\Check-AOT-Compatibility.ps1 core Azure.Core "Azure.Core/tests/compatibility/ExpectedWarnings.txt"
 ```
 
-## How to resolve trimming warnings
+## Resolve AOT warnings
 
 The following three articles provide comprehensive guidance for how to resolve trimming warnings and make libraries compatible with AOT:
 - ["How to make libraries compatible with native AOT"](https://devblogs.microsoft.com/dotnet/creating-aot-compatible-libraries/) by Eric Erhardt on November 30th, 2023
@@ -72,3 +28,28 @@ The following three articles provide comprehensive guidance for how to resolve t
 Learning how to use source generation for serialization/deserialization:
 - [Introduction of C# source generation in .NET 6](https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-source-generator/)
 - [How to use source generation](https://learn.microsoft.com/dotnet/standard/serialization/system-text-json/source-generation)
+
+
+## Baseline expected warnings
+
+In an ideal scenario, this pipeline will pass with no baselined warnings. However, there are cases where a library needs to define an expected set of warnings to ignore when running this check. Sometimes warnings are not straightforward \(or are even impossible\) to resolve, but are not expected to impact the majority of customer use cases. In other cases, warnings may be dependent on other work finishing first (for example, adding a net6.0 target, or upgrading a package dependency version).
+
+The text file should be formatted with each warning on its own line. The pipeline uses pattern matching to validate warnings. You can use BaselineExistingWarnings.ps1 in eng/scripts/compatibility to create the expected warnings file from the AOT regressions output from Check-AOT-Compatibility.
+
+```
+```
+
+
+### Update ci.yml file
+
+To baseline warnings, pass the filepath to the CI in ci.yml using the following structure:
+```yml
+extends:
+  template: /eng/pipelines/templates/stages/archetype-sdk-client.yml
+  parameters:
+    ServiceDirectory: [service directory]
+    # [... other inputs]
+    ExpectedAOTWarnings:
+    - ArtifactName: [Name of package]
+      ExpectedWarningsFilePath: [Filepath of errors relative to the service directory]
+```
@@ -13,8 +13,8 @@ parameters:
     default: all
   - name: CheckAOTCompat
     type: boolean
-    default: false
-  - name: AOTTestInputs
+    default: true
+  - name: ExpectedAOTWarnings
     type: object
     default: []
   - name: BuildSnippets
@@ -70,7 +70,7 @@ jobs:
           ServiceDirectory: ${{ parameters.ServiceDirectory }}
           SDKType: ${{ parameters.SDKType }}
           CheckAOTCompat: ${{ parameters.CheckAOTCompat }}
-          AOTTestInputs: ${{ parameters.AOTTestInputs }}
+          ExpectedAOTWarnings: ${{ parameters.ExpectedAOTWarnings }}
 
   - job:
     displayName: "Analyze"

@@ -19,8 +19,8 @@ parameters:
     default: true
   - name: CheckAOTCompat
     type: boolean
-    default: false
-  - name: AOTTestInputs
+    default: true
+  - name: ExpectedAOTWarnings
     type: object
     default: []
   - name: TestSetupSteps
@@ -72,7 +72,7 @@ jobs:
           ServiceDirectory: ${{ parameters.ServiceDirectory }}
           SDKType: ${{ parameters.SDKType }}
           CheckAOTCompat: ${{ parameters.CheckAOTCompat }}
-          AOTTestInputs: ${{ parameters.AOTTestInputs }}
+          ExpectedAOTWarnings: ${{ parameters.ExpectedAOTWarnings }}
           BuildSnippets: ${{ parameters.BuildSnippets }}
         PreGenerationSteps:
           - template: /eng/pipelines/templates/steps/pr-matrix-presteps.yml
@@ -104,7 +104,7 @@ jobs:
             ServiceDirectory: ${{ parameters.ServiceDirectory }}
             SDKType: ${{ parameters.SDKType }}
             CheckAOTCompat: ${{ parameters.CheckAOTCompat }}
-            AOTTestInputs: ${{ parameters.AOTTestInputs }}
+            ExpectedAOTWarnings: ${{ parameters.ExpectedAOTWarnings }}
 
     - job: "Analyze"
       timeoutInMinutes: ${{ parameters.TestTimeoutInMinutes }}

@@ -19,8 +19,8 @@ parameters:
     default: true
   - name: CheckAOTCompat
     type: boolean
-    default: false
-  - name: AOTTestInputs
+    default: true
+  - name: ExpectedAOTWarnings
     type: object
     default: []
   - name: TestSetupSteps
@@ -79,7 +79,7 @@ extends:
               LimitForPullRequest: ${{ parameters.LimitForPullRequest }}
               BuildSnippets: ${{ parameters.BuildSnippets }}
               CheckAOTCompat: ${{ parameters.CheckAOTCompat }}
-              AOTTestInputs: ${{ parameters.AOTTestInputs }}
+              ExpectedAOTWarnings: ${{ parameters.ExpectedAOTWarnings }}
               TestSetupSteps: ${{ parameters.TestSetupSteps }}
               TestTimeoutInMinutes: ${{ parameters.TestTimeoutInMinutes }}
               MatrixConfigs:

@@ -1,7 +1,8 @@
 parameters:
   ServiceDirectory: ''
   PackageInfoFolder: ''
-  AOTTestInputs: []
+  Artifacts: []
+  ExpectedAOTWarnings: []
 
 steps:
 
@@ -16,14 +17,14 @@ steps:
         -ProjectNames "$(ProjectNames)"
       workingDirectory: $(Build.SourcesDirectory)/eng/scripts/compatibility
 - ${{ else }}:
-  - ${{ each aotTestInput in parameters.AOTTestInputs }}:
+  - ${{ each artifact in parameters.Artifacts }}:
     - task: Powershell@2
-      displayName: Check for AOT compatibility regressions in ${{ aotTestInput.ArtifactName }}
+      displayName: Check for AOT compatibility regressions in ${{ artifact.name }}
       inputs:
         targetType: filepath
         filePath: $(Build.SourcesDirectory)/eng/scripts/compatibility/Check-AOT-Compatibility.ps1
         arguments: >-
           -ServiceDirectory ${{ parameters.ServiceDirectory }}
-          -PackageName ${{ aotTestInput.ArtifactName }}
-          -ExpectedWarningsFilePath ${{ aotTestInput.ExpectedWarningsFilePath }}
+          -PackageName ${{ artifact.name }}
+          -ExpectedWarningsFilePath ${{ parameters.ExpectedAOTWarnings }}
         workingDirectory: $(Build.SourcesDirectory)/eng/scripts/compatibility
@@ -13,8 +13,8 @@ parameters:
     default: all
   - name: CheckAOTCompat
     type: boolean
-    default: false
-  - name: AOTTestInputs
+    default: true
+  - name: ExpectedAOTWarnings
     type: object
     default: []
   - name: BuildSnippets
@@ -155,7 +155,7 @@ steps:
     - template: /eng/pipelines/templates/steps/aot-compatibility.yml
       parameters:
         ServiceDirectory: ${{ parameters.ServiceDirectory }}
-        AOTTestInputs: ${{ parameters.AOTTestInputs }}
+        ExpectedAOTWarnings: ${{ parameters.ExpectedAOTWarnings }}
         PackageInfoFolder: '$(Build.ArtifactStagingDirectory)/PackageInfo'
 
   - task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0

@@ -49,25 +49,31 @@ function Get-AllPackageInfoFromRepo($serviceDirectory)
     $ciProps = $pkgProp.GetCIYmlForArtifact()
 
     if ($ciProps) {
-      # CheckAOTCompat is opt _in_, so we should default to false if not specified
+      # CheckAOTCompat is opt _out_, so we should default to true if not specified
       $shouldAot = GetValueSafelyFrom-Yaml $ciProps.ParsedYml @("extends", "parameters", "CheckAOTCompat")
+      $parsedAOTBool = $true
       if ($null -ne $shouldAot) {
-        $parsedBool = $null
-        if ([bool]::TryParse($shouldAot, [ref]$parsedBool)) {
-          $pkgProp.CIParameters["CheckAOTCompat"] = $parsedBool
+        if ([bool]::TryParse($shouldAot, [ref]$parsedAOTBool)) {
+          $pkgProp.CIParameters["CheckAOTCompat"] = $parsedAOTBool
         }
-
-        # when AOTCompat is true, there is an additional parameter we need to retrieve
+      }
+      else {
+        $pkgProp.CIParameters["CheckAOTCompat"] = $true
+      }
+
+      # Initialize AOTTestInputs as empty array by default
+      $pkgProp.CIParameters["ExpectedAOTWarnings"] = @()
+
+      # Check for AOTTestInputs if shouldAot is null or true
+      if ($null -eq $shouldAot -or $parsedAOTBool) {
         $aotArtifacts = GetValueSafelyFrom-Yaml $ciProps.ParsedYml @("extends", "parameters", "AOTTestInputs")
         if ($aotArtifacts) {
-          $aotArtifacts = $aotArtifacts | Where-Object { $_.ArtifactName -eq $pkgProp.ArtifactName }
-          $pkgProp.CIParameters["AOTTestInputs"] = $aotArtifacts
+          $matchingArtifact = $aotArtifacts | Where-Object { $_.ArtifactName -eq $pkgProp.ArtifactName }
+          if ($matchingArtifact) {
+            $pkgProp.CIParameters["ExpectedAOTWarnings"] = $matchingArtifact
+          }
         }
       }
-      else {
-        $pkgProp.CIParameters["CheckAOTCompat"] = $false
-        $pkgProp.CIParameters["AOTTestInputs"] = @()
-      }
 
       # BuildSnippets is opt _out_, so we should default to true if not specified
       $shouldSnippet = GetValueSafelyFrom-Yaml $ciProps.ParsedYml @("extends", "parameters", "BuildSnippets")
@@ -84,8 +90,8 @@ function Get-AllPackageInfoFromRepo($serviceDirectory)
     # if the package isn't associated with a CI.yml, we still want to set the defaults values for these parameters
     # so that when we are checking the package set for which need to "Build Snippets" or "Check AOT" we won't crash due to the property being null
     else {
-      $pkgProp.CIParameters["CheckAOTCompat"] = $false
-      $pkgProp.CIParameters["AOTTestInputs"] = @()
+      $pkgProp.CIParameters["CheckAOTCompat"] = $true
+      $pkgProp.CIParameters["ExpectedAOTWarnings"] = @()
       $pkgProp.CIParameters["BuildSnippets"] = $true
     }
 

@@ -0,0 +1,107 @@
+<#
+.SYNOPSIS
+    Finds all packages in ci.yml files with name entries within the Artifacts section.
+
+.PARAMETER PackageName
+    The PackageName to collect AOT warnings for.
+
+.PARAMETER ServiceDirectory
+    The ServiceDirectory holding the package (e.g., "core", "identity").
+
+.PARAMETER DirectoryName
+    Optional. The name of a specific directory to scan within the service directory.
+    This is assumed to be PackageName if not provided.
+#>
+
+param (
+    [string]$PackageName,
+    [string]$ServiceDirectory,
+    [string]$DirectoryName
+)
+
+
+Write-Host "`nProcessing $($package.PackageName) in $($package.ServiceDirectory)..." -ForegroundColor Yellow
+
+    # Construct path to the Check-AOT-Compatibility.ps1 script
+    $checkAotScriptPath = Join-Path -Path $PSScriptRoot -ChildPath "Check-AOT-Compatibility.ps1"
+
+    # Construct the path to the package directory following the structure: repoRoot/sdk/serviceDirectory/packageName
+    $packageDir = Join-Path -Path (Join-Path -Path $sdkDir -ChildPath $package.ServiceDirectory) -ChildPath $package.PackageName
+
+    # Create the compatibility directory for the output
+    $testsDir = Join-Path -Path $packageDir -ChildPath "tests"
+    $compatibilityDir = Join-Path -Path $testsDir -ChildPath "compatibility"
+
+    # Create directory if it doesn't exist
+    if (!(Test-Path $testsDir)) {
+        New-Item -ItemType Directory -Path $testsDir -Force | Out-Null
+        Write-Host "  Created directory: $testsDir" -ForegroundColor Gray
+    }
+
+    if (!(Test-Path $compatibilityDir)) {
+        New-Item -ItemType Directory -Path $compatibilityDir -Force | Out-Null
+        Write-Host "  Created directory: $compatibilityDir" -ForegroundColor Gray
+    }
+      # Determine the output file path
+    $outputFilePath = Join-Path -Path $compatibilityDir -ChildPath "ExpectedWarnings.txt"
+      # Run the AOT compatibility check and extract warnings
+    Write-Host "  Running Check-AOT-Compatibility.ps1 for $($package.PackageName)..." -ForegroundColor Gray
+    try {        # Run the compatibility check with "None" to force it to report all warnings and capture all output
+        $output = & $checkAotScriptPath -ServiceDirectory $package.ServiceDirectory -PackageName $package.PackageName -ExpectedWarningsFilePath "None" *>&1
+
+        # Split the output into lines and find warnings
+        $outputLines = $output -split "`r`n"
+        $warningLines = @()
+
+        if ($warningLines.Count -eq 0) {
+            $warningLines = $outputLines | Where-Object { $_ -match "IL\d+:" }
+        }
+
+        # Process each warning line
+        $warningPatterns = @()
+        foreach ($line in $warningLines) {
+            # Clean up the line - remove color codes and normalize whitespace
+            $cleanLine = $line -replace '\e\[\d+(;\d+)*m', '' -replace '\s+', ' ' -replace '^\s+|\s+$', ''
+
+            # Remove filepath in brackets at the end if it exists
+            $cleanLine = $cleanLine -replace '\s+\[.+?\]$', ''
+              # Handle filepath at the beginning (replace with .* up to package name)
+            if ($cleanLine -match "\\sdk\\.*?\\($($package.PackageName))\\") {
+                $cleanLine = $cleanLine -replace "^.*\\sdk\\.*?\\($($package.PackageName))", ".*$($package.PackageName)"
+            }
+
+            # Replace line numbers with \d*
+            $cleanLine = $cleanLine -replace '\(\d+\)', '(\d*)'
+
+            # Escape special regex characters (except those we want to keep as wildcards)
+            $escapedLine = [regex]::Escape($cleanLine)
+
+            $escapedLine = $escapedLine -replace '\\\\d\\\*', '\d*'  # Keep \d* wildcards
+            $escapedLine = $escapedLine -replace '\\.\\\*', '.*'  # Keep .* wildcards
+
+
+            $warningPatterns += $escapedLine
+        }
+
+        # Save the warning patterns to the file
+        if ($warningPatterns.Count -gt 0) {
+            $warningPatterns | Out-File -FilePath $outputFilePath -Force
+            Write-Host "  Saved $($warningPatterns.Count) warning patterns to: $outputFilePath" -ForegroundColor Green
+        } else {
+            # Create an empty file if no warnings found
+            Write-Host "  No warnings found." -ForegroundColor Yellow
+        }
+
+        $successfulPackages++
+    } catch {
+        Write-Warning "  Error running Check-AOT-Compatibility.ps1 for $($package.PackageName): $_"
+        $failedPackages++
+    }
+
+# Display summary of results
+Write-Host "`nAOT compatibility check summary:" -ForegroundColor Cyan
+Write-Host "  Total packages processed: $processedPackages" -ForegroundColor White
+Write-Host "  Successful: $successfulPackages" -ForegroundColor Green
+Write-Host "  Failed: $failedPackages" -ForegroundColor $(if ($failedPackages -gt 0) { "Red" } else { "Green" })
+
+Write-Host "`nCompleted AOT compatibility checks for all packages" -ForegroundColor Green