Pipeline Performance Optimization: Make Static Analysis Run Parallelly with Build

[MuyuanMS]

This PR shortens build time by around 50 minutes for foundation repo, while only increase absolute build time by 25 minutes.

Below are running results for Foundation PR Build:

• Before Change

  • Total Build Time: 2h 26m
  • Absolute Build Time (Sum of All jobs): 582 min
    

• After Change

  • Total Build Time: 1h 27m (-40%)
  • Absolute Build Time (Sum of All jobs): 607 min (+4.3%)
    

Currently largest bottleneck building foundation is static analysis, which can be run independently but currently blocks dependent stage from running. After this change, now during build, two build stage will start in parallel - one without static analysis to allow dependent steps like Test or Pack to run, the other with static analysis

---

A microsoft employee must use /azp run to validate using the pipelines below.

WARNING:
Comments made by azure-pipelines bot maybe inaccurate.
Please see pipeline link to verify that the build is being ran.

For status checks on the main branch, please use TransportPackage-Foundation-PR
(https://microsoft.visualstudio.com/ProjectReunion/_build?definitionId=81063&_a=summary)
and run the build against your PR branch with the default parameters.

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines failed to run 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines failed to run 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

/azp run

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

This is great work. I think there is one thing we need to address, though. The existing way to avoid spending too much time on PREfast is to simply run PREfast only for x64fre. Please see this code in WindowsAppSDK-BuildMRT-Steps.yml and WindowsAppSDK-BuildFoundation-Steps.yml:

• task: SDLNativeRules@3
  ...
  condition: and(succeeded(), eq(variables['buildPlatform'], 'x64'), eq(variables['buildConfiguration'], 'Release'))

Therefore, the Build_x86_StaticAnalysis stage is basically doing the same work as the Build_x86 stage, and that explains why they take similar time to finish. The same story applies to the Build_arm64_StaticAnalysis stage and the Build_arm64 stage. In contrast, the Build_arm64_StaticAnalysis stage takes much longer to run than Build_arm64 due to running PREfast, as expected.

I see 2 basic options: 1) continue to only run PREfast on x64fre only, that means we skip the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages, which hopefully will allow the pipeline to finish even sooner, 2) adjust the code cited above to enable running PREfast for x86 and arm64 in the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages. This is expected to increase the "total new workload" of the pipeline and is thus likely to reduce the performance gain. IMHO, #1 is a legitimate choice because it does not regress today's pipeline behavior. But if we prefer to data to support the decision, we can also first prototype #2 to confirm that running PREfast only for x64 only is "good enough" (we initially did verify our PREfast results are basically the same for x86, x64, and arm64, but that was a couple of years ago, things might have changed, probably worth another check). If the PREfast results are good, we should feel more comfortable going for option #1. Just my 2 cents. Hence, I expect this PR to take more changes. If so, I suggest rebasing this PR on the branch with the merged PR Performance Optimization for Foundation Build Pipeline: Break down build and test stage per platform by haonanttt · Pull Request #5363 · microsoft/WindowsAppSDK in place, to simplify this PR. Thank you very much!

This is great work. I think there is one thing we need to address, though. The existing way to avoid spending too much time on PREfast is to simply run PREfast only for x64fre. Please see this code in WindowsAppSDK-BuildMRT-Steps.yml and WindowsAppSDK-BuildFoundation-Steps.yml:

• task: SDLNativeRules@3
  ...
  condition: and(succeeded(), eq(variables['buildPlatform'], 'x64'), eq(variables['buildConfiguration'], 'Release'))

Therefore, the Build_x86_StaticAnalysis stage is basically doing the same work as the Build_x86 stage, and that explains why they take similar time to finish. The same story applies to the Build_arm64_StaticAnalysis stage and the Build_arm64 stage. In contrast, the Build_arm64_StaticAnalysis stage takes much longer to run than Build_arm64 due to running PREfast, as expected.

I see 2 basic options: 1) continue to only run PREfast on x64fre only, that means we skip the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages, which hopefully will allow the pipeline to finish even sooner, 2) adjust the code cited above to enable running PREfast for x86 and arm64 in the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages. This is expected to increase the "total new workload" of the pipeline and is thus likely to reduce the performance gain. IMHO, #1 is a legitimate choice because it does not regress today's pipeline behavior. But if we prefer to data to support the decision, we can also first prototype #2 to confirm that running PREfast only for x64 only is "good enough" (we initially did verify our PREfast results are basically the same for x86, x64, and arm64, but that was a couple of years ago, things might have changed, probably worth another check). If the PREfast results are good, we should feel more comfortable going for option #1. Just my 2 cents. Hence, I expect this PR to take more changes. If so, I suggest rebasing this PR on the branch with the merged PR Performance Optimization for Foundation Build Pipeline: Break down build and test stage per platform by haonanttt · Pull Request #5363 · microsoft/WindowsAppSDK in place, to simplify this PR. Thank you very much!

Currently I tried to go with option 2 (Results: https://dev.azure.com/microsoft/ProjectReunion/_build/results?buildId=121725042&view=results), given that PREfast has been moved to a separate stage and they can run parallelly if resource allows. If we agree on running prefast for all three archs, I'll proceed to make the code changes in building MRT steps as well. Do you know how I can confirm whether PREfast results are the same for x86, x64, and arm64? (Just by build log, or do I actually have to manually make code changes to fail some PREfast checks and compare results?)

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[MuyuanMS]

I see 2 basic options: 1) continue to only run PREfast on x64fre only, that means we skip the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages, which hopefully will allow the pipeline to finish even sooner, 2) adjust the code cited above to enable running PREfast for x86 and arm64 in the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages.

@alexlamtest I have updated current PR with option 1. Below are some of the results while making the decision:

• Original flow
  • Built Time: 2h 26m 32s
  • Sum of All Jobs' Time: 582 min
• Run PREfast for all 3 archs
  • Built Time: 1h 40m 25s (-31.5%)
  • Sum of All Jobs' Time: 793 min (+36.3%)
• Run PREfast for x64
  • Built Time: 1h 27m 36s (-40%)
  • Sum of All Jobs' Time: 607 min (+4.3%)

[MuyuanMS]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[yeelam-gordon]

I see 2 basic options: 1) continue to only run PREfast on x64fre only, that means we skip the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages, which hopefully will allow the pipeline to finish even sooner, 2) adjust the code cited above to enable running PREfast for x86 and arm64 in the Build_x86_StaticAnalysis and Build_arm64_StaticAnalysis stages.

@alexlamtest I have updated current PR with option 1. Below are some of the results while making the decision:

• Original flow

  • Built Time: 2h 26m 32s
  • Sum of All Jobs' Time: 582 min

• Run PREfast for all 3 archs

  • Built Time: 1h 40m 25s (-31.5%)
  • Sum of All Jobs' Time: 793 min (+36.3%)

• Run PREfast for x64

  • Built Time: 1h 27m 36s (-40%)
  • Sum of All Jobs' Time: 607 min (+4.3%)

@alexlamtest, I will suggest let Muyuan to checkin the one only improve performance first, which the result is promising and without increase much build agent cost (i.e. 4.3% only)
And if we believe it is necessary to run statistic analysis against all the build architect, it should be an easy change later on.