BI going red even when all errors are known errors

[premun]

https://github.com/dotnet/runtime/runs/100537181991

[premun]

I investigated check run https://github.com/dotnet/runtime/runs/100537181991 in Build Insights production telemetry.

The analysis itself completed successfully. It analyzed all 3 failing tests and all 3 associated Helix logs, and all 3 failures matched known issues. The resulting telemetry recorded both outcomes:

• checkResult = Failed
• checkResultWithKnownIssues = Passed

The check was nevertheless published as failed because the effective ShouldMergeOnFailureWithKnownIssues repository policy for dotnet/runtime was disabled (the repository default is explicitly False). CheckResultService.GetCheckResult() therefore selected the raw Azure DevOps build result instead of the known-issue-adjusted result, and GitHubChecksProvider mapped it to GitHub's Failure conclusion.

So this was expected policy behavior rather than an unknown problem or failed analysis: known issues are reported, but they do not make the runtime check green unless that repository policy is enabled or the /ba-g override is used.

The 409 ContainerAlreadyExists storage traces seen during processing were benign; the work item completed successfully and posted the check normally.

[premun]

I also compared this behavior with the previous implementation in dotnet-helix-service.

The default behavior for dotnet/runtime would not have been different. The legacy service used effectively the same decision path:

• MergedBuildAnalysisProvider.GetMergeOnKnownIssuesConfiguration() read the repository/branch configuration and used repoConfig?.ShouldMergeOnFailureWithKnownIssues ?? false.
• CheckResultProvider.GetCheckResult() selected the known-issue-adjusted result only when that flag was true; otherwise it returned the raw build result.
• The legacy unit tests explicitly expect a build containing only known failures to be Passed with the flag enabled and Failed with it disabled.

The configuration storage changed, but the fallback did not:

Service	Configuration storage	Missing-config default
Legacy Helix service	Azure Table Storage, with repository/branch rows and a * branch fallback	false
Current Build Insights	SQL repository configuration, with branch-specific and repository-wide fallback	false

The current service also seeds dotnet/runtime explicitly with ShouldMergeOnFailureWithKnownIssues = False. That value was already present when Build Insights was initially moved out of Arcade Services; it was not introduced by the later configuration-model refactoring.

Therefore, unless the legacy production Azure Table contained an explicit dotnet-runtime configuration row setting this flag to true, the old service would also have published this check as failed. I could not verify that historical production row directly because my current identity does not have Storage Table Data Reader access, so that is the only remaining caveat; the legacy code fallback and the migrated runtime configuration are both definitively false.