Race in Roslyn API for source generated files causing them to get stuck

[TessenR]

Version Used:

Microsoft Visual Studio Professional
Version 17.14.16
Version 17.14.16 Preview 1.0
Version: 18.0.0 Insiders [11104.47]

Steps to Reproduce:

1. Subscribe to WorkspaceChanged event either directly or via RegisterWorkspaceChangedHandler
2. Call Microsoft.CodeAnalysis.Project.GetSourceGeneratedDocumentsAsync on projects from Worksspace.CurrentSolution in response to some workspace events like SolutionChanged

Expected Behavior:
You can safely request source generated files via the public Roslyn API

Actual Behavior:
Sometimes, calling Project.GetSourceGeneratedDocumentsAsync causes Roslyn to cache an outdated version of source generators until some future event like solution build forces an updated.
All subsequent calls to Project.GetSourceGeneratedDocumentsAsync return outdated version.
This affect VS's own code analysis as it will see outdated source generated documents, e.g. after navigating to a file that uses them

Notes:
I did some investigation and it looks like the following race happens inside Roslyn:

• There are 2 concurrent activities in Roslyn:
  • Workspace.EnqueueUpdateSourceGeneratorVersion in response to finishing a build
  • ProjectSystemProjectFactory.StartRefreshingAnalyzerReferenceForFileAsync in response to a file system event about changed analyzer reference assembly
• During build, Workspace.EnqueueUpdateSourceGeneratorVersion tirggers first and updates source generators' version for the solution.
• Here, if you call Project.GetSourceGeneratedDocumentsAsync Roslyn will use the old version of the generator but will cache it with the updated SourceGeneratorExecutionVersion since it was already incremented by the previous step
• Afterwards, a file system watcher event triggers ProjectSystemProjectFactory.StartRefreshingAnalyzerReferenceForFileAsync which reloads the generator without incrementing SourceGeneratorExecutionVersion
• All future requests to Project.GetSourceGeneratedDocumentsAsync will return the cached outdated version of source generated documents

It looks impossible to tell when it's safe to call Project.GetSourceGeneratedDocumentsAsync because
there's no guarantee when an async refresh of an analyzer reference file will happen and whether it will happen at all after some workspace change

It's also not possible to determine that such a reload happened. It sends a ProjectChanged workspace change event, but checking changes between old and new solution returns empty diff as the project just removes and adds the same analyzer reference in a single modification.

[CyrusNajmabadi]

We could consider a workspace event for this. But the rest of the behavior is by design. Fundamentally, it is now the host that controls if SGs should rerun when in balanced mode.

With a specific event you could at least know when this happened and that SG docs were stale.

[TessenR]

We could consider a workspace event for this. But the rest of the behavior is by design. Fundamentally, it is now the host that controls if SGs should rerun when in balanced mode.

@CyrusNajmabadi Is it by design that if I use public Roslyn API to obtain source generated documents it can randomly break Roslyn's own functions like code analysis? Is there any safe way to get them?

Consider the following solution:
SourceGeneratorTest.zip

• open the solution
• rebuild solution
• navigate to Program.cs

if you don't call Roslyn API to get source generated documents, Roslyn will correctly recognize the change and highlight the code as correct because the first call for source generated documents arrives after it reloads the source generator assembly.
If you called Roslyn's public API to get source generated documents, Roslyn will say that the generator doesn't generate any files in solution explorer, highlight code as erroneous and show errors in the error list because it already cached the output of the previous (missing) version as the "up to date" one