System.Text.RegularExpressions.Tests Fails with 'Language C# is not supported` Error when TestReadyToRun is Enabled

[ivdiazsa]

Description

When building and running System.Text.RegularExpressions.Tests with TestReadyToRun=true, the test run will crash and fail with an error about the language C# not being supported.

Reproduction Steps

1. Clone the repo and build the clr and libs subsets. Make sure you specify TestReadyToRun, so that some necessary pretest components get built.

git clone https://github.com/dotnet/runtime
cd runtime
./build.sh --subset clr+libs --configuration Release -p:TestReadyToRun=true

2. Build and run the test

cd src/libraries/System.Text.RegularExpressions/tests/FunctionalTests
/path/to/runtime/dotnet.sh build -c Release -t:Test -p:TestReadyToRun=true

Expected behavior

All tests pass, except the skipped ones.

Actual behavior

[FAIL] System.Text.RegularExpressions.Tests.RegexKnownPatternTests.Docs_GroupingConstructs_ZeroWidthPositiveLookbehindAssertions(engine: SourceGenerated)
System.NotSupportedException : The language 'C#' is not supported.
   at Microsoft.CodeAnalysis.Host.HostWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.Host.Mef.MefWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.SolutionState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.SolutionCompilationState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:line 193
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:line 133
   at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.Tests.Common.cs:line 117
   at System.Text.RegularExpressions.Tests.RegexKnownPatternTests.Docs_GroupingConstructs_ZeroWidthPositiveLookbehindAssertions(RegexEngine engine) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.KnownPattern.Tests.cs:line 443
--- End of stack trace from previous location ---
Finished System.Text.RegularExpressions.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51

It is worth noting there are lots (1,000+) failures and this output extract is just one example. However, all of them are with the same error.

Regression?

No

Known Workarounds

None, other than keeping the test disabled in tests.proj.

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When building and running System.Text.RegularExpressions.Tests with TestReadyToRun=true, the test run will crash and fail with an error about the language C# not being supported.

Reproduction Steps

1. Clone the repo and build the clr and libs subsets. Make sure you specify TestReadyToRun, so that some necessary pretest components get built.

git clone https://github.com/dotnet/runtime
cd runtime
./build.sh --subset clr+libs --configuration Release -p:TestReadyToRun=true

2. Build and run the test

cd src/libraries/System.Text.RegularExpressions/tests/FunctionalTests
/path/to/runtime/dotnet.sh build -c Release -t:Test -p:TestReadyToRun=true

Expected behavior

All tests pass, except the skipped ones.

Actual behavior

[FAIL] System.Text.RegularExpressions.Tests.RegexKnownPatternTests.Docs_GroupingConstructs_ZeroWidthPositiveLookbehindAssertions(engine: SourceGenerated)
System.NotSupportedException : The language 'C#' is not supported.
   at Microsoft.CodeAnalysis.Host.HostWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.Host.Mef.MefWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.SolutionState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.SolutionCompilationState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:line 193
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/RegexGeneratorHelper.netcoreapp.cs:line 133
   at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.Tests.Common.cs:line 117
   at System.Text.RegularExpressions.Tests.RegexKnownPatternTests.Docs_GroupingConstructs_ZeroWidthPositiveLookbehindAssertions(RegexEngine engine) in /path/to/runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.KnownPattern.Tests.cs:line 443
--- End of stack trace from previous location ---
Finished System.Text.RegularExpressions.Tests, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51

Regression?

No

Known Workarounds

None, other than keeping the test disabled in tests.proj.

Author:	ivdiazsa
Assignees:	-
Labels:	area-System.Text.RegularExpressions, disabled-test, test-failure
Milestone:	Future

[stephentoub]

What does TestReadyToRun do?

[ivdiazsa]

It is a new mode that allows to run libraries tests compiled with ReadyToRun enabled. The original PR #75230 goes more in depth about the specifics of this mode.

[stephentoub]

It is a new mode that allows to run libraries tests compiled with ReadyToRun enabled. The original PR #75230 goes more in depth about the specifics of this mode.

I assume this issue then is about fixing that mode's handling of various things or disabling tests that for whatever reason are incompatible with it? This doesn't seem actionable from a regex perspective.

[ivdiazsa]

I'm not sure I'm understanding. I was under the impression that in our product (dotnet), we should be able to run apps and get the same results, regardless of whether it is jitted or precompiled. Differences would come in size, performance, etc, but the app should return the same results. Or am I misunderstanding?

I ask because if this is not the case (me misunderstanding), then the test should be able to work under R2R as well right?

[stephentoub]

However this is packaged / executed / built is doing something that's causing assemblies to behave differently. The failures here aren't coming from Regex; they're coming from Roslyn (the regex tests are loading Roslyn to test the source generator):

System.NotSupportedException : The language 'C#' is not supported.
   at Microsoft.CodeAnalysis.Host.HostWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.Host.Mef.MefWorkspaceServices.GetLanguageServices(String languageName)
   at Microsoft.CodeAnalysis.SolutionState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.SolutionCompilationState.AddProject(ProjectInfo projectInfo)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectInfo projectInfo)

[ivdiazsa]

Ok I understand, thanks for the explanation Stephen. I have another question I would like to hear your opinion on. Would you consider this a bug in Roslyn and/or ReadyToRun, or would you consider it one of those cases where the test is simply incompatible and not running it is the way to go?

[stephentoub]

I don't know enough about what TestReadyToRun is actually doing to answer that. Can you point me to a write-up about it and/or to its implementation?

[ivdiazsa]

Can you help us real quick here with some deeper insight on R2R? @trylek @davidwrighton

[davidwrighton]

@ivdiazsa You'll need to debug the test, and see what's happening. My suspicion is that some dll isn't loadable into the process for some reason. It will probably be throwing some sort of exception from an assembly load api. You will need to identify which dll is failing to load, and preferably once you have done so, stop the whole thing in the debugger and get one of us to take a look at what is happening.