Jetbrains Rider : Steps not detected

[AlexTeixeira]

Hi,

Thanks for your great plugin.
I'm currently using Rider as development IDE, and seems that the test are not discovered by the runner (I test with MSTest and Xunit with default settings).

I don't know if is a bug or a lack of the IDE. So I'm just asking if it should work with Rider ?

[DaveAurionix]

Hi Alex,

Thanks for raising that question. We don't use Rider (currently) so the test adapter hasn't been tested with that IDE. Having said that, I would hope that Rider used the same process as dotnet test so it should "just work".

One thing to try - if you specifically rebuild just your test project (instead of building the whole solution), does the runner suddenly detect the tests?

It's also possible that the tests may not appear in the test runner if an exception happens during test discovery phase. We had hoped that we caught and handled that, but maybe we've missed something. Visual Studio has an output window showing test discovery and execution logs, do you know if Rider has the same concept? If so, is there anything interesting in the output (like an exception and stacktrace)?

Thanks,
Dave.

[AlexTeixeira]

Hi,

I am using the example provided in the documentation : https://github.com/GivePenny/GherkinSpec.SimpleExample

I create a second Unit Test project with XUnit Template :

The runner discovers only the base unit test from Xunit project :

And the log file generated when I run the test from the play button :

14:43:56.672 |T| Launch: af961c47-a191-4008-b3c0-7ab5a925d976 >> RunHostController.AfterLaunchStarted
14:43:56.672 |T| Launch: af961c47-a191-4008-b3c0-7ab5a925d976 << RunHostController.AfterLaunchStarted
14:43:56.672 |I| BuildStage started
14:43:56.673 |I| Build policy: Automatic
14:43:57.227 |I| 0 projects built
14:43:57.227 |I| BuildStage completed
14:43:57.227 |I| RefreshProjectPropertiesStage started
14:43:57.228 |I| RefreshProjectPropertiesStage completed
14:43:57.228 |I| DiscoveryStage started
14:43:57.228 |I| Test exploration required for 0 projects
14:43:57.228 |I| DiscoveryStage completed
14:43:57.228 |I| BuildPipelineStage started
14:43:57.228 |I| BuildPipelineStage completed
14:43:57.228 |I| ApplyRulesStage started
14:43:57.230 |V| Running elements from criterion: {ConjunctiveCriterion: {SolutionCriterion}, {ProjectCriterion: C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit}}
14:43:57.230 |I| Got 2 elements (0 explicit) to run
14:43:57.231 |V| Marking 2 elements as Pending
14:43:57.231 |V| Marking elements as Pending completed
14:43:57.231 |V| Distributing 2 elements into runs
14:43:57.234 |V| Distributing elements completed
14:43:57.234 |I| Created 1 unit-test runs
14:43:57.234 |I| ApplyRulesStage completed
14:43:57.234 |I| RunTestsStage started
14:43:57.234 |I| Running up to 1 unit-test runs in parallel
14:43:57.234 |I| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d - Starting
14:43:57.234 |V|     Provider: xUnit.net
    Target Framework: .NETCoreApp,Version=v2.2
    Strategy: XunitDotNetVsTestRunStrategy
    Runtime Enviroment: DotNetCoreRuntimeEnvironment
  Project: GherkinSpec.SimpleExample.XUnit
  TargetFrameworkId: .NETCoreApp,Version=v2.2
  TargetPlatform: X64
14:43:57.234 |T| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d >> RunHostController.PrepareForRun
14:43:57.234 |T| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d << RunHostController.PrepareForRun
14:43:57.234 |T| Register handler for message type 'TestSession.Message', protocol version '1'
14:43:57.234 |T| Register handler for message type 'ProtocolVersion', protocol version '<unset>'
14:43:57.234 |T| Register handler for message type 'TestSession.Connected', protocol version '<unset>'
14:43:57.234 |T| Register handler for message type 'TestExecution.CustomTestHostLaunch', protocol version '1'
14:43:57.234 |T| Register handler for message type 'TestExecution.Completed', protocol version '1'
14:43:57.234 |T| Register handler for message type 'TestExecution.StatsChange', protocol version '1'
14:43:57.236 |I| [ProcessStartInfoPatchResult]
  Original request: DotNetCoreRequest
  Success: True
  ErrorMessage: <Empty>
  Original StartInfo: FileName = 'vstest', Arguments = '/port:54121  ', WorkDir = 'C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit'
  Patched  StartInfo: FileName = 'C:\Program Files\dotnet\dotnet.exe', Arguments = '"vstest" /port:54121  ', WorkDir = 'C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit'

14:43:57.236 |I| Starting process: C:\Program Files\dotnet\dotnet.exe "vstest" /port:54121  
14:43:57.243 |I| Started process with pid 15788
14:43:57.607 |T| Processing message 'TestSession.Connected'
14:43:57.607 |T| Processed message 'TestSession.Connected'
14:43:57.614 |T| Processing message 'ProtocolVersion'
14:43:57.614 |T| Processed message 'ProtocolVersion'
14:43:57.615 |I| Using test adapter: C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\xunit.runner.visualstudio.dotnetcore.testadapter.dll
14:43:57.615 |T| RunSettings: 
<?xml version='1.0' encoding='utf-8'?>
<RunSettings>
  <RunConfiguration>
    <TargetFrameworkVersion>.NETCoreApp,Version=v2.2</TargetFrameworkVersion>
    <TargetPlatform>X64</TargetPlatform>
  </RunConfiguration>
</RunSettings>
14:43:57.713 |T| Processing message 'TestExecution.CustomTestHostLaunch'
14:43:57.713 |T| Processed message 'TestExecution.CustomTestHostLaunch'
14:43:57.713 |I| [ProcessStartInfoPatchResult]
  Original request: DirectRequest
  Success: True
  ErrorMessage: <Empty>
  Original StartInfo: FileName = 'C:\Program Files\dotnet\dotnet.exe', Arguments = 'exec --runtimeconfig "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.runtimeconfig.json" --depsfile "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.deps.json" "C:\Users\dev4\.nuget\packages\microsoft.testplatform.testhost/16.2.0-preview-20190606-02\lib/netstandard1.5/testhost.dll" --port 54123 --endpoint 127.0.0.1:054123 --role client --parentprocessid 6656 --telemetryoptedin false', WorkDir = 'C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2'
  Patched  StartInfo: FileName = 'C:\Program Files\dotnet\dotnet.exe', Arguments = 'exec --runtimeconfig "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.runtimeconfig.json" --depsfile "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.deps.json" "C:\Users\dev4\.nuget\packages\microsoft.testplatform.testhost/16.2.0-preview-20190606-02\lib/netstandard1.5/testhost.dll" --port 54123 --endpoint 127.0.0.1:054123 --role client --parentprocessid 6656 --telemetryoptedin false', WorkDir = 'C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2'

14:43:57.713 |I| Starting process: C:\Program Files\dotnet\dotnet.exe exec --runtimeconfig "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.runtimeconfig.json" --depsfile "C:\Users\dev4\Downloads\GherkinSpec.SimpleExample-master\GherkinSpec.SimpleExample.XUnit\bin\Debug\netcoreapp2.2\GherkinSpec.SimpleExample.XUnit.deps.json" "C:\Users\dev4\.nuget\packages\microsoft.testplatform.testhost/16.2.0-preview-20190606-02\lib/netstandard1.5/testhost.dll" --port 54123 --endpoint 127.0.0.1:054123 --role client --parentprocessid 6656 --telemetryoptedin false
14:43:57.720 |I| Started process with pid 17480
14:43:58.189 |T| Processing message 'TestSession.Message'
14:43:58.190 |T| Processed message 'TestSession.Message'
14:43:58.190 |I| [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27617.05)
14:43:58.581 |T| Processing message 'TestSession.Message'
14:43:58.581 |T| Processed message 'TestSession.Message'
14:43:58.581 |I| [xUnit.net 00:00:00.40]   Starting:    GherkinSpec.SimpleExample.XUnit
14:43:58.707 |T| Processing message 'TestSession.Message'
14:43:58.707 |T| Processed message 'TestSession.Message'
14:43:58.707 |I| [xUnit.net 00:00:00.52]   Finished:    GherkinSpec.SimpleExample.XUnit
14:43:58.794 |I| Process C:\Program Files\dotnet\dotnet.exe:17480 has exited with code (0)
Output stream: 
14:43:58.824 |T| Processing message 'TestExecution.StatsChange'
14:43:58.824 |T| Processed message 'TestExecution.StatsChange'
14:43:58.829 |T| Processing message 'TestExecution.Completed'
14:43:58.829 |T| Processed message 'TestExecution.Completed'
14:43:58.854 |I| Process C:\Program Files\dotnet\dotnet.exe:15788 has exited with code (0)
Output stream: Outil en ligne de commande d'ex‚cution de tests Microsoft (R), version 15.9.0
Copyright (c) Microsoft Corporation. Tous droits r‚serv‚s.

14:43:58.854 |T| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d >> RunHostController.CleanupAfterRun
14:43:58.854 |T| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d << RunHostController.CleanupAfterRun
14:43:58.854 |I| Run: 8d735274-39c0-4ac7-bbea-ace0998a6b8d - Finished
14:43:58.854 |I| C:\Program Files\dotnet\dotnet.exe exited
14:43:58.854 |I| RunTestsStage completed
14:43:58.854 |I| FinishLaunchStage started
14:43:58.855 |I| Finishing launch
14:43:58.855 |I| Launch is finished
14:43:58.855 |I| FinishLaunchStage completed
14:43:58.855 |T| Launch: af961c47-a191-4008-b3c0-7ab5a925d976 >> RunHostController.BeforeLaunchFinished
14:43:58.855 |T| Launch: af961c47-a191-4008-b3c0-7ab5a925d976 << RunHostController.BeforeLaunchFinished

[DaveAurionix]

@AlexTeixeira Thanks for sending through the detail and the screenshots. Did you try rebuilding the GherkinSpec.SimpleExample.Tests or GherkinSpec.SampleExample.XUnit projects by right-clicking on them in Rider and choosing the relevant build command (I'm guessing that Rider has a right-click menu with a Build command or similar)?

I ask because the test runner won't discover tests in those two projects until the projects have definitely been built. Launching the tests using the Play button will only run unit tests in built assemblies that the test runner has already discovered, so I don't think that clicking the Play button is enough.

[DaveAurionix]

Sorry, to clarify: the unit test adapters for MSTest, XUnit and I think NUnit can detect tests in projects before they are manually built. GherkinSpec cannot do this yet. If I recall correctly, I think I may have read something recently that indicates Microsoft are working on a change to make this easier for test adapter developers to support.

[AlexTeixeira]

Hi,

Yes, I try to rebuild each test project unitary. I also unload both project and reload them.
Yes rider has basically the same features as Visual Studio.

When I add a test in one of test project, the runner discovered the new test.
I think I will need for now to use Visual Studio for BDD :/

[DaveAurionix]

Thanks Alex - checking these logs again, it looks as if the test runner hasn't picked up the GherkinSpec test adapter at all. We'll leave this issue open as a feature request for GherkinSpec to support Rider. Anyone in the community could pick this up and send us a pull request, unless we at GivePenny get there sooner.

Thanks again for raising this.

[Dreagen]

For anyone still having this issue, it can be solved by enabling VSTest adapters support in Rider. From the Rider documentation:

Custom test frameworks that use Visual Studio test adapter extensibility.
To enable VSTest adapters support, select the corresponding checkbox on the Build, Execution, Deployment | Unit Testing |
VSTest page of JetBrains Rider settings Ctrl+Alt+S.

[mscottford]

For anyone still having this issue, it can be solved by enabling VSTest adapters support in Rider. From the Rider documentation:

Custom test frameworks that use Visual Studio test adapter extensibility.
To enable VSTest adapters support, select the corresponding checkbox on the Build, Execution, Deployment | Unit Testing |
VSTest page of JetBrains Rider settings Ctrl+Alt+S.

This worked for me, but only after I restarted Rider.