Skip to content

Commit 68576c1

Browse files
Test Explorer Support via VSTest (#1383)
* Set up the stub of a test explorer test project * Add the test explorer tests to the sln * Set up a trival test case to scaffold for VSTestAdapter work * Update Expecto for performance improvements * Demonstrate the most basic test discovery with vstest * Demonstrate connection between ionide and language server * Implement and mvp test discovery endpoint on the Language Server * Push test discovery down to AdaptiveState This is preparation for adding notifications for incremental discovery reporting * Publish incremental test discovery updates * Demonstration a test run using VSTest * Rename the namespace for test features. VSTestAdapter is too specific. We already know we'll need to expand to accomodate Microsoft.Testing.Platform * Add an lsp endpoint for running tests * Fix bug where the TestExplorer tests don't build the required sample projects * Demonstrate basic streaming of test run results from langauge server * Add test filtering to the VSTest wrapper * Add testCaseFilter support to test Runs Still has a bug where NUnit doesn't respect the filter, but works for all other frameworks * Prove that I can scrape test host process Ids Also required figuring out graceful cancellation behavior for test runs * Convert VsTestWrapper methods to async-only for graceful cancellation * Pass a server-level test for the attaching debugger to test run. * Format files * Delete some diagnostic code * fix incorrect message type on the runTests lsp message * Pass unit tests using RunTestsWithCustomTestHost RunTestsWithCustomTestHost is the official way to attach a debugger with VSTest * Debug test runs Broke out the debug attach message because it needs to be a blocking call. VSTest will continue executing the tests as soon as AttachDebuggerToProcess returns I also fixed the LSP level test explorer tests, which were failing because the sample test projects needed to be built at part of the unit test, since the build artifacts are being cleared before every test run * Test that client environment variables are available in a test run * Decouple SampleTestProjects for Lsp and TestExplorer test projects FsAutoComplete.Tests.TestExplorer expects the sample projects to be built once before the tests are run, but FsAutoComplete.Tests.Lsp expects the sample projects to be build for every test. Splitting up the sample projects reduces chances that they could break each other when the run in parallel or in random order * Forward test logs * Allow client to limit which test projects are run This was mainly motivated by test debugging. Before this feature, the debugger would launch for every test project even if you only wanted to debug one test. With this change, it'll only launch the one project. This also partially mitigates the bug where NUnit doesn't respect filters, now those test won't show unless some tests from an nunit project are selected * Workaround for NUnit not respecting test filters NUnit doesn't respect test filters when VSTest is in DesignMode, which it is by default with VsTestConsoleWrapper #1383 (comment) * Forward test discovery logs to the client for improved error diagnostics * Show error notification in client for expectable test discovery issues (i.e projects not restored or built) * Separate discovery vs run lsp-level test explorer tests * Split the VSTest wrapper code from TestServer contracts for clarity * Clarify dictionary name (projectLookup -> projectsByBinaryPath) * Satisfy fantomas formatting check * Remove print statements per copilot review * Add OpenTelemetry to the Test Explorer langauge server endpoints * Avoid duplicate project load when discovery test projects * Fix net9 build --------- Co-authored-by: Jimmy Byrd <[email protected]>
1 parent 19bc8e5 commit 68576c1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+1657
-210
lines changed

FsAutoComplete.sln

Lines changed: 155 additions & 84 deletions
Large diffs are not rendered by default.

paket.dependencies

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ nuget Microsoft.NET.Test.Sdk >= 17.4
4444
nuget Dotnet.ReproducibleBuilds copy_local:true
4545

4646
nuget Ionide.KeepAChangelog.Tasks copy_local: true
47+
nuget Expecto ~> 10
4748
nuget Expecto.Diff
4849
nuget YoloDev.Expecto.TestSdk
4950
nuget AltCover
@@ -58,3 +59,8 @@ nuget CommunityToolkit.HighPerformance
5859
nuget System.Security.Cryptography.Pkcs
5960
nuget System.Net.Http
6061
nuget System.Text.RegularExpressions
62+
63+
64+
## Test Explorer
65+
nuget Microsoft.TestPlatform.TranslationLayer
66+
nuget Microsoft.TestPlatform.ObjectModel

paket.lock

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ NUGET
3636
Serilog (>= 3.1.1)
3737
DiffPlex (1.7.2) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= net6.0)) (&& (== netstandard2.1) (>= net6.0))
3838
Dotnet.ReproducibleBuilds (1.2.25) - copy_local: true
39-
Expecto (10.2.1) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= net6.0)) (&& (== netstandard2.1) (>= net6.0))
39+
Expecto (10.2.3)
4040
FSharp.Core (>= 7.0.200) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= net6.0)) (&& (== netstandard2.1) (>= net6.0))
4141
Mono.Cecil (>= 0.11.4 < 1.0) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= net6.0)) (&& (== netstandard2.1) (>= net6.0))
4242
Expecto.Diff (10.2.1)
@@ -400,6 +400,8 @@ NUGET
400400
Microsoft.TestPlatform.TestHost (17.12) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= netcoreapp3.1)) (&& (== netstandard2.1) (>= netcoreapp3.1))
401401
Microsoft.TestPlatform.ObjectModel (>= 17.12) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= netcoreapp3.1)) (&& (== netstandard2.1) (>= netcoreapp3.1))
402402
Newtonsoft.Json (>= 13.0.1) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= netcoreapp3.1)) (&& (== netstandard2.1) (>= netcoreapp3.1))
403+
Microsoft.TestPlatform.TranslationLayer (17.13)
404+
NETStandard.Library (>= 2.0)
403405
Microsoft.VisualStudio.SolutionPersistence (1.0.28) - restriction: || (== net8.0) (== net9.0) (&& (== netstandard2.0) (>= net8.0)) (&& (== netstandard2.1) (>= net8.0))
404406
Microsoft.VisualStudio.Threading (17.12.19)
405407
Microsoft.Bcl.AsyncInterfaces (>= 6.0) - restriction: || (&& (== net8.0) (>= net472)) (&& (== net8.0) (< net6.0)) (&& (== net9.0) (>= net472)) (&& (== net9.0) (< net6.0)) (== netstandard2.0) (== netstandard2.1)
@@ -423,6 +425,8 @@ NUGET
423425
Microsoft.VisualStudio.Validation (>= 17.8.8)
424426
System.IO.Pipelines (>= 8.0)
425427
System.Runtime.CompilerServices.Unsafe (>= 6.0) - restriction: || (&& (== net8.0) (< net6.0)) (&& (== net8.0) (< netstandard2.1)) (&& (== net9.0) (< net6.0)) (&& (== net9.0) (< netstandard2.1)) (== netstandard2.0) (== netstandard2.1)
428+
NETStandard.Library (2.0.3)
429+
Microsoft.NETCore.Platforms (>= 1.1)
426430
Newtonsoft.Json (13.0.3)
427431
Nuget.Frameworks (6.12.1) - copy_local: false
428432
OpenTelemetry (1.10)

src/FsAutoComplete.Core/FsAutoComplete.Core.fsproj

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
<TargetFrameworks>net8.0</TargetFrameworks>
44
<TargetFrameworks Condition="'$(BuildNet9)' == 'true'">net8.0;net9.0</TargetFrameworks>
55
<IsPackable>false</IsPackable>
6-
<NoWarn>$(NoWarn);FS0057</NoWarn> <!-- Allows using experimental FCS APIs -->
6+
<NoWarn>$(NoWarn);FS0057</NoWarn>
7+
<!-- Allows using experimental FCS APIs -->
78
</PropertyGroup>
89
<ItemGroup>
910
<ProjectReference Include="..\FsAutoComplete.Logging\FsAutoComplete.Logging.fsproj" />
@@ -18,6 +19,8 @@
1819
<Compile Include="Debug.fs" />
1920
<Compile Include="Utils.fsi" />
2021
<Compile Include="Utils.fs" />
22+
<Compile Include="VSTestWrapper.fs" />
23+
<Compile Include="TestServer.fs" />
2124
<Compile Include="TestAdapter.fs" />
2225
<Compile Include="DotnetNewTemplate.fs" />
2326
<Compile Include="DotnetCli.fs" />
@@ -64,4 +67,4 @@
6467
<Compile Include="Commands.fs" />
6568
</ItemGroup>
6669
<Import Project="..\..\.paket\Paket.Restore.targets" />
67-
</Project>
70+
</Project>
Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
namespace FsAutoComplete.TestServer
2+
3+
open System
4+
5+
type TestFileRange = { StartLine: int; EndLine: int }
6+
7+
type TestItem =
8+
{
9+
FullName: string
10+
DisplayName: string
11+
/// Identifies the test adapter that ran the tests
12+
/// Example: executor://xunit/VsTestRunner2/netcoreapp
13+
/// Used for determining the test library, which effects how tests names are broken down
14+
ExecutorUri: string
15+
ProjectFilePath: string
16+
TargetFramework: string
17+
CodeFilePath: string option
18+
CodeLocationRange: TestFileRange option
19+
}
20+
21+
module TestItem =
22+
let ofVsTestCase
23+
(projFilePath: string)
24+
(targetFramework: string)
25+
(testCase: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase)
26+
: TestItem =
27+
{ FullName = testCase.FullyQualifiedName
28+
DisplayName = testCase.DisplayName
29+
ExecutorUri = testCase.ExecutorUri |> string
30+
ProjectFilePath = projFilePath
31+
TargetFramework = targetFramework
32+
CodeFilePath = Some testCase.CodeFilePath
33+
CodeLocationRange =
34+
Some
35+
{ StartLine = testCase.LineNumber
36+
EndLine = testCase.LineNumber } }
37+
38+
let tryTestCaseToDTO
39+
(projectLookup: string -> Ionide.ProjInfo.Types.ProjectOptions option)
40+
(testCase: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase)
41+
: TestItem option =
42+
match projectLookup testCase.Source with
43+
| None -> None // this should never happen. We pass VsTest the list of executables to test, so all the possible sources should be known to us
44+
| Some project -> ofVsTestCase project.ProjectFileName project.TargetFramework testCase |> Some
45+
46+
[<RequireQualifiedAccess>]
47+
type TestOutcome =
48+
| Failed = 0
49+
| Passed = 1
50+
| Skipped = 2
51+
| None = 3
52+
| NotFound = 4
53+
54+
module TestOutcome =
55+
type VSTestOutcome = Microsoft.VisualStudio.TestPlatform.ObjectModel.TestOutcome
56+
57+
let ofVSTestOutcome (vsTestOutcome: VSTestOutcome) =
58+
match vsTestOutcome with
59+
| VSTestOutcome.Passed -> TestOutcome.Passed
60+
| VSTestOutcome.Failed -> TestOutcome.Failed
61+
| VSTestOutcome.Skipped -> TestOutcome.Skipped
62+
| VSTestOutcome.NotFound -> TestOutcome.NotFound
63+
| VSTestOutcome.None -> TestOutcome.None
64+
| _ -> TestOutcome.None
65+
66+
type TestResult =
67+
{ TestItem: TestItem
68+
Outcome: TestOutcome
69+
ErrorMessage: string option
70+
ErrorStackTrace: string option
71+
AdditionalOutput: string option
72+
Duration: TimeSpan }
73+
74+
module TestResult =
75+
type VSTestResult = Microsoft.VisualStudio.TestPlatform.ObjectModel.TestResult
76+
77+
let ofVsTestResult (projFilePath: string) (targetFramework: string) (vsTestResult: VSTestResult) : TestResult =
78+
let stringToOption (text: string) = if String.IsNullOrEmpty(text) then None else Some text
79+
80+
{ Outcome = TestOutcome.ofVSTestOutcome vsTestResult.Outcome
81+
ErrorMessage = vsTestResult.ErrorMessage |> stringToOption
82+
ErrorStackTrace = vsTestResult.ErrorStackTrace |> stringToOption
83+
AdditionalOutput =
84+
match vsTestResult.Messages |> Seq.toList with
85+
| [] -> None
86+
| messages -> messages |> List.map _.Text |> String.concat Environment.NewLine |> Some
87+
Duration = vsTestResult.Duration
88+
TestItem = TestItem.ofVsTestCase projFilePath targetFramework vsTestResult.TestCase }
Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
namespace FsAutoComplete.TestServer
2+
3+
open System
4+
5+
module VSTestWrapper =
6+
open Microsoft.TestPlatform.VsTestConsole.TranslationLayer
7+
open Microsoft.VisualStudio.TestPlatform.ObjectModel
8+
open Microsoft.VisualStudio.TestPlatform.ObjectModel.Client
9+
open Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging
10+
open Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.Interfaces
11+
12+
type TestProjectDll = string
13+
14+
type TestDiscoveryUpdate =
15+
| Progress of TestCase list
16+
| LogMessage of TestMessageLevel * string
17+
18+
type private TestDiscoveryHandler(notifyDiscoveryProgress: TestDiscoveryUpdate -> unit) =
19+
20+
member val DiscoveredTests: TestCase ResizeArray = ResizeArray() with get, set
21+
22+
interface ITestDiscoveryEventsHandler with
23+
member this.HandleDiscoveredTests(discoveredTestCases: System.Collections.Generic.IEnumerable<TestCase>) : unit =
24+
if (not << isNull) discoveredTestCases then
25+
this.DiscoveredTests.AddRange(discoveredTestCases)
26+
notifyDiscoveryProgress (discoveredTestCases |> List.ofSeq |> Progress)
27+
28+
member this.HandleDiscoveryComplete
29+
(_totalTests: int64, lastChunk: System.Collections.Generic.IEnumerable<TestCase>, _isAborted: bool)
30+
: unit =
31+
if (not << isNull) lastChunk then
32+
this.DiscoveredTests.AddRange(lastChunk)
33+
notifyDiscoveryProgress (lastChunk |> List.ofSeq |> Progress)
34+
35+
member this.HandleLogMessage(level: TestMessageLevel, message: string) : unit =
36+
notifyDiscoveryProgress (LogMessage(level, message))
37+
38+
member this.HandleRawMessage(_rawMessage: string) : unit = ()
39+
40+
type ProcessId = int
41+
type DidDebuggerAttach = bool
42+
43+
type TestRunUpdate =
44+
| Progress of TestRunChangedEventArgs
45+
| LogMessage of TestMessageLevel * string
46+
47+
type private TestRunHandler(notifyTestRunProgress: TestRunUpdate -> unit) =
48+
49+
member val TestResults: TestResult ResizeArray = ResizeArray() with get, set
50+
51+
interface ITestRunEventsHandler with
52+
member _.HandleLogMessage(level: TestMessageLevel, message: string) : unit =
53+
notifyTestRunProgress (LogMessage(level, message))
54+
55+
member _.HandleRawMessage(_rawMessage: string) : unit = ()
56+
57+
member this.HandleTestRunComplete
58+
(
59+
_testRunCompleteArgs: TestRunCompleteEventArgs,
60+
lastChunkArgs: TestRunChangedEventArgs,
61+
_runContextAttachments: System.Collections.Generic.ICollection<AttachmentSet>,
62+
_executorUris: System.Collections.Generic.ICollection<string>
63+
) : unit =
64+
if ((not << isNull) lastChunkArgs && (not << isNull) lastChunkArgs.NewTestResults) then
65+
this.TestResults.AddRange(lastChunkArgs.NewTestResults)
66+
notifyTestRunProgress (Progress lastChunkArgs)
67+
68+
member this.HandleTestRunStatsChange(testRunChangedArgs: TestRunChangedEventArgs) : unit =
69+
if
70+
((not << isNull) testRunChangedArgs
71+
&& (not << isNull) testRunChangedArgs.NewTestResults)
72+
then
73+
this.TestResults.AddRange(testRunChangedArgs.NewTestResults)
74+
notifyTestRunProgress (Progress testRunChangedArgs)
75+
76+
member _.LaunchProcessWithDebuggerAttached(_testProcessStartInfo: TestProcessStartInfo) : int =
77+
raise (System.NotImplementedException())
78+
79+
type private TestHostLauncher(isDebug: bool, onAttachDebugger: ProcessId -> DidDebuggerAttach) =
80+
// IMPORTANT: RunTestsWithCustomTestHost says it takes an ITestHostLauncher, but it actually calls a method that is only available on ITestHostLauncher3
81+
82+
interface ITestHostLauncher3 with
83+
member _.IsDebug: bool = isDebug
84+
85+
member _.LaunchTestHost(_defaultTestHostStartInfo: TestProcessStartInfo) : int = raise (NotImplementedException())
86+
87+
member _.LaunchTestHost
88+
(_defaultTestHostStartInfo: TestProcessStartInfo, _cancellationToken: Threading.CancellationToken)
89+
: int =
90+
raise (NotImplementedException())
91+
92+
member _.AttachDebuggerToProcess
93+
(attachDebuggerInfo: AttachDebuggerInfo, _cancellationToken: Threading.CancellationToken)
94+
: bool =
95+
onAttachDebugger attachDebuggerInfo.ProcessId
96+
97+
member _.AttachDebuggerToProcess(pid: int) : bool = onAttachDebugger pid
98+
99+
member _.AttachDebuggerToProcess(pid: int, _cancellationToken: Threading.CancellationToken) : bool =
100+
onAttachDebugger pid
101+
102+
103+
module TestPlatformOptions =
104+
let withTestCaseFilter (options: TestPlatformOptions) filterExpression = options.TestCaseFilter <- filterExpression
105+
106+
module private RunSettings =
107+
let defaultRunSettings =
108+
"<RunSettings>
109+
<RunConfiguration>
110+
<DesignMode>False</DesignMode>
111+
</RunConfiguration>
112+
</RunSettings>"
113+
114+
let discoverTestsAsync
115+
(vstestPath: string)
116+
(onDiscoveryProgress: TestDiscoveryUpdate -> unit)
117+
(sources: TestProjectDll list)
118+
: Async<TestCase list> =
119+
async {
120+
let consoleParams = ConsoleParameters()
121+
let vstest = new VsTestConsoleWrapper(vstestPath, consoleParams)
122+
let discoveryHandler = TestDiscoveryHandler(onDiscoveryProgress)
123+
124+
use! _onCancel = Async.OnCancel(fun () -> vstest.CancelDiscovery())
125+
126+
vstest.DiscoverTests(sources, null, discoveryHandler)
127+
return discoveryHandler.DiscoveredTests |> List.ofSeq
128+
}
129+
130+
/// onAttachDebugger assumes that the debugger is attached when the method returns. The test project will continue execution as soon as attachDebugger returns
131+
let runTestsAsync
132+
(vstestPath: string)
133+
(onTestRunProgress: TestRunUpdate -> unit)
134+
(onAttachDebugger: ProcessId -> DidDebuggerAttach)
135+
(sources: TestProjectDll list)
136+
(testCaseFilter: string option)
137+
(shouldDebug: bool)
138+
: Async<TestResult list> =
139+
async {
140+
let consoleParams = ConsoleParameters()
141+
let vstest = new VsTestConsoleWrapper(vstestPath, consoleParams)
142+
let runHandler = TestRunHandler(onTestRunProgress)
143+
let runSettings = RunSettings.defaultRunSettings
144+
145+
let options = new TestPlatformOptions()
146+
testCaseFilter |> Option.iter (TestPlatformOptions.withTestCaseFilter options)
147+
148+
use! _cancel = Async.OnCancel(fun () -> vstest.CancelTestRun())
149+
150+
if shouldDebug then
151+
let hostLauncher = TestHostLauncher(shouldDebug, onAttachDebugger)
152+
vstest.RunTestsWithCustomTestHost(sources, runSettings, options, runHandler, hostLauncher)
153+
else
154+
vstest.RunTests(sources, runSettings, options, runHandler)
155+
156+
return runHandler.TestResults |> List.ofSeq
157+
}
158+
159+
open System.IO
160+
161+
let tryFindVsTestFromDotnetRoot (dotnetRoot: string) (workspaceRoot: string option) : Result<FileInfo, string> =
162+
let cwd =
163+
defaultArg workspaceRoot System.Environment.CurrentDirectory |> DirectoryInfo
164+
165+
let dotnetBinary =
166+
if dotnetRoot |> Directory.Exists then
167+
if
168+
System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(
169+
System.Runtime.InteropServices.OSPlatform.Windows
170+
)
171+
then
172+
FileInfo(Path.Combine(dotnetRoot, "dotnet.exe"))
173+
else
174+
FileInfo(Path.Combine(dotnetRoot, "dotnet"))
175+
else
176+
dotnetRoot |> FileInfo
177+
178+
match Ionide.ProjInfo.SdkDiscovery.versionAt cwd dotnetBinary with
179+
| Ok sdkVersion ->
180+
let sdks = Ionide.ProjInfo.SdkDiscovery.sdks dotnetBinary
181+
182+
match sdks |> Array.tryFind (fun sdk -> sdk.Version = sdkVersion) with
183+
| Some sdk ->
184+
let vstestBinary = Path.Combine(sdk.Path.FullName, "vstest.console.dll") |> FileInfo
185+
186+
if vstestBinary.Exists then
187+
Ok vstestBinary
188+
else
189+
Error $"Found the correct dotnet sdk, but vstest was not at the expected sub-path: {vstestBinary.FullName}"
190+
| None -> Error $"Couldn't find the install location for dotnet sdk version: {sdkVersion}"
191+
| Error _ -> Error $"Couldn't identify the dotnet version for working directory: {cwd.FullName}"

src/FsAutoComplete.Core/paket.references

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,6 @@ FSharp.Analyzers.SDK
1717
Ionide.Analyzers
1818
FSharp.Analyzers.Build
1919
Serilog.Extensions.Logging
20+
21+
Microsoft.TestPlatform.ObjectModel
22+
Microsoft.TestPlatform.TranslationLayer

src/FsAutoComplete/CommandResponse.fs

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -702,3 +702,17 @@ module CommandResponse =
702702
PrecedingNonPipeExprLine = pnp })
703703

704704
serialize { Kind = "pipelineHint"; Data = ctn }
705+
706+
707+
708+
type DiscoverTestsResponse = TestServer.TestItem list
709+
710+
let discoverTests (serialize: Serializer) (content: DiscoverTestsResponse) =
711+
serialize
712+
{ Kind = "discoverTests"
713+
Data = content }
714+
715+
let runTests (serialize: Serializer) (content: TestServer.TestResult list) =
716+
serialize
717+
{ Kind = "runTests"
718+
Data = content |> Array.ofList }

src/FsAutoComplete/CommandResponse.fsi

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,3 +261,8 @@ module CommandResponse =
261261
val compile: serialize: Serializer -> errors: #FSharpDiagnostic array * code: int -> string
262262
val fsharpLiterate: serialize: Serializer -> content: string -> string
263263
val pipelineHint: serialize: Serializer -> content: (int * int option * string list)[] -> string
264+
265+
type DiscoverTestsResponse = TestServer.TestItem list
266+
267+
val discoverTests: serialize: Serializer -> content: DiscoverTestsResponse -> string
268+
val runTests: serialize: Serializer -> content: TestServer.TestResult list -> string

0 commit comments

Comments
 (0)