-
Notifications
You must be signed in to change notification settings - Fork 162
Test Explorer Support via VSTest #1383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 36 commits
Commits
Show all changes
39 commits
Select commit
Hold shift + click to select a range
5b8b3b8
Set up the stub of a test explorer test project
farlee2121 6018b12
Add the test explorer tests to the sln
farlee2121 1b1bb74
Set up a trival test case to scaffold for VSTestAdapter work
farlee2121 f3d2f3f
Update Expecto for performance improvements
farlee2121 3a386f6
Demonstrate the most basic test discovery with vstest
farlee2121 6b6c243
Demonstrate connection between ionide and language server
farlee2121 0210fd9
Implement and mvp test discovery endpoint on the Language Server
farlee2121 3712a46
Push test discovery down to AdaptiveState
farlee2121 f73c2bb
Publish incremental test discovery updates
farlee2121 e054ef6
Demonstration a test run using VSTest
farlee2121 8fd2f86
Rename the namespace for test features.
farlee2121 724c158
Add an lsp endpoint for running tests
farlee2121 24ceafc
Fix bug where the TestExplorer tests don't build the required sample …
farlee2121 5842f82
Demonstrate basic streaming of test run results from langauge server
farlee2121 768acb4
Add test filtering to the VSTest wrapper
farlee2121 487a6ce
Add testCaseFilter support to test Runs
farlee2121 5d9e2f3
Prove that I can scrape test host process Ids
farlee2121 7de0832
Convert VsTestWrapper methods to async-only for graceful cancellation
farlee2121 80c6ead
Pass a server-level test for the attaching debugger to test run.
farlee2121 8c99c72
Format files
farlee2121 7065eaa
Delete some diagnostic code
farlee2121 8114ca9
fix incorrect message type on the runTests lsp message
farlee2121 b49d397
Pass unit tests using RunTestsWithCustomTestHost
farlee2121 3cae415
Debug test runs
farlee2121 0be6678
Test that client environment variables are available in a test run
farlee2121 7c7b2fd
Decouple SampleTestProjects for Lsp and TestExplorer test projects
farlee2121 43c398e
Forward test logs
farlee2121 d4bb314
Allow client to limit which test projects are run
farlee2121 16d7222
Workaround for NUnit not respecting test filters
farlee2121 740cfb1
Forward test discovery logs to the client for improved error diagnostics
farlee2121 c967543
Show error notification in client for expectable test discovery issue…
farlee2121 72728d2
Separate discovery vs run lsp-level test explorer tests
farlee2121 85a4e0f
Split the VSTest wrapper code from TestServer contracts for clarity
farlee2121 ee7dc5a
Clarify dictionary name (projectLookup -> projectsByBinaryPath)
farlee2121 fbc5868
Satisfy fantomas formatting check
farlee2121 23bf364
Remove print statements per copilot review
farlee2121 1bdb095
Add OpenTelemetry to the Test Explorer langauge server endpoints
farlee2121 edccad0
Avoid duplicate project load when discovery test projects
farlee2121 c4c5345
Fix net9 build
TheAngryByrd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| namespace FsAutoComplete.TestServer | ||
|
|
||
| open System | ||
|
|
||
| type TestFileRange = { StartLine: int; EndLine: int } | ||
|
|
||
| type TestItem = | ||
| { | ||
| FullName: string | ||
| DisplayName: string | ||
| /// Identifies the test adapter that ran the tests | ||
| /// Example: executor://xunit/VsTestRunner2/netcoreapp | ||
| /// Used for determining the test library, which effects how tests names are broken down | ||
| ExecutorUri: string | ||
| ProjectFilePath: string | ||
| TargetFramework: string | ||
| CodeFilePath: string option | ||
| CodeLocationRange: TestFileRange option | ||
| } | ||
|
|
||
| module TestItem = | ||
| let ofVsTestCase | ||
| (projFilePath: string) | ||
| (targetFramework: string) | ||
| (testCase: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase) | ||
| : TestItem = | ||
| { FullName = testCase.FullyQualifiedName | ||
| DisplayName = testCase.DisplayName | ||
| ExecutorUri = testCase.ExecutorUri |> string | ||
| ProjectFilePath = projFilePath | ||
| TargetFramework = targetFramework | ||
| CodeFilePath = Some testCase.CodeFilePath | ||
| CodeLocationRange = | ||
| Some | ||
| { StartLine = testCase.LineNumber | ||
| EndLine = testCase.LineNumber } } | ||
|
|
||
| let tryTestCaseToDTO | ||
| (projectLookup: string -> Ionide.ProjInfo.Types.ProjectOptions option) | ||
| (testCase: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase) | ||
| : TestItem option = | ||
| match projectLookup testCase.Source with | ||
| | None -> None // this should never happen. We pass VsTest the list of executables to test, so all the possible sources should be known to us | ||
| | Some project -> ofVsTestCase project.ProjectFileName project.TargetFramework testCase |> Some | ||
|
|
||
| [<RequireQualifiedAccess>] | ||
| type TestOutcome = | ||
| | Failed = 0 | ||
| | Passed = 1 | ||
| | Skipped = 2 | ||
| | None = 3 | ||
| | NotFound = 4 | ||
|
|
||
| module TestOutcome = | ||
| type VSTestOutcome = Microsoft.VisualStudio.TestPlatform.ObjectModel.TestOutcome | ||
|
|
||
| let ofVSTestOutcome (vsTestOutcome: VSTestOutcome) = | ||
| match vsTestOutcome with | ||
| | VSTestOutcome.Passed -> TestOutcome.Passed | ||
| | VSTestOutcome.Failed -> TestOutcome.Failed | ||
| | VSTestOutcome.Skipped -> TestOutcome.Skipped | ||
| | VSTestOutcome.NotFound -> TestOutcome.NotFound | ||
| | VSTestOutcome.None -> TestOutcome.None | ||
| | _ -> TestOutcome.None | ||
|
|
||
| type TestResult = | ||
| { TestItem: TestItem | ||
| Outcome: TestOutcome | ||
| ErrorMessage: string option | ||
| ErrorStackTrace: string option | ||
| AdditionalOutput: string option | ||
| Duration: TimeSpan } | ||
|
|
||
| module TestResult = | ||
| type VSTestResult = Microsoft.VisualStudio.TestPlatform.ObjectModel.TestResult | ||
|
|
||
| let ofVsTestResult (projFilePath: string) (targetFramework: string) (vsTestResult: VSTestResult) : TestResult = | ||
| let stringToOption (text: string) = if String.IsNullOrEmpty(text) then None else Some text | ||
|
|
||
| { Outcome = TestOutcome.ofVSTestOutcome vsTestResult.Outcome | ||
| ErrorMessage = vsTestResult.ErrorMessage |> stringToOption | ||
| ErrorStackTrace = vsTestResult.ErrorStackTrace |> stringToOption | ||
| AdditionalOutput = | ||
| match vsTestResult.Messages |> Seq.toList with | ||
| | [] -> None | ||
| | messages -> messages |> List.map _.Text |> String.concat Environment.NewLine |> Some | ||
| Duration = vsTestResult.Duration | ||
| TestItem = TestItem.ofVsTestCase projFilePath targetFramework vsTestResult.TestCase } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| namespace FsAutoComplete.TestServer | ||
|
|
||
| open System | ||
|
|
||
| module VSTestWrapper = | ||
| open Microsoft.TestPlatform.VsTestConsole.TranslationLayer | ||
| open Microsoft.VisualStudio.TestPlatform.ObjectModel | ||
| open Microsoft.VisualStudio.TestPlatform.ObjectModel.Client | ||
| open Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging | ||
| open Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.Interfaces | ||
|
|
||
| type TestProjectDll = string | ||
|
|
||
| type TestDiscoveryUpdate = | ||
| | Progress of TestCase list | ||
| | LogMessage of TestMessageLevel * string | ||
|
|
||
| type private TestDiscoveryHandler(notifyDiscoveryProgress: TestDiscoveryUpdate -> unit) = | ||
|
|
||
| member val DiscoveredTests: TestCase ResizeArray = ResizeArray() with get, set | ||
|
|
||
| interface ITestDiscoveryEventsHandler with | ||
| member this.HandleDiscoveredTests(discoveredTestCases: System.Collections.Generic.IEnumerable<TestCase>) : unit = | ||
| if (not << isNull) discoveredTestCases then | ||
| this.DiscoveredTests.AddRange(discoveredTestCases) | ||
| notifyDiscoveryProgress (discoveredTestCases |> List.ofSeq |> Progress) | ||
|
|
||
| member this.HandleDiscoveryComplete | ||
| (_totalTests: int64, lastChunk: System.Collections.Generic.IEnumerable<TestCase>, _isAborted: bool) | ||
| : unit = | ||
| if (not << isNull) lastChunk then | ||
| this.DiscoveredTests.AddRange(lastChunk) | ||
| notifyDiscoveryProgress (lastChunk |> List.ofSeq |> Progress) | ||
|
|
||
| member this.HandleLogMessage(level: TestMessageLevel, message: string) : unit = | ||
| notifyDiscoveryProgress (LogMessage(level, message)) | ||
|
|
||
| member this.HandleRawMessage(_rawMessage: string) : unit = () | ||
|
|
||
| type ProcessId = int | ||
| type DidDebuggerAttach = bool | ||
|
|
||
| type TestRunUpdate = | ||
| | Progress of TestRunChangedEventArgs | ||
| | LogMessage of TestMessageLevel * string | ||
|
|
||
| type private TestRunHandler(notifyTestRunProgress: TestRunUpdate -> unit) = | ||
|
|
||
| member val TestResults: TestResult ResizeArray = ResizeArray() with get, set | ||
|
|
||
| interface ITestRunEventsHandler with | ||
| member _.HandleLogMessage(level: TestMessageLevel, message: string) : unit = | ||
| notifyTestRunProgress (LogMessage(level, message)) | ||
|
|
||
| member _.HandleRawMessage(_rawMessage: string) : unit = () | ||
|
|
||
| member this.HandleTestRunComplete | ||
| ( | ||
| _testRunCompleteArgs: TestRunCompleteEventArgs, | ||
| lastChunkArgs: TestRunChangedEventArgs, | ||
| _runContextAttachments: System.Collections.Generic.ICollection<AttachmentSet>, | ||
| _executorUris: System.Collections.Generic.ICollection<string> | ||
| ) : unit = | ||
| if ((not << isNull) lastChunkArgs && (not << isNull) lastChunkArgs.NewTestResults) then | ||
| this.TestResults.AddRange(lastChunkArgs.NewTestResults) | ||
| notifyTestRunProgress (Progress lastChunkArgs) | ||
|
|
||
| member this.HandleTestRunStatsChange(testRunChangedArgs: TestRunChangedEventArgs) : unit = | ||
| if | ||
| ((not << isNull) testRunChangedArgs | ||
| && (not << isNull) testRunChangedArgs.NewTestResults) | ||
| then | ||
| this.TestResults.AddRange(testRunChangedArgs.NewTestResults) | ||
| notifyTestRunProgress (Progress testRunChangedArgs) | ||
|
|
||
| member _.LaunchProcessWithDebuggerAttached(_testProcessStartInfo: TestProcessStartInfo) : int = | ||
| raise (System.NotImplementedException()) | ||
|
|
||
| type private TestHostLauncher(isDebug: bool, onAttachDebugger: ProcessId -> DidDebuggerAttach) = | ||
| // IMPORTANT: RunTestsWithCustomTestHost says it takes an ITestHostLauncher, but it actually calls a method that is only available on ITestHostLauncher3 | ||
|
|
||
| interface ITestHostLauncher3 with | ||
| member _.IsDebug: bool = isDebug | ||
|
|
||
| member _.LaunchTestHost(_defaultTestHostStartInfo: TestProcessStartInfo) : int = raise (NotImplementedException()) | ||
|
|
||
| member _.LaunchTestHost | ||
| (_defaultTestHostStartInfo: TestProcessStartInfo, _cancellationToken: Threading.CancellationToken) | ||
| : int = | ||
| raise (NotImplementedException()) | ||
|
|
||
| member _.AttachDebuggerToProcess | ||
| (attachDebuggerInfo: AttachDebuggerInfo, _cancellationToken: Threading.CancellationToken) | ||
| : bool = | ||
| onAttachDebugger attachDebuggerInfo.ProcessId | ||
|
|
||
| member _.AttachDebuggerToProcess(pid: int) : bool = onAttachDebugger pid | ||
|
|
||
| member _.AttachDebuggerToProcess(pid: int, _cancellationToken: Threading.CancellationToken) : bool = | ||
| onAttachDebugger pid | ||
|
|
||
|
|
||
| module TestPlatformOptions = | ||
| let withTestCaseFilter (options: TestPlatformOptions) filterExpression = options.TestCaseFilter <- filterExpression | ||
|
|
||
| module private RunSettings = | ||
| let defaultRunSettings = | ||
| "<RunSettings> | ||
| <RunConfiguration> | ||
| <DesignMode>False</DesignMode> | ||
| </RunConfiguration> | ||
| </RunSettings>" | ||
|
|
||
| let discoverTestsAsync | ||
| (vstestPath: string) | ||
| (onDiscoveryProgress: TestDiscoveryUpdate -> unit) | ||
| (sources: TestProjectDll list) | ||
| : Async<TestCase list> = | ||
| async { | ||
| let consoleParams = ConsoleParameters() | ||
| let vstest = new VsTestConsoleWrapper(vstestPath, consoleParams) | ||
| let discoveryHandler = TestDiscoveryHandler(onDiscoveryProgress) | ||
|
|
||
| use! _onCancel = Async.OnCancel(fun () -> vstest.CancelDiscovery()) | ||
|
|
||
| vstest.DiscoverTests(sources, null, discoveryHandler) | ||
| return discoveryHandler.DiscoveredTests |> List.ofSeq | ||
| } | ||
|
|
||
| /// onAttachDebugger assumes that the debugger is attached when the method returns. The test project will continue execution as soon as attachDebugger returns | ||
| let runTestsAsync | ||
| (vstestPath: string) | ||
| (onTestRunProgress: TestRunUpdate -> unit) | ||
| (onAttachDebugger: ProcessId -> DidDebuggerAttach) | ||
| (sources: TestProjectDll list) | ||
| (testCaseFilter: string option) | ||
| (shouldDebug: bool) | ||
| : Async<TestResult list> = | ||
| async { | ||
| let consoleParams = ConsoleParameters() | ||
| let vstest = new VsTestConsoleWrapper(vstestPath, consoleParams) | ||
| let runHandler = TestRunHandler(onTestRunProgress) | ||
| let runSettings = RunSettings.defaultRunSettings | ||
|
|
||
| let options = new TestPlatformOptions() | ||
| testCaseFilter |> Option.iter (TestPlatformOptions.withTestCaseFilter options) | ||
|
|
||
| use! _cancel = Async.OnCancel(fun () -> vstest.CancelTestRun()) | ||
|
|
||
| if shouldDebug then | ||
| let hostLauncher = TestHostLauncher(shouldDebug, onAttachDebugger) | ||
| vstest.RunTestsWithCustomTestHost(sources, runSettings, options, runHandler, hostLauncher) | ||
| else | ||
| vstest.RunTests(sources, runSettings, options, runHandler) | ||
|
|
||
| return runHandler.TestResults |> List.ofSeq | ||
| } | ||
|
|
||
| open System.IO | ||
|
|
||
| let tryFindVsTestFromDotnetRoot (dotnetRoot: string) (workspaceRoot: string option) : Result<FileInfo, string> = | ||
| let cwd = | ||
| defaultArg workspaceRoot System.Environment.CurrentDirectory |> DirectoryInfo | ||
|
|
||
| let dotnetBinary = | ||
| if dotnetRoot |> Directory.Exists then | ||
| if | ||
| System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform( | ||
| System.Runtime.InteropServices.OSPlatform.Windows | ||
| ) | ||
| then | ||
| FileInfo(Path.Combine(dotnetRoot, "dotnet.exe")) | ||
| else | ||
| FileInfo(Path.Combine(dotnetRoot, "dotnet")) | ||
| else | ||
| dotnetRoot |> FileInfo | ||
|
|
||
| match Ionide.ProjInfo.SdkDiscovery.versionAt cwd dotnetBinary with | ||
| | Ok sdkVersion -> | ||
| let sdks = Ionide.ProjInfo.SdkDiscovery.sdks dotnetBinary | ||
|
|
||
| match sdks |> Array.tryFind (fun sdk -> sdk.Version = sdkVersion) with | ||
| | Some sdk -> | ||
| let vstestBinary = Path.Combine(sdk.Path.FullName, "vstest.console.dll") |> FileInfo | ||
|
|
||
| if vstestBinary.Exists then | ||
| Ok vstestBinary | ||
| else | ||
| Error $"Found the correct dotnet sdk, but vstest was not at the expected sub-path: {vstestBinary.FullName}" | ||
| | None -> Error $"Couldn't find the install location for dotnet sdk version: {sdkVersion}" | ||
| | Error _ -> Error $"Couldn't identify the dotnet version for working directory: {cwd.FullName}" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it worth logging the message on the server side as well?