How to halt dotnet-trace in an automated context (CI job)

[danielo-unity3d]

Hi.
The dotnet-trace tool expects to end collection when it receives a Ctrl-C/break command or a newline, however, I've been unsuccessful setting up a script to do this in automated fashion. How exactly are we to send the termination event to the tracing tool.

The basic setup is:

echo "Creating dotnet-trace with diagnostic port"

$psi = New-Object System.Diagnostics.ProcessStartInfo;
$psi.FileName = "dotnet-trace"
$psi.Arguments = @("collect", "--output", "AppToMonitor-diag.nettrace", "--diagnostic-port", "AppToMonitor-diag.sock")
$psi.UseShellExecute = $false
$psi.RedirectStandardInput = $true

$p = [System.Diagnostics.Process]::Start($psi);

# Time passes, the application is run, one or more times, etc
# Perhaps it is run from another window in parallel, having set
sleep 10

The output of the trace program is:

No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0x0000F00000000000  Informational(4)    --profile 
Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile 

Waiting for connection on /home/username/Unity.Licensing.Client-diag.sock
Start an application with the following environment variable: DOTNET_DiagnosticPorts=/home/username/AppToMonitor-diag.sock

Then, to attempt to halt the application, any input should work, since a newline also should terminate the tool:

$p.StandardInput.WriteLine([System.Text.Encoding]::ASCII.GetString(3));

I can't seem to get it to stop gracefully on Windows (such that the trace is then legible, and the symbols are correctly identified, ie. missing Rundown events?).
On Linux or MacOS (because we need this to be cross-platform), then we could just resort to calling kill -SIGQUIT $pid, which appears to notify the trace tool correctly.

See issue #976, where using the --duration flag is difficult for us, as we don't know how long the job will run (varies by multiple orders of magnitude, and changes according to load on the host)

[danielo-unity3d]

Note that, I also tried this:

$job = start-job -scriptblock {
   echo "Creating dotnet-trace with diagnostic port"
   dotnet-trace collect --output test-results\AppToMonitor-diag.nettrace --diagnostic-port "AppToMonitor-diag.sock"
   echo "After dotnet-trace"
}
start-sleep -Seconds 1
receive-job $job

echo "Launching UTR"
AppToMonitor --runFor10Seconds
stop-job $job # unfortunately kills dotnet-trace without giving it a chance to shutdown properly
receive-job -Wait $job

[danielo-unity3d]

I also thought it might be possible to user taskkill /pid <pid> to gracefully terminate the trace tool, but I had no such luck:

$ taskkill -pid 23412
ERROR: The process with PID 23412 could not be terminated.
Reason: This process can only be terminated forcefully (with /F 
option).

$ taskkill -f -pid 23412
SUCCESS: The process with PID 23412 has been terminated.

When forcefully terminated, the trace file is incomplete:

$ dotnet-trace convert --format=Chromium test.nettrace
Writing:        C:\Users\username\AppToMonitor\test.chromium.json
Detected a potentially broken trace. Continuing with best-efforts to convert the trace, but resulting speedscope file may contain broken stacks as a result.
Conversion complete

[mikelle-rogers]

Hello @danielo-unity3d, would you please clarify the motivation behind running dotnet-trace in automation? (How many processes are you tracing, where will you use this script, etc.) Understanding your motivation/purpose of running in automation will help us decide the best way to assist you.

[danielo-unity3d]

The motivation is to find a bug in our application, which fails to fully launch.

It hangs during startup when we call ConfigurationBuilder().AddJsonFile with reloadOnChange: true. We think the underlying file watchers are responsible, at least, this is what the stacks we have collected from hung processes show (but we need more data to understand the circumstances). Funny thing is that using DOTNET_USE_POLLING_FILE_WATCHER=1 doesn't seem to change the outcome, but that is (possibly) a different issue. Eventually, I would also like to collect counters, and other performance data, but for now, I wanted the traces.

We don't strictly need to wait until the monitored task ends, because this happens early in the program startup (we can use the --duration flag to exit, and still get useful data). What if it was happening later, during steady state? This is happening in production (on build machines during CI runs), and is an instability so it happens very rarely, thus the need for automation. Note that, by "production", I mean the consumers of our component (this doesn't happen on our own CI, during either unit or functional test runs, though we share the same general infrastructure as the consumers).

The real issue is how to cooperatively interrupt a program on Windows (send it a Ctrl-C or Ctrl-Break). On MacOS and Linux, kill -SIGQUIT seems to work.
To get it to work on Windows, the target process needs to be launched within a separate process group (CREATE_NEW_PROCESS_GROUP), and the break event is sent to the console using (GenerateConsoleCtrlEvent). It's all rather complicated to get the target process to shutdown gracefully (either from Powershell, from C++, or C#).

So this isn't really a bug for the dotnet-trace program, but the impact on you is that we (users) end up with a trace file that cannot resolve symbols. Perhaps this could be fixed (or simplified) in the dotnet runtime, or in Windows itself; I don't know.

Thanks for looking at this.

[mikelle-rogers]

Thank you for your detailed response regarding the situation in which this is occurring. I am looking into some options and will continue to update you as I move forward.

[mikelle-rogers]

@danielo-unity3d, this seems like an issue unique to Windows, in order to accomplish your request, we may need to add some functionality to the tools in Windows. We have added the enhancement tag and will track this request.

Suggestions going forward: creating a new tool for Windows (or adding the feature onto dotnet-trace) that uses the Stop method from the EventPipe Providers class

[jonathanpeppers]

@mikelle-rogers we need this for MAUI as well. Mobile requires usage of dotnet-trace requires dotnet-dsrouter and other setup. Because it's complicated, we wanted to make MSBuild targets to make it easier:

> dotnet build -t:BuildAndStartProfiling
...
> dotnet build -t:FinishProfiling

Which results in a .nettrace in the current directory. I have it working, except for a way to shutdown dotnet-trace gracefully. Is there anything we can do in .NET 7 today for this? Thanks!

[jeremy-visionaid]

@jonathanpeppers MSBuild targets for profiling in MAUI would be amazing. Is there an upstream issue for that?

[jonathanpeppers]

There is a general issue for improving diagnostics here:

• Enhance diagnostic experience for Maui maui#29319

We're thinking eventually we can get something like this to work:

# android
$ dotnet-trace collect --dsrouter android -- dotnet build YourMauiApp.csproj -c Release -f net9.0-android -t:Run
# ios
$ dotnet-trace collect --dsrouter ios -- dotnet build YourMauiApp.csproj -c Release -f net9.0-ios -t:Run

We will need to make a new -p:DiagnosticPorts property that dotnet-trace/dsrouter will know to pass to the second command automatically.