Support `dotnet trace` running in a sidecar container

[MichaelSimons]

1. Run a containerized ASP.NET Core app within a container.

docker run -it -p 8000:80 --name aspnetapp mcr.microsoft.com/dotnet/core/samples:aspnetapp

2. Build a image which contains the dotnet trace tool

FROM mcr.microsoft.com/dotnet/core/sdk:3.1

RUN dotnet tool install --global dotnet-trace

ENV PATH="${PATH}:/root/.dotnet/tools"

docker build -t dotnet/trace .

3. Run the dotnet trace container

docker run -it --net=container:aspnetapp --pid=container:aspnetapp --cap-add ALL --privileged dotnet/trace

4. Try to collect a trace of the ASP.NET Core app.

Expected Results:

I should be able to collect a dotnet trace. It is a common technique to utilize sidecar containers to profile applications running in another container. This allows you to profile existing application images without having to modify them. You can package all of your tools into a completely separate tools image.

An example of running perfcollect with this pattern is documented in this blog post

Actual Results

1. dotnet trace ps does not match ps -aux

root@17be50bbd0fd:/# ps -aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  2.1 21288796 87672 pts/0  SLsl+ 22:36   0:03 dotnet aspnetapp.dll
root      1747  0.0  0.0   3988  3036 pts/0    Ss   23:19   0:00 bash
root      2231  0.0  0.0   7640  2704 pts/0    R+   23:23   0:00 ps -aux
root@17be50bbd0fd:/# dotnet trace ps
      2254 dotnet     /usr/share/dotnet/dotnet
      2272 dotnet-trace /root/.dotnet/tools/dotnet-trace

2. The PID reported by dotnet trace cannot be found when running collect

root@17be50bbd0fd:/# dotnet trace collect -p 2254
No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0xFFFFFFFFFFFFFFFF  Verbose(5)          --profile
Microsoft-Windows-DotNETRuntime         0x00000004C14FCCBD  Informational(4)    --profile

[ERROR] System.ArgumentException: Process with an Id of 2254 is not running.
   at System.Diagnostics.Process.GetProcessById(Int32 processId, String machineName)
   at System.Diagnostics.Process.GetProcessById(Int32 processId)
   at Microsoft.Diagnostics.Tools.Trace.CollectCommandHandler.Collect(CancellationToken ct, IConsole console, Int32 processId, FileInfo output, UInt32 buffersize, String providers, String profile, TraceFileFormat format, TimeSpan duration) in /_/src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs:line 89

3. The PID reported by ps is not a valid .NET app when running collect

root@17be50bbd0fd:/# dotnet trace collect -p 1
No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0xFFFFFFFFFFFFFFFF  Verbose(5)          --profile
Microsoft-Windows-DotNETRuntime         0x00000004C14FCCBD  Informational(4)    --profile

[ERROR] System.PlatformNotSupportedException: Process 1 not running compatible .NET Core runtime
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.GetTransport(Int32 processId) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 50
   at Microsoft.Diagnostics.Tools.RuntimeClient.DiagnosticsIpc.IpcClient.SendMessage(Int32 processId, IpcMessage message, IpcMessage& response) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/DiagnosticsIpc/IpcClient.cs:line 84
   at Microsoft.Diagnostics.Tools.RuntimeClient.EventPipeClient.CollectTracing(Int32 processId, SessionConfiguration configuration, UInt64& sessionId) in /_/src/Microsoft.Diagnostics.Tools.RuntimeClient/Eventing/EventPipeClient.cs:line 80
   at Microsoft.Diagnostics.Tools.Trace.CollectCommandHandler.Collect(CancellationToken ct, IConsole console, Int32 processId, FileInfo output, UInt32 buffersize, String providers, String profile, TraceFileFormat format, TimeSpan duration) in /_/src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs:line 104

Notes:

1. I was using the dotnet trace version 3.1.57502+6767a9ac24bde3a58d7b51bdaff7c7d75aab9a65
2. dotnet trace worked as expected if I added it within my application container. This required me to install to .NET SDK since it is a global tool.
3. It is possible I am messed up something obvious here and this is something that is already supported 😄

[hoyosjs]

@josalem

[josalem]

This is something we are actively working on lighting up, but is unsupported right now. There are some PRs in flight that will enable this scenario in the 5.0 timeframe. dotnet/runtime#1600 and #770 when merged will make this easy to configure. We are thinking of changing the names of the configuration variables from what was merged in the runtime PR, but the functionality should be the same. Documentation on how we recommend setting this scenario up will come before release. It will involve a shared volume mount inside the pod.

CC @shirhatti

[SidShetye]

Our use case: When encountering a low memory situation in a container (e.g. host = 16 GB, each containers = 2GB), the tools within that same container cannot do a memory dump because the tool doesn't have enough memory either. But if there was another container, this would be much simpler.

[bss-git]

.NET Core app on linux creates domain socket files in /tmp catalog. You can establish IPC session with this file.
You should start your target container with option that maps /tmp somewhere to host, e.g.
--v /tmp/container_sockets:/tmp
And start your tracing container with option that maps host catalog to /tmp in container:
--v /tmp/container_sockets:/tmp
(and with other your options like --pid).
Then if you start tracing it should just work.

Pid in this case in nothing more than abstraction. Internally Microsoft.Diagnostics.NETCore.Client.IpcClient uses pid just to find socket file name and start an IPC session:

                string ipcPort;
                try
                {
                    ipcPort = Directory.GetFiles(IpcRootPath, $"dotnet-diagnostic-{processId}-*-socket") // Try best match.
                                .OrderByDescending(f => new FileInfo(f).LastWriteTime)
                                .FirstOrDefault();
                    if (ipcPort == null)
                    {
                        throw new ServerNotAvailableException($"Process {processId} not running compatible .NET Core runtime.");
                    }
                }

I've configured intercontainer IPC in my monitoring app so it collects counters just by socket file names from two other containers. You even don't need target pid in this scenario.

[glitch100]

@MichaelSimons is this still the primary issue for this? I saw in #1737 that you are using this for tracking?

Following the steps in the issue above still has issues with the PIDs being inaccurate between ps -aux and dotnet trace ps. I am not sure if I am missing something with the sidecar approach or if the socket based approach posted by @bss-git is the way it should be done. Is there any documentation for this?

[MichaelSimons]

@glitch100 - as far as I am aware, this is the primary issue.

@shirhatti - can you help @glitch100?

[shirhatti]

@glitch100 As noted in an earlier comment, access to the PID namespace isn't really required.

You just need access to the diagnostic server created by the runtime. As it stands today (3.0/3.1), this socket is always created in the /tmp directory.

Tooling and runtime changes are incoming for 5.0 that allow you customize how the diagnostics server is created and how the tools attach.

@bss-git's suggestion of sharing the temp directory across both containers should suffice.

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS tools
RUN dotnet tool install --tool-path /tools dotnet-trace

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS runtime

COPY --from=tools /tools /tools
ENV PATH="/tools:${PATH}"

ENV COMPlus_EnableDiagnostics="0"

WORKDIR /tools

docker run -d -p 8000:80 -v /container:/tmp mcr.microsoft.com/dotnet/core/samples:aspnetapp
docker run --rm -u root -it -v /container:/tmp trace /bin/sh

# Inside the trace container
dotnet-trace collect -p 1

[glitch100]

@shirhatti
Firstly for this to work will my dotnet-app also need:

ENV COMPlus_EnableDiagnostics="0"

In the dockerfile or we hit that issue around CoreClr starting (Due to /tmp directory.)

Once running, and having another container with the Dockerfile you provided I still hit an issue:

No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0x0000000000000000  Informational(4)    --profile 
Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile 

Unable to start a tracing session: Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Process 1 not running compatible .NET Core runtime.
No profile or providers specified, defaulting to trace profile 'cpu-sampling'

Provider Name                           Keywords            Level               Enabled By
Microsoft-DotNETCore-SampleProfiler     0x0000000000000000  Informational(4)    --profile 
Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile 

Unable to start a tracing session: Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Process 1 not running compatible .NET Core runtime.
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.GetTransport(Int32 processId) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 63
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessage(Int32 processId, IpcMessage message, IpcMessage& response) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 104
   at Microsoft.Diagnostics.NETCore.Client.EventPipeSession..ctor(Int32 processId, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 30
   at Microsoft.Diagnostics.Tools.Trace.CollectCommandHandler.Collect(CancellationToken ct, IConsole console, Int32 processId, FileInfo output, UInt32 buffersize, String providers, String profile, TraceFileFormat format, TimeSpan duration, String clrevents, String clreventlevel) in /_/src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs:line 130
Unable to create session.

I have confirmed that both have volumes for /tmp mapped to the host.

Could you provide a working example? Also is this documented anywhere?

[shirhatti]

Firstly for this to work my dotnet-app also will need:
ENV COMPlus_EnableDiagnostics="0"

That's not going to work. If you disable the diagnostics server, you can't trace the application.

If creation of pipes fails, do you mind creating a new issue for that? Let's work through that first.

Could you provide a working example?

I'll publish a gist later today.

[glitch100]

@shirhatti Any luck on that gist?

[glitch100]

Worth noting that I have tried what you posted with a vanilla ASPNET app with docker support (As per Visual Studio), as well as the trace container. I have tried in both Docker Compose and via the CLI with the commands above.

Docker Compose gives me that tmp CORECLR issues so I suspect I am doing something wrong there regards to the volumes/mounts.

docker run... successfully starts both containers, but again I get the same error as I posted above.

I look forward to seeing the gist as I am hoping I have done something silly rather than this being an issue on the dotnet side.