dumpheap should show progress on large dumps

[danmoseley]

dumpheap -stat on a 19 GB dump (248M objects) completes successfully in ~66 seconds, but produces zero output during the heap walk phase. On a large dump this feels indistinguishable from a hang -- the user sees no CPU activity in the console, no progress indication, and no way to know whether the command is working or stuck.

This came up during investigation of a C# Dev Kit memory leak. We initially tried dotnet-dump analyze interactively, sent dumpheap -stat, and waited over 10 minutes with apparently zero CPU or disk activity before giving up. We then wrote a custom ClrMD script using DataTarget.LoadDump() and heap.EnumerateObjects() with periodic progress output -- that completed in ~83 seconds and successfully analyzed the dump. It was only later, by attaching dotnet-stack to the dotnet-dump process and running non-interactively with file redirection (dotnet-dump analyze <dump> -c "dumpheap -stat" > output.txt), that we confirmed dotnet-dump was actually working fine the whole time.

What happens today

1. User runs dumpheap -stat on a large dump
2. The heap walk takes 60+ seconds with no output at all
3. Then 8,649 lines (1.46 MB) of results are written all at once
4. In an interactive/piped terminal, the burst of output can also cause pipe buffer backpressure, further contributing to the appearance of a hang

Suggestion

Print periodic progress to stderr during heap enumeration, e.g.:

> dumpheap -stat
Scanning heap: 50,000,000 / ~248,000,000 objects (3.2 GB / 16.3 GB)...
Scanning heap: 100,000,000 / ~248,000,000 objects (6.5 GB / 16.3 GB)...
...

The total object count may not be known in advance, but total heap size is available from segment metadata and bytes scanned could serve as the progress metric. Writing to stderr avoids polluting stdout for piped/redirected scenarios.

Environment

• dotnet-dump 9.0.661903
• 19.3 GB dump, .NET 11.0.26.10518, Server GC, 248M objects
• Windows, 64 GB RAM

Related issues

• dotnet-dump is incredibly slow in dumpheap scenario #1637 -- dumpheap was rewritten for performance (closed, fixed in SOS v7). The perf is now good; this issue is about the remaining UX gap.
• Single File module scan takes a very long time (symbol lookup) #3737 -- Similar "appears hung" UX problem caused by slow symbol loading with no progress feedback.

[danmoseley]

This may be more relevant how we're using AI to run it. In my case Copilot gave up as it apparently wasn't making progress - and it had zero CPU and disk part of this time, perhaps because the heavy IO was attributed to System process - so it killed it and gave up. Whereas a human might be more familiar and willing to wait without indication of progress.

[hoyosjs]

Does copilot read stderr? Usually, we don't report progress on tools if the output is redirected (counters and trace have a precedent on this) - by design we consider it as a non-interactive request. We might need to reevaluate for things like agents

@noahfalk @steveisok

[danmoseley]

Yes I have seen it read stderr often, although I have seen it run tools, they error with code 1, then try again redirecting stderr as well.

I think we should rethink all our tools behaviors' suitability for an agentic world. It may not be long before they are the primary users. Their output, progress etc need to suit agents well. cc @richlander as he's thought about this.

[noahfalk]

interactively, sent dumpheap -stat, and waited over 10 minutes with apparently zero CPU or disk activity before giving up

I know its not directly what you were asking about, but 10 minutes without output/CPU/disk activity when CLRMD finished in 83 seconds certainly sounds like something wasn't working right.

We might need to reevaluate for things like agents

Even for humans the current behavior sounds sub-optimal :) I expect if most users run FancyDumpHeapCommand > dumpheap.txt and it is still running after a minute then they'd probably like to see some progress somewhere.

I think we should rethink all our tools behaviors' suitability for an agentic world

Not that we shouldn't try to make our tools better, but ultimately it might easier and more powerful for Copilot to drop right into the APIs behind the tools. For example instead of downloading and running a command-line tool Copilot could write and execute a small file based app that gives it the exact analysis and output format best suited for the situation:

   #:package Microsoft.Diagnostics.Runtime@3.* 
   using Microsoft.Diagnostics.Runtime;
 
   using DataTarget target = DataTarget.LoadDump({filename});
   ClrRuntime runtime = target.ClrVersions[0].CreateRuntime();
   ClrHeap heap = runtime.Heap;
   var counts = new Dictionary<string, int>(StringComparer.Ordinal);  
   foreach (ClrObject obj in heap.EnumerateObjects())
   {
       string? name = obj.Type?.Name;
       if (name is null) continue;
       counts[name] = counts.TryGetValue(name, out int n) ? n + 1 : 1;
   }
   Console.WriteLine($"{"Count",12}  Type");
   Console.WriteLine(new string('-', 60));
   foreach (var entry in counts.OrderByDescending(x => x.Value))
       Console.WriteLine($"{entry.Value,12}  {entry.Key}");

[danmoseley]

"Copilot should" -- I don't know how easy it is to change what copilot (and all other agents) do, so it may be more interesting to optimize our tools for what we see them actually doing, or any emerging standard.

Sometimes it is happy to write scripts for itself, but as a counter example, it prefers to grep C# code rather than use Roslyn API over it.

[richlander]

I have been using stderr a lot for related needs.

There are really two separate topics:

• Reporting progress to avoid the sense of a process hanging
• Picking the form factor (tools, MCP, skill) per the knowledge required and interaction model (instant, streaming, blocking-batch)
• Determining what the right output format is for agents, both format and quantity

The current experience sounds like a problem for agents. Objectively, 1.6MB of stdout output sounds like a problem. The ideal case is that the tool writes a log and then is able to answer much quicker questions about the log.

[danmoseley]

@richlander do we feel that in general tools will emit output in ways balance both human and agent readers? Or tools will have two ways to emit, and essentially there's an --agent-output-format to pick?

It may well be in this particular case that output would be the same -- some limited but steady progress indication. But if we were discussing how it presents the output, I could easily imagine that a human eyeball would want one thing, an agent might want something more easy to parse.

I think you gave me an example of /? in the conventional form vs some json form.

[richlander]

As a team, we have progressively adopted one scheme for process launch args, with S.CL. I think we should move to a uniform way of writing logging output. As we learn, we can update our fleet of tools more easily.

This line of thinking is why I made: https://github.com/richlander/markout. It is what I'm using for all my tools.

I don't think --agent-output-format is a good idea. Tools should have skills that describe how to use them.

Progress should be course grained. git, docker, and curl are all anti-patterns with fine-grained progress. here's an anti-pattern I recorded: https://gist.github.com/richlander/e0146e0e613f7f5120c1de0a920b135b. AFAICT, Copilot is the most prone to high-detail progress being a problem.

[richlander]

@baronfel

[danmoseley]

I did some more systematic testing on the same 19.3 GB dump (248M objects, dotnet-dump 9.0.661903) to characterize which commands show progress and which appear hung. For each command I captured stdout and stderr separately, polling every 5 seconds to measure output growth.

Commands with zero progress indication (dead silence on both stdout and stderr)

Command	Silent Duration	Total Wall Time	Output Pattern
verifyheap	7 min 10s	7:16	Nothing, then "248,737,850 objects verified"
dumpheap -stat	~80s	1:30	Nothing, then 1.49 MB all at once
dumpasync	~60s	1:10	Nothing, then 7.9 MB burst
gcheapstat	~45s	0:50	Nothing, then a few lines

Commands that stream output continuously (visible within seconds)

Command	Output Pattern
dumpheap (no -stat)	~3.3 MB/5s steady stream
dumpheap -mt <addr>	~3.3 MB/5s steady stream
finalizequeue	~3.2 MB/5s steady stream
gchandles	~2.9 MB/5s steady stream
eeheap -gc	Fast (5s total, reads segment metadata only)

Observations

• verifyheap is the worst offender — over 7 minutes with zero output on either stream, indistinguishable from a hang.
• The streaming commands prove the heap walk itself produces data continuously; the silent commands just accumulate results internally before writing anything.
• stderr is completely unused by all commands tested — it could carry progress without polluting stdout.
• dumpheap -stat takes ~80s of silence, which matches the issue description. With warm cache it's tolerable but annoying; with cold cache or over a network share this could easily be 10+ minutes.

All times are with warm disk cache (second+ run). Cold-cache times would be significantly longer, making the silence even more problematic. In the original cold-cache scenario, dotnet-dump didn't even show meaningful CPU or disk activity in Task Manager — the time was spent waiting for the kernel to page-fault in dump contents, which is attributed to the System process rather than dotnet-dump itself. So there was literally no visible sign of life.

Ask

Any command that walks the heap should print periodic progress to stderr — ideally a percentage based on bytes scanned vs. total heap size (available from segment metadata), but even just a "." every few seconds would be a huge improvement. The bar should be: no more than ~10 seconds should ever pass without some visible indication that the tool is still working.

[danmoseley]

If you do want what is described immediately above this seems like a great candiate to assign to copilot...

[danmoseley]

Since it's 30 seconds these days, I had copilot create a PR in case you want it.

[noahfalk]

Yeah these days we are rapidly trending towards the bottleneck is around design and review rather than generating the code :)