Optimize nettrace-to-TraceLog Conversion (#2403)

* Use min-heap for EventCache.SortAndDispatch

Replace the O(N*T) linear-scan merge in SortAndDispatch with an O(N*log(T))
min-heap merge, where N is the number of events and T is the number of
threads. The previous implementation rebuilt a List from LINQ on every call
and linearly scanned all thread queues for the minimum timestamp per event.

The new implementation uses an array-backed binary min-heap keyed by
timestamp. After extracting the minimum, only a single O(log T) sift-down
is needed to restore the heap property. The heap list is reused across
calls to avoid per-call allocations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Track active thread queues in EventCache.SortAndDispatch

Maintain a HashSet of thread queues that have pending events instead of
iterating all threads in the dictionary on every SortAndDispatch call.
Queues are added to the active set when their first event is enqueued
and removed when drained. This eliminates the Dictionary.Values
enumeration which was ~28% of CPU during nettrace-to-TraceLog conversion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Cache ParsedSymbolMetadata to avoid repeated JSON deserialization

Cache the result of ProcessMappingSymbolMetadataParser.TryParse() in
ProcessMappingMetadataTraceData so that repeated accesses to the
ParsedSymbolMetadata property do not re-invoke JSON deserialization.
The property is accessed twice per mapping event (for PE and ELF
metadata checks), and the metadata objects are shared across multiple
mappings via MetadataId. This was ~10% of CPU during conversion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Avoid redundant FileName string allocations in ProcessMapping handler

Read ProcessMappingTraceData.FileName once into a local variable and pass
it directly to UniversalMapping(string, ...) instead of going through the
UniversalMapping(ProcessMappingTraceData, ...) overload. Previously,
FileName was accessed 3 times per mapping event (IsNullOrEmpty check,
StartsWith check, and inside UniversalMapping), each time allocating a
new string via GetShortUTF8StringAt(). String allocation was ~12% of CPU
in Release-mode profiling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reset ParsedSymbolMetadata cache in Dispatch to prevent stale data

TraceEvent objects are reused across callbacks. The cached
ParsedSymbolMetadata fields were never cleared between dispatches,
which could return metadata from a previous event if the property
was accessed on the template object rather than a clone.

Reset _parsedSymbolMetadataCached and _parsedSymbolMetadata at the
start of Dispatch() before invoking the callback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address code review feedback: MinHeap class, Clone override, comments

Refactor the min-heap helpers into a self-contained private MinHeap
class with XML doc comments on all public methods. Add comments
explaining the binary heap child index formulas (2i+1, 2i+2). Use
C# tuple swap syntax instead of a temp variable.

Add Clone() override to ProcessMappingMetadataTraceData to explicitly
copy the cached ParsedSymbolMetadata fields into the clone. Strings
are immutable so a shallow copy of the reference is sufficient.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add MinHeap unit tests and improve Build() comment

Make MinHeap generic (MinHeap<TValue>) and internal so it can be
tested from the test project. Add 13 unit tests covering: empty heap,
single element, ascending/descending/random input, duplicate keys,
ReplaceRoot, RemoveRoot, Clear, and mixed operations. Add a comment
to Build() explaining why iteration starts at Count/2-1.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: brianrob

src/TraceEvent/EventPipe/EventCache.cs

@@ -1,7 +1,6 @@
 using System;
 using System.Collections.Generic;
 using System.Diagnostics;
-using System.Linq;
 using System.Runtime.InteropServices;
 
 namespace Microsoft.Diagnostics.Tracing.EventPipe
@@ -68,6 +67,10 @@ public void ProcessEventBlock(Block block)
                 }
                 else
                 {
+                    if (thread.Events.Count == 0)
+                    {
+                        _activeThreadQueues.Add(thread.Events);
+                    }
                     thread.Events.Enqueue(eventMarker);
                 }
 
@@ -165,49 +168,178 @@ private void CheckForPendingThreadRemoval()
 
         private unsafe void SortAndDispatch(long stopTimestamp)
         {
-            // This sort could be made faster by using a min-heap but this is a simple place to start
-            List<Queue<EventMarker>> threadQueues = new List<Queue<EventMarker>>(_threads.Values.Select(t => t.Events));
-            while(true)
+            // Build a min-heap from active thread queues (those with pending events) whose
+            // front event is before stopTimestamp. Using _activeThreadQueues avoids iterating
+            // all threads in the dictionary — only threads that have had events enqueued are checked.
+            // This gives O(N * log(T)) merge performance where N is the number of events and
+            // T is the number of active threads.
+            _heap.Clear();
+            foreach (Queue<EventMarker> q in _activeThreadQueues)
             {
-                long lowestTimestamp = stopTimestamp;
-                Queue<EventMarker> oldestEventQueue = null;
-                foreach(Queue<EventMarker> threadQueue in threadQueues)
+                if (q.Count > 0)
                 {
-                    if(threadQueue.Count == 0)
+                    long ts = q.Peek().Header.TimeStamp;
+                    if (ts < stopTimestamp)
                     {
-                        continue;
+                        _heap.Add(ts, q);
                     }
-                    long eventTimestamp = threadQueue.Peek().Header.TimeStamp;
-                    if (eventTimestamp < lowestTimestamp)
+                }
+            }
+
+            if (_heap.Count == 0)
+            {
+                return;
+            }
+
+            _heap.Build();
+
+            // Merge events in timestamp order using the min-heap.
+            while (_heap.Count > 0)
+            {
+                Queue<EventMarker> minQueue = _heap.PeekValue;
+                EventMarker eventMarker = minQueue.Dequeue();
+                OnEvent?.Invoke(ref eventMarker.Header);
+
+                if (minQueue.Count > 0)
+                {
+                    long nextTs = minQueue.Peek().Header.TimeStamp;
+                    if (nextTs < stopTimestamp)
                     {
-                        oldestEventQueue = threadQueue;
-                        lowestTimestamp = eventTimestamp;
+                        // Update the root with the next timestamp and restore the heap property.
+                        _heap.ReplaceRoot(nextTs, minQueue);
+                    }
+                    else
+                    {
+                        _heap.RemoveRoot();
                     }
                 }
-                if(oldestEventQueue == null)
+                else
                 {
-                    break;
+                    _heap.RemoveRoot();
+                    // Remove from active set and free internal storage to prevent unbounded
+                    // memory growth when the application creates and destroys threads.
+                    _activeThreadQueues.Remove(minQueue);
+                    minQueue.TrimExcess();
+                }
+            }
+        }
+
+        #region Min-heap Implementation
+
+        /// <summary>
+        /// A min-heap that pairs a long key with a value of type <typeparamref name="TValue"/>.
+        /// Entries are ordered by key so the minimum key is always at the root.
+        /// </summary>
+        internal class MinHeap<TValue>
+        {
+            private struct Entry
+            {
+                public long Key;
+                public TValue Value;
+
+                public Entry(long key, TValue value)
+                {
+                    Key = key;
+                    Value = value;
+                }
+            }
+
+            private readonly List<Entry> _entries = new List<Entry>();
+
+            public int Count => _entries.Count;
+
+            public TValue PeekValue => _entries[0].Value;
+
+            public void Clear() => _entries.Clear();
+
+            public void Add(long key, TValue value)
+            {
+                _entries.Add(new Entry(key, value));
+            }
+
+            /// <summary>
+            /// Establishes the heap property over all entries. Call once after adding all
+            /// entries via Add, before extracting from the heap.
+            /// </summary>
+            /// <remarks>
+            /// Starts from the last non-leaf node (_entries.Count / 2 - 1) and sifts each
+            /// node down to its correct position. Leaves (the second half of the array) are
+            /// already trivially valid heaps of size 1, so they are skipped.
+            /// </remarks>
+            public void Build()
+            {
+                // Start from the last non-leaf node and work backwards to the root.
+                // Nodes at indices [Count/2 .. Count-1] are leaves that need no adjustment.
+                for (int i = _entries.Count / 2 - 1; i >= 0; i--)
+                {
+                    SiftDown(i);
+                }
+            }
+
+            /// <summary>
+            /// Replaces the root entry with a new key/value pair and restores the heap property.
+            /// Use when the root element has been consumed but its source still has more items.
+            /// </summary>
+            public void ReplaceRoot(long newKey, TValue value)
+            {
+                _entries[0] = new Entry(newKey, value);
+                SiftDown(0);
+            }
+
+            /// <summary>
+            /// Removes the root (minimum) entry from the heap and restores the heap property.
+            /// </summary>
+            public void RemoveRoot()
+            {
+                int lastIndex = _entries.Count - 1;
+                if (lastIndex == 0)
+                {
+                    _entries.Clear();
                 }
                 else
                 {
-                    EventMarker eventMarker = oldestEventQueue.Dequeue();
-                    OnEvent?.Invoke(ref eventMarker.Header);
+                    _entries[0] = _entries[lastIndex];
+                    _entries.RemoveAt(lastIndex);
+                    SiftDown(0);
                 }
             }
 
-            // If the app creates and destroys threads over time we need to flush old threads
-            // from the cache or memory usage will grow unbounded. AddThread handles the
-            // the thread objects but the storage for the queue elements also does not shrink
-            // below the high water mark unless we free it explicitly.
-            foreach (Queue<EventMarker> q in threadQueues)
+            /// <summary>
+            /// Restores the min-heap property by moving the element at index i down the tree
+            /// until it is smaller than both children or reaches a leaf position.
+            /// </summary>
+            private void SiftDown(int i)
             {
-                if(q.Count == 0)
+                int count = _entries.Count;
+                while (true)
                 {
-                    q.TrimExcess();
+                    int smallest = i;
+
+                    // In a binary heap stored as an array, the children of node i are at
+                    // indices 2i+1 (left) and 2i+2 (right).
+                    int left = 2 * i + 1;
+                    int right = 2 * i + 2;
+
+                    if (left < count && _entries[left].Key < _entries[smallest].Key)
+                    {
+                        smallest = left;
+                    }
+                    if (right < count && _entries[right].Key < _entries[smallest].Key)
+                    {
+                        smallest = right;
+                    }
+                    if (smallest == i)
+                    {
+                        break;
+                    }
+                    (_entries[i], _entries[smallest]) = (_entries[smallest], _entries[i]);
+                    i = smallest;
                 }
             }
         }
 
+        #endregion
+
         private void FreeOldEventBuffers(long stopTimestamp)
         {
             while (_buffers.Count > 0)
@@ -277,6 +409,8 @@ public EventBlockBuffer(FixedBuffer buffer, long maxTimestamp)
         EventPipeEventSource _source;
         ThreadCache _threads;
         Queue<EventBlockBuffer> _buffers = new Queue<EventBlockBuffer>();
+        MinHeap<Queue<EventMarker>> _heap = new MinHeap<Queue<EventMarker>>();
+        HashSet<Queue<EventMarker>> _activeThreadQueues = new HashSet<Queue<EventMarker>>();
     }
 
     internal class EventMarker

src/TraceEvent/Parsers/UniversalSystemTraceEventParser.cs

@@ -355,7 +355,20 @@ public sealed class ProcessMappingMetadataTraceData : TraceEvent
 
         public string SymbolMetadata {get { return GetShortUTF8StringAt(SkipVarInt(0)); } }
 
-        internal ProcessMappingSymbolMetadata ParsedSymbolMetadata { get { return ProcessMappingSymbolMetadataParser.TryParse(SymbolMetadata); } }
+        internal ProcessMappingSymbolMetadata ParsedSymbolMetadata
+        {
+            get
+            {
+                if (!_parsedSymbolMetadataCached)
+                {
+                    _parsedSymbolMetadata = ProcessMappingSymbolMetadataParser.TryParse(SymbolMetadata);
+                    _parsedSymbolMetadataCached = true;
+                }
+                return _parsedSymbolMetadata;
+            }
+        }
+        private ProcessMappingSymbolMetadata _parsedSymbolMetadata;
+        private bool _parsedSymbolMetadataCached;
 
         public string VersionMetadata {get {return GetShortUTF8StringAt(SkipShortUTF8String(SkipVarInt(0))); } }
 
@@ -367,9 +380,19 @@ internal ProcessMappingMetadataTraceData(Action<ProcessMappingMetadataTraceData>
         }
         protected internal override void Dispatch()
         {
+            _parsedSymbolMetadataCached = false;
+            _parsedSymbolMetadata = null;
             Action(this);
         }
 
+        public override unsafe TraceEvent Clone()
+        {
+            var clone = (ProcessMappingMetadataTraceData)base.Clone();
+            clone._parsedSymbolMetadata = _parsedSymbolMetadata;
+            clone._parsedSymbolMetadataCached = _parsedSymbolMetadataCached;
+            return clone;
+        }
+
         protected internal override Delegate Target
         {
             get { return Action; }

src/TraceEvent/SourceConverters/NettraceUniversalConverter.cs

@@ -62,15 +62,16 @@ public void BeforeProcess(TraceLog traceLog, TraceEventDispatcher source)
                 }
                 processes.Add(process);
 
-                if (!string.IsNullOrEmpty(data.FileName) && data.FileName.StartsWith(DotnetJittedCodeMappingName, StringComparison.Ordinal))
+                string fileName = data.FileName;
+                if (!string.IsNullOrEmpty(fileName) && fileName.StartsWith(DotnetJittedCodeMappingName, StringComparison.Ordinal))
                 {
                     // Don't create a module for jitted code.
                     // These will be created for each jitted code symbol.
                     return;
                 }
 
                 _mappingMetadata.TryGetValue(data.MetadataId, out ProcessMappingMetadataTraceData metadata);
-                TraceModuleFile moduleFile = process.LoadedModules.UniversalMapping(data, metadata);
+                TraceModuleFile moduleFile = process.LoadedModules.UniversalMapping(fileName, data.StartAddress, data.EndAddress, data.TimeStampQPC, metadata);
             };
             universalSystemParser.ProcessMappingMetadata += delegate (ProcessMappingMetadataTraceData data)
             {