Diagnostics: Fixes possible Thread contention in Traces (#5455)

# Pull Request Template

## Description

This PR fixes a regression in tracing introduced by
#5422, which can
lead to thread contention. The bug is that the locking expects
Trace-names to be unique for different logical operations - but many of
the strings passed into new root traces (logical operations) are not new
string instances (with possibly same value) but constants - so same
singelton string instances. In tehse cases the new code will provide
unnecessary lock contention. The locking scope should be just the scope
of the ITrace instance - so, this PR introduces a new instance-level
lock object (something I tried to avoid initially to avoid allocation of
a new object-instance per trace). The alternative (Id - is a ValueType)
and (children is a List that is exposed via properties - locking it
would not be guaranteed to be dead-lock free - woudl be unliekly but not
impossble if caller locks on the returned value from the Childred
property).

## Type of change

Please delete options that are not relevant.

- [x] Bug fix (non-breaking change which fixes an issue)
- [] New feature (non-breaking change which adds functionality)
- [] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [] This change requires a documentation update

## Closing issues

To automatically close an issue: closes #IssueNumber

Author: FabianMeiswinkel

Microsoft.Azure.Cosmos/src/Tracing/Trace.cs

@@ -12,6 +12,7 @@ namespace Microsoft.Azure.Cosmos.Tracing
     internal sealed class Trace : ITrace
     {
         private static readonly IReadOnlyDictionary<string, object> EmptyDictionary = new Dictionary<string, object>();
+        private readonly object lockObject;
         private volatile List<ITrace> children;
         private volatile Dictionary<string, object> data;
         private ValueStopwatch stopwatch;
@@ -25,6 +26,7 @@ private Trace(
             TraceSummary summary)
         {
             this.Name = name ?? throw new ArgumentNullException(nameof(name));
+            this.lockObject = new object();
             this.Id = Guid.NewGuid();
             this.StartTime = DateTime.UtcNow;
             this.stopwatch = ValueStopwatch.StartNew();
@@ -122,7 +124,7 @@ public ITrace StartChild(
 
         public void AddChild(ITrace child)
         {
-            lock (this.Name)
+            lock (this.lockObject)
             {
                 if (!this.isBeingWalked)
                 {
@@ -172,7 +174,7 @@ public void AddDatum(string key, TraceDatum traceDatum)
 
         public void AddDatum(string key, object value)
         {
-            lock (this.Name)
+            lock (this.lockObject)
             {
                 this.data ??= new Dictionary<string, object>();
 
@@ -192,7 +194,7 @@ public void AddDatum(string key, object value)
 
         public void AddOrUpdateDatum(string key, object value)
         {
-            lock (this.Name)
+            lock (this.lockObject)
             {
                 this.data ??= new Dictionary<string, object>();
 
@@ -217,7 +219,7 @@ internal void SetWalkingStateRecursively()
                 return; // Already set, return early
             }
 
-            lock (this.Name)
+            lock (this.lockObject)
             {
                 if (this.isBeingWalked)
                 {
@@ -239,7 +241,7 @@ internal void SetWalkingStateRecursively()
 
         bool ITrace.TryGetDatum(string key, out object datum)
         {
-            lock (this.Name)
+            lock (this.lockObject)
             {
                 if (this.data == null)
                 {