fix(sdk/metric): avoid deadlock when registering instruments in callb…

[garyatsierra]

Summary

• Fix deadlock in pipeline.produce() that occurs when an observable callback registers a new instrument (e.g. creates a
  counter via meter.Int64Counter())
• produce() held the pipeline mutex while executing callbacks; if a callback called addSync() (via instrument creation),
  it tried to re-acquire the same mutex → deadlock
• Fix: snapshot callbacks under the lock, release it, execute callbacks lock-free, then re-acquire for the aggregation phase
• Add regression test TestPipelineNoDeadlockOnInstrumentCreationDuringCallback with a 5s timeout to catch deadlocks

Details

We ran into this and we were a bit surprised by the behavior of being able to deadlock from using sdk apis. The test shows the scenario doing which is on measure we incr a counter if it met a certain threshold and would deadlock only if it wasn't cached.

Looking at the doc we do see:

Callback functions SHOULD be reentrant safe. The SDK expects to evaluate callbacks for each MetricReader independently.

In our case we weren't acquiring and outside mutex but rather the pipeline itself was trying to reaquire the pipeline deadlock

The deadlock call chain was:

produce() [holds p.Lock()]
  → callback execution
    → meter.Int64Counter()
      → cachedAggregator()
        → pipeline.addSync() [tries p.Lock()]
          → DEADLOCK

Executing callbacks without the lock is safe because:

• Callbacks operate on aggregate.Measure functions which use atomic operations internally
• The callbacks and multiCallbacks data structures are snapshotted under the lock before release
• Concurrent addCallback/addMultiCallback calls are still properly synchronized

[linux-foundation-easycla]

• ✅ login: garyatsierra / name: Gary Tsang (06835cf)
• ✅ login: garyatsierra / name: garyatsierra (6d17d62)
• ❌ The email address for the commit (06835cf) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please submit a support request ticket.

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla