Skip to content

Deadlock in EventPipe when reader tries to take EventPipe config lock while tracing self #1892

Open
@josalem

Description

@josalem

As detailed in #1794, when an application is tracing itself there is the potential for a deadlock to occur if the reader tries to take the EventPipe configuration lock while EventPipe::Disable (specifically EventPipe::RunWithCallbackPostponed) is in progress.

This occurs when EventPipeEventSource is reading because the Trace Event library uses Regex in its logic and Regex now has a lazily initialized ConcurrentDictionary cache that generates events when used. This causes the static EventSource provider for concurrent collections to get created (unless its been instantiated earlier in the process) which will call into native code to create the provider. This requires the config lock in EventPipe. If this occurs during EventPipe::RunWithCallbackPostponed, you may deadlock because the reader is waiting on the lock, but the writer is holding it and is blocked because the pipe's buffer is full.

See the stacks in #1794 for details.

A couple potential paths to fixing this:

  • break up the logic of EventPipe::Disable so that it isn't holding the config lock for the entire duration. Rundown and other parts of disable can run for minutes in big applications, so this shouldn't be holding a global lock like that the entire time.
  • find ways to defer the registration of new providers that get registered while EventPipe::Disable is in flight.

This issue breaks our EventPipe tests, but #1794 added a bypass that needs to be removed once this issue is closed.

CC - @tommcdon @noahfalk @sywhang

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions