Panic and debug session hang caused by re-entrant events when executing LLDB custom Python command

[johngladp]

OS: Windows on client, Ubuntu server 22.04 on remote.
VSCode version: 1.101.1
CodeLLDB version: 1.11.5
Compiler: clang++ v19.1.7
Debuggee: Linux x64 executable.

• Panic caused by LLDB sending a breakpoint event whilst a custom Python command is still executing:
  "thread 'main' panicked at adapter/codelldb/src/debug_session/breakpoints.rs:644:16:
  Cannot start a runtime from within a runtime. This happens because a function (like block_on) attempted to block the current thread while the thread is being used to drive asynchronous tasks."
• The panic hangs the debug session. VSCode doesn't ever jump to the breakpoint that has been hit.
• The breakpoint in question is on the first line of main(). It seems I can place breakpoint elsewhere in the code, this is a race condition, the later in the processes execution the breakpoint occurs the less likely the issue is to occur, presumably because this is a race to exit the Python command before the breakpoint event is fired.
• Adding "debugger.SetAsync(True)" to the beginning of the Python command mostly remedies the issue but I suspect we're just altering the "race conditions", not actually addressing the issue.
• The custom Python command where the issue occurs has called target.Launch(). We see "Executing Launch...", the "DONE" message at the end of the command is never printed:

• Fragment of the launch.json being used:
  

Verbose log

  
thread 'main' panicked at adapter/codelldb/src/debug_session/breakpoints.rs:644:16:
Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: tokio::runtime::context::runtime::enter_runtime
   3: codelldb::debug_session::breakpoints::::init_bp_actions::{{closure}}
   4: lldb::sb::sbbreakpoint::SBBreakpoint::set_callback::callback_thunk
   5: _ZN4lldb25SBBreakpointCallbackBaton28PrivateBreakpointHitCallbackEPvPN12lldb_private24StoppointCallbackContextEmm
   6: 
   7: 
   8: 
   9: 
  10: 
  11: 
  12: 
  13: 
  14: 
  15: 
  16: 
  17: _ZN4lldb8SBTarget6LaunchERNS_12SBLaunchInfoERNS_7SBErrorE
  18: 
  19: 
  20: PyObject_Call
  21: _PyEval_EvalFrameDefault
  22: 
  23: 
  24: 
  25: 
  26: 
  27: 
  28: 
  29: 
  30: _ZN4lldb20SBCommandInterpreter13HandleCommandEPKcRNS_18SBExecutionContextERNS_21SBCommandReturnObjectEb
  31: _ZN4lldb20SBCommandInterpreter13HandleCommandEPKcRNS_21SBCommandReturnObjectEb
  32: __cpp_closure_17879338975969274467
  33: lldb::strings::with_cstr
  34: lldb::sb::sbcommandinterpreter::SBCommandInterpreter::handle_command
  35: codelldb::debug_session::DebugSession::exec_commands
  36: codelldb::debug_session::launch::::complete_launch
  37: codelldb::debug_session::launch::::handle_launch::{{closure}}
  38: codelldb::debug_session::DebugSession::handle_request::{{closure}}
  39: tokio::runtime::task::core::Core::poll
  40: tokio::runtime::task::raw::poll
  41: tokio::task::local::LocalSet::tick
  42: ::poll
  43: codelldb::run_debug_session::{{closure}}
  44: tokio::runtime::context::runtime::enter_runtime
  45: tokio::runtime::runtime::Runtime::block_on
  46: codelldb::debug_server
  47: codelldb::main
Full trace - PANIC callstack is on line 140:
Verbose logging on.txt

[vadimcn]

What does dp_process_create do?

[johngladp]

Below is a slightly stripped down version of that command, full Python file which will enable re-production hopefully.

The command is setting up arguments on the target and launching the process.

My current workaround at the moment is to remove the target.Launch(..) call and put process launch in the "processCreateCommand" array in launch.json instead.

import inspect
import lldb
from lldb.plugins.parsed_cmd import ParsedCommand
import sys


# __________________________________________________________________________________________________
def make_launch_info(
        debugger: lldb.SBDebugger,
        command_line_arguments: list[str],
        use_remote_tty: bool) -> lldb.SBLaunchInfo:

    target = debugger.GetSelectedTarget()
    if not target.IsValid():
        raise RuntimeError("No valid target selected")

    launch_info = target.GetLaunchInfo()

    launch_info.SetArguments(command_line_arguments, append=False)

    # -- stdio paths -----------------------------------------------------
    #
    # These three setters correspond to:
    #   settings set target.input-path  /dev/tty   (and output/error)
    #
    if (use_remote_tty):
        launch_info.SetStandardInputPath("/dev/tty")
        launch_info.SetStandardOutputPath("/dev/tty")
        launch_info.SetStandardErrorPath("/dev/tty")

    return launch_info


# __________________________________________________________________________________________________
class DPProcessCreate(ParsedCommand):
    program = "dp_process_create"

    # ______________________________________
    @classmethod
    def register_lldb_command(cls, debugger):
        ParsedCommand.do_register_cmd(cls, debugger, __name__)

    # ______________________________________
    def __init__(self, debugger, dict):
        """This call should initialize the command with respect to the command interpreter for the passed-in debugger"""
        super().__init__(debugger, dict)

    # ______________________________________
    def setup_command_definition(self):
        """This is called when your command is added to lldb."""
        try:
            parser = self.get_parser()

            parser.add_option(
                "u",
                "use-remote-tty",
                dest="use_remote_tty",
                help="Redirect stdio to the remote platform’s TTY instead of /dev/tty "
                "on the host.",
                default=False,
                required=False,
                value_type=lldb.eArgTypeBoolean)

            run_elem = parser.make_argument_element(lldb.eArgTypeRunArgs,
                                                    repeat="optional")
            parser.add_argument_set([run_elem])
        except Exception as e:
            print(f"Exception in setup_command_definition: {e}", flush=True)

    # ______________________________________
    def __call__(self, debugger, args_array, exe_ctx, result):
        try:
            print(f"{self.program} starting...", flush=True)

            run_args = [
                args_array.GetItemAtIndex(i).GetStringValue(4096)
                for i in range(args_array.GetSize())
            ]

            parser = self.get_parser()
            use_remote_tty = parser.use_remote_tty

            result.AppendMessage(f"run-args     : {run_args!r}")
            result.AppendMessage(f"use-remote-tty : {use_remote_tty}")

            launch_info = make_launch_info(
                debugger,
                run_args,
                use_remote_tty)

            target = debugger.GetSelectedTarget()
            target.SetLaunchInfo(launch_info)

            err = lldb.SBError()
            print("Executing launch...", flush=True)
            process = target.Launch(launch_info, err)
            if err.Fail():
               raise RuntimeError(err.GetCString())
            if not process.IsValid():
               raise RuntimeError("Failed to launch process")
            result.AppendMessage(f"Launched process: {process.GetProcessID()}")
            result.SetStatus(lldb.eReturnStatusSuccessFinishResult)

            print(f"{self.program} DONE.", flush=True)
        except Exception as ex:
            print(f"Exception in __call__: {ex}", flush=True)
            result.SetError(f"Exception in __call__: {ex}")

            # ______________________________________
    def get_short_help(self):
        """This call should return the short help text for this command[1]"""
        return None

    # ______________________________________
    def get_long_help(self):
        """This call should return the long help text for this command[1]"""
        return None

    # ______________________________________
    def get_flags(self):
        """This will be called when the command is added to the command interpreter,
        and should return a flag field made from or-ing together the appropriate
        elements of the lldb.CommandFlags enum to specify the requirements of this command.
        The CommandInterpreter will make sure all these requirements are met, and will        
        return the standard lldb error if they are not."""
        try:
            return 0
        except Exception as e:
            print(f"Exception in get_flags: {e}")

    # ______________________________________
    def get_repeat_command(self, command):
        return ""


# __________________________________________________________________________________________________
def __lldb_init_module(debugger, internal_dict):
    """Setup the LLDB commands we adding to facilitate debugging."""
    print(f"Running __lldb_init_module in {__file__}.")
    try:
        # Register all classes that have a register_lldb_command method
        for _name, cls in inspect.getmembers(sys.modules[__name__]):
            if inspect.isclass(cls) and callable(
                getattr(cls, "register_lldb_command", None)
            ):
                cls.register_lldb_command(debugger)
    except Exception as e:
        print(f"Exception in __lldb_init_module: {e}")

[vadimcn]

I see. I will have a look at what can be done about re-entrancy. However, the way I envisioned similar scenarios working was that you would modify the target launch info with your script but let CodeLLDB perform the actual launch. Or at least use the process launch command rather than target.Launch() in Python.