Launch and Step does not properly step

[hintron]

When I run launch and step, I expect it to break the moment it enters main. However, it's hit and miss. Debug Subprocesses is false.

Attempts on raddbg 0.9.26:

1. Program launched as running (no step); Debugger view broke on the first line of main.
2. Program launched as running (no step).
3. Program launched as running (no step).
4. Program launched as running (no step); Debugger view broke on the first line of main.
5. Program launched as running (no step); Debugger view broke on the first line of main.
6. Program launched as running (no step); Debugger view broke on the first line of main.
7. Program launched as running (no step); Debugger view broke on the first line of main.
8. Program launched as running (no step); Debugger view broke on MSVC tuple code forward_as_tuple() definition.
9. Program launched as running (no step); Debugger view broke on the first line of main.
10. Program launched as running (no step); Debugger view broke on the first line of main.
11. Program launched as running (no step); Debugger view broke on a line in the MSVC utility pair code.
12. Program launched as running (no step); Debugger view broke on the first line of main.

I exited raddbg, started 0.9.25, and did some more Launch and Steps:

1. Program launched as running (no step).
2. Program launched as running (no step).
3. Program launched and stepped!; Debugger broke on the second or third line of my program (second or third function deep of first line).
4. Program launched as running (no step).
5. Program launched and stepped!; Debugger broke on the second or third line of my program (second or third function deep of first line).
6. Program launched and stepped!; Debugger broke on the first line of main.
7. Program launched and stepped!; Debugger broke on std::pair code in 7th line of second function call.
8. Program launched and stepped!; Debugger broke on the first line of main.
9. Program launched as running (no step).
10. Program launched as running (no step).

Deleting the .pdb and .rdi and rerunning it with 0.9.25 produced similar results.

Any ideas on what is going on here? Is Launch and Step known to be broken? (I seem to remember it not working consistently even a while back)

[hintron]

(Btw, I'm clicking the Launch and Step button from a Target, not from the top of the UI. I assume those trigger the same code paths though)

[ryanfleury]

It is not known to be broken; I can't seem to reproduce this. Would you be able to send an EXE/PDB for which you've reproduced the issue?

[hintron]

Unfortunately, I'm not allowed to share the exe/pdb of the thing that is not working (let's call that A).

So I tried to create a simple single-file reproducer (C++) with the same build system (let's call that B), but that program seems to mostly Launch and Step just fine. Obviously there is something weird with A. I'll try to see if I can make B look more like A to reproduce it and get back to you.

There is one issue I do see even with B, though: If I run A, and stop A, then the very next time I Launch and Step B, it skips the step part and only runs. But then on subsequent Launch and Steps of B, it steps. (Maybe this can give you a clue with what is happening with A, but at the very least, this shows some weirdness with previous runs messing up subsequent unrelated runs.) (Also, Running B and then a copy of the B target back and forth does NOT cause this issue, so it's really something A is doing to mess up the debugger)

[hintron]

The reproducer implicitly loads a long list of 50 DLLs on startup, some of them with fairly large PDBs > 1 GB. This takes 3-4 seconds before raddbg's Modules panel is done adding new rows. Then raddbg tries to step, and fails to do so correctly.

The chain of 50 DLLs + PDBs seems to be the required condition to reproduce. Here's a redacted view of what DLLs are loaded on start:

Reproducer
│
├── AAA  [INTERFACE]
│   └── BBB  [INTERFACE]
│
├── CCC  [INTERFACE]
│   └── DDD  [INTERFACE]
│
├── EEE  (static — absorbed into FFF.pdb)
│   ├── EEE_INTERFACE  [INTERFACE — headers only]
│   ├── TBB::tbb  [DLL — Intel TBB, own PDB]
│   ├── AAA  [INTERFACE]
│   ├── 116  (static)
│   ├── GGG_INTERFACE  [INTERFACE]
│   ├── GGG  (static)
│   ├── HHH_INTERFACE  [INTERFACE]
│   ├── III_INTERFACE  [INTERFACE]
│   ├── JJJ_INTERFACE  [INTERFACE]
│   ├── KKK  (static)
│   └── fmt::fmt  (static / header-only)
│       + Boost::iostreams / coroutine / timer / chrono  (static)
│
└── FFF  [DLL ← THE BIG ONE, 1.2 GB PDB]
    │   (absorbs ALL of the following static libs into its PDB)
    ├── CCC  [INTERFACE]
    ├── EEE  (static, see above)
    ├── Utilities  (static)
    │   ├── AAA  [INTERFACE]
    │   ├── EEE  (static, circular — LINK_INTERFACE_MULTIPLICITY 3)
    │   ├── 114  (static)
    │   ├── 115  (static)
    │   └── KKK  (static)
    ├── LLL  (static)  + LLL_INTERFACE [INTERFACE]
    ├── MMM  (static)
    ├── NNN  (static)
    ├── OOO  (static)
    ├── PPP  (static)
    ├── QQQ  (static)
    ├── RRR  (static)
    ├── SSS  (static)
    ├── TTT  (static)
    ├── UUU  (static)
    ├── VVV  (static)
    ├── WWW  (static)
    ├── XXX  (static)
    ├── YYY  (static)
    │   └── ZZZ  (static)
    ├── 100  (static)
    ├── 101  (static)
    ├── 102  (static)
    │   └── → 103, 104, ...
    ├── 105  (static)
    │   └── → 106_INTERFACE [INTERFACE], ...
    ├── 107  (static)
    ├── 108  (static)
    ├── 109  (static)  ← large
    ├── 110  (static)  ← very large
    ├── 111  (static)
    ├── 106_INTERFACE  [INTERFACE]
    ├── 112_INTERFACE  [INTERFACE]
    ├── 113_INTERFACE  [INTERFACE]
    └── ${Boost_LIBRARIES}  (static — all Boost libs)

Perhaps it's some race condition during loading in an individual thread or perhaps the 50 DLLs somehow exceed some kind of PDB loader thread pool and causes issues. Just throwing out ideas.

I'm not sure how to hit this issue with a simple reproducer. I've tried over a dozen variations of a simple program that I could share with you that loads either a large DLL or a chain of large DLLs, with a few large PDBs, but those work just fine with raddbg.

Anyways, I'm sorry I could not provide a reproducer. Perhaps there is some large, open source C++ codebase out there that loads a lot of DLLs on startup that could be used as a reproducer.

[hintron]

I just deleted all of my breakpoints (50-100 of them) in different parts of the DLLs, some of them with conditions, and now I can't reproduce the issue anymore! It now also only takes about 1.5 seconds to step after launch, and is 100% reliable.