Skip to content

Commit 0bef49f

Browse files
bkaradzicbkaradzic
committed
Remove --renderdoc-dll preload code; rely on renderdoccmd/rdc capture for DLL injection.
1 parent 74718d8 commit 0bef49f

7 files changed

Lines changed: 74 additions & 630 deletions

File tree

‎.github/instructions/babylon-native-debugging.instructions.md‎

Lines changed: 20 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -130,16 +130,11 @@ per flag** (intentional design -- long-name aliases were removed).
130130
still runs on the test's original renderCount, so
131131
pass/fail is unaffected. Output: <cwd>/temp/
132132
bgfx_frame<N>.rdc. Combine with --once / --test /
133-
--test-index for a single capture.
134-
--renderdoc-dll=PATH Pin which renderdoc.dll bgfx adopts. PATH may be
135-
a file or a directory (\renderdoc.dll appended).
136-
Validated at parse time: missing path / wrong
137-
kind exits 2. Eagerly preloaded before bgfx::init
138-
so bgfx's findModule("renderdoc.dll") adopts it.
139-
Always prints a diagnostic line:
140-
RenderDoc: <full path> (API X.Y.Z, FileVersion ...)
141-
Use this instead of $env:PATH manipulation -- it
142-
survives PATH ordering and works in headless CI.
133+
--test-index for a single capture. Requires
134+
renderdoc.dll to be loaded into the process; easiest
135+
is to launch via `renderdoccmd capture -w` or
136+
`rdc capture --trigger -w`, both of which inject
137+
the paired DLL before main.
143138
-t, --test=PATTERN Run tests whose title contains PATTERN (substring,
144139
case-insensitive). Repeatable; multiple = OR.
145140
--test-index=LIST Run only the listed indices.
@@ -226,30 +221,27 @@ returns `true`, so the legacy throw-on-missing behavior is preserved.
226221

227222
### RenderDoc capture (one-liner)
228223
```powershell
229-
# Preferred: pin the DLL explicitly. Survives PATH ordering, works in CI.
230-
.\Playground.exe --headless --once --test-index=286 --include-excluded `
231-
--capture=5 --renderdoc-dll="C:\path\to\renderdoc-py" `
224+
# Launch under renderdoccmd: it injects the paired renderdoc.dll into the
225+
# Playground process before main, so bgfx::findModule adopts it. Version
226+
# always matches what `rdc open` accepts -- no PATH-ordering bugs.
227+
& "<renderdoc-py-dir>\renderdoccmd.exe" capture -w .\Playground.exe `
228+
--headless --once --test-index=286 --include-excluded --capture=5 `
232229
app:///Scripts/validation_native.js
233230
# .rdc lives at <cwd>\temp\bgfx_frame5.rdc
234231
```
235-
Alternatives that also work (in resolution order -- first match wins):
236-
1. `--renderdoc-dll=PATH` (CLI flag, parse-time validated; preferred).
237-
2. `BN_RENDERDOC_DLL=<file-or-dir>` env var (only loaded when `--capture` set).
238-
3. `RENDERDOC_PYTHON_PATH=<rdc-cli renderdoc-py dir>` (rdc-cli's own var, only when `--capture` set).
239-
4. `renderdoc.dll` discoverable via `LoadLibrary` (PATH search; fragile).
240-
241-
Confirm the diagnostic line in stdout -- it tells you exactly which DLL was loaded:
232+
Equivalent with the rdc-cli wrapper (inject-only, no auto-capture handshake):
233+
```powershell
234+
& rdc capture --trigger --wait-for-exit -- .\Playground.exe `
235+
--headless --once --test-index=286 --include-excluded --capture=5 `
236+
app:///Scripts/validation_native.js
242237
```
243-
RenderDoc: C:\Users\...\renderdoc-py\renderdoc.dll (API 1.6.0, FileVersion 1.41.0.0)
238+
Verify `renderdoc.dll` is loaded:
239+
```powershell
240+
(Get-Process Playground).Modules | Where-Object ModuleName -eq 'renderdoc.dll'
244241
```
245-
If you see `RenderDoc: WARNING: --capture requested but renderdoc.dll could not
246-
be loaded.` -- none of the four sources resolved. Pin with `--renderdoc-dll=...`.
247-
If you see `RenderDoc: WARNING: loaded DLL differs from RENDERDOC_PYTHON_PATH
248-
pair.` -- the rdc-cli replay version won't match the captured version; the
249-
warning text includes the exact `--renderdoc-dll=...` to add. Pixel pass/fail
250-
is unchanged by `--capture`. See
242+
Pixel pass/fail is unchanged by `--capture`. See
251243
`playground/playground-renderdoc-capture.instructions.md` for the full recipe
252-
(DLL version match, multiple captures, `rdc` CLI inspection).
244+
(launcher selection, multiple captures, `rdc` CLI inspection).
253245

254246
### "App crashed but I only see one short line"
255247
Scroll **up** from the error line. The full `--- CRASH ---` /

‎.github/instructions/playground/playground-renderdoc-capture.instructions.md‎

Lines changed: 46 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,8 @@ Read this when the user wants to RenderDoc-debug a specific Playground test
99
(by name or index), or asks "why is this pixel-diff test failing?" after
1010
visual inspection of the result/diff PNGs hasn't been enough. **Don't add
1111
new RenderDoc integration to BN code** -- bgfx's built-in `debug_renderdoc.cpp`
12-
is already wired through `TestUtils.captureNextFrame()`.
12+
is already wired through `TestUtils.captureNextFrame()`, and the launcher
13+
(`renderdoccmd` / `rdc capture`) handles DLL injection.
1314

1415
For generic `rdc-cli` usage and inspection workflows, see
1516
`../renderdoc/renderdoc-gpu-debug.instructions.md` and
@@ -34,121 +35,71 @@ disqualifying patterns in stdout -- none of them are pixel-diff bugs and a
3435
`Pixel difference: N pixels.` in stdout.** That's the signature of a real
3536
GPU-side mismatch worth a RenderDoc capture session.
3637

37-
## Why `rdc capture` (the launcher) doesn't work for Playground
38+
## How DLL loading works
3839

39-
The `rdc capture` mode launches the target executable under a RenderDoc
40-
injector and waits for swapchain Present calls. It fails for Playground in
41-
two common modes:
40+
Playground itself does NOT preload `renderdoc.dll`. bgfx calls
41+
`findModule("renderdoc.dll")` at `bgfx::init`, adopts whatever is already in
42+
the process, and exposes the trigger via `bgfx::renderDocTriggerCapture` /
43+
`TestUtils.captureNextFrame()`. To get the DLL into the process before
44+
`bgfx::init`, launch under `renderdoccmd` or `rdc capture --trigger`. Both
45+
inject the `renderdoc.dll` paired with their own executable into the target
46+
via `CreateRemoteThread + LoadLibrary` before `wWinMain` runs. Because the
47+
DLL comes from the launcher's directory, the version always matches what
48+
`rdc open` will accept -- no PATH-ordering bugs, no version mismatch class.
4249

43-
| Setup | Why it fails |
44-
|---|---|
45-
| `--headless` | No swapchain -> no Present calls -> RenderDoc has no frame boundary to capture. |
46-
| Windowed `--once` | App exits before the RenderDoc child-injection handshake completes. Error: "failed to connect to target". |
47-
48-
Use bgfx's **in-process** capture path instead -- it doesn't need an injector
49-
and works in headless mode too because the `TriggerCapture` API records the
50-
in-flight D3D command stream regardless of whether a real Present happens.
51-
52-
## Recipe (works for both windowed and headless)
50+
Verify (while the process is alive):
5351

54-
### Step 1 -- Pick which RenderDoc DLL to load
55-
56-
`rdc-cli` ships with `renderdoc-py` whose `renderdoc.pyd` exposes a specific
57-
RenderDoc replay version. The capture must be made by a `renderdoc.dll` of
58-
the **same** RenderDoc version, otherwise `rdc open` rejects it with:
59-
60-
```
61-
OpenCapture failed: Captured API data was made on a newer incompatible
62-
version of RenderDoc: D3D11 capture is incompatible version 20, newest
63-
supported by this build of RenderDoc is 19
52+
```powershell
53+
(Get-Process Playground).Modules |
54+
Where-Object ModuleName -eq 'renderdoc.dll' |
55+
Select-Object FileName
56+
# -> C:\Users\<you>\Private\util\renderdoc-py\renderdoc.dll
6457
```
6558

66-
The DLL paired with `rdc-cli` lives next to `renderdoc.pyd` in
67-
`renderdoc-py`'s install dir. Verify with `rdc doctor` -- both
68-
`renderdoc-module` and `win-renderdoc-install` rows should show the same
69-
version (e.g. 1.41).
59+
## Recipe
7060

71-
Playground accepts the DLL location from any of these (first match wins,
72-
top-down):
61+
### Step 1 -- Launch with `--capture=N` under the launcher
7362

74-
| Source | Notes |
75-
|---|---|
76-
| `--renderdoc-dll=<file-or-dir>` | **Preferred.** Parse-time validated (exit 2 if path missing or directory has no `renderdoc.dll`). Works without `--capture` (lets you confirm what would load). Survives PATH ordering. |
77-
| `BN_RENDERDOC_DLL` env var | File or directory. Only consulted when `--capture` is set (so a stale env var doesn't silently load RenderDoc into every Playground run). |
78-
| `RENDERDOC_PYTHON_PATH` env var | Same var rdc-cli already reads -- set it once, both rdc-cli and Playground pick the matching DLL. Only consulted when `--capture` is set. `\renderdoc.dll` is appended automatically. |
79-
| PATH `LoadLibrary("renderdoc.dll")` | Fallback. Whichever directory PATH resolves first wins. Fragile -- easy to load the wrong version. |
63+
```powershell
64+
$exe = "<build>\Apps\Playground\Debug\Playground.exe"
65+
$rdcmd = "<renderdoc-py-dir>\renderdoccmd.exe"
66+
Set-Location (Split-Path $exe)
8067
81-
Always confirm in stdout:
82-
```
83-
RenderDoc: C:\Users\...\renderdoc-py\renderdoc.dll (API 1.6.0, FileVersion 1.41.0.0)
68+
& $rdcmd capture -w $exe `
69+
--once --include-excluded --test-index=<N> --capture=5 `
70+
app:///Scripts/validation_native.js
8471
```
85-
That line appears whenever any of the above resolved. Possible warnings:
8672

87-
| Line prefix | Meaning |
88-
|---|---|
89-
| `RenderDoc: WARNING: --capture requested but renderdoc.dll could not be loaded.` | None of the four sources found a DLL. Pin one with `--renderdoc-dll=...`. |
90-
| `RenderDoc: WARNING: loaded DLL differs from RENDERDOC_PYTHON_PATH pair.` | A DLL was loaded, but it doesn't match what rdc-cli expects -> `rdc open` will likely reject the capture. The warning prints the exact `--renderdoc-dll=...` argument to add for a fix. |
91-
| `RenderDoc: ERROR: --renderdoc-dll=X could not be honored.` | Something (e.g. an injector) had already loaded a different `renderdoc.dll` before `wWinMain` ran. Use the path printed alongside, or relaunch outside the injector. |
73+
Or with the rdc-cli wrapper (same effect, plus `--trigger` skips the
74+
launcher's own auto-capture handshake):
9275

93-
**Legacy PATH-based recipe** (still works):
9476
```powershell
95-
$env:Path = "<dir-with-matching-renderdoc.dll>;$env:Path"
77+
& rdc capture --trigger --wait-for-exit -- $exe `
78+
--once --include-excluded --test-index=<N> --capture=5 `
79+
app:///Scripts/validation_native.js
9680
```
97-
Use only when you can't use `--renderdoc-dll`. Subject to PATH-ordering bugs.
98-
99-
### Step 2 -- Launch with `--capture=N`
100-
101-
The `--capture=N` CLI flag does all the wiring for you:
10281

103-
- Triggers `TestUtils.captureNextFrame()` on the Nth rendered frame of
104-
every executed test.
105-
- Auto-extends each test's render budget by 5 frames after the trigger so
106-
RenderDoc has time to finalize the `.rdc`. (Single-frame default tests
107-
with `renderCount: 1` would otherwise exit before finalize -- that was
108-
the historical reason for hand-editing config.json.)
109-
- Pixel comparison still happens at the test's original `renderCount`, so
110-
pass/fail is unchanged. You can compare the run output with-and-without
111-
`--capture` and the "First pixel off / Pixel difference" lines should be
112-
identical.
82+
What the flags do:
11383

114-
```powershell
115-
$exe = "<build>\Apps\Playground\Debug\Playground.exe"
116-
$wd = "<build>\Apps\Playground\Debug"
117-
Set-Location $wd
118-
119-
& $exe --headless --once --include-excluded --test-index=<N> `
120-
--capture=5 --renderdoc-dll="<dir-with-matching-renderdoc.dll>" `
121-
app:///Scripts/validation_native.js
122-
```
84+
| Flag | Purpose |
85+
|---|---|
86+
| `renderdoccmd capture -w` | Inject paired `renderdoc.dll` before `main`; wait for target exit. |
87+
| `rdc capture --trigger -w` | Same, but explicitly inject-only (no launcher-side auto-capture). |
88+
| `--capture=N` | Trigger `TestUtils.captureNextFrame()` on the Nth rendered frame of every executed test. Auto-extends the test's render budget by 5 frames after the trigger so RenderDoc has time to finalize. Pixel comparison still happens at the test's original `renderCount`, so pass/fail is unchanged. |
89+
| `--once`, `--include-excluded`, `--test-index=N` | Standard test selection. |
12390

12491
`--capture=5` is a good default: trigger mid-run, plenty of finalize headroom.
125-
Use a different N if you specifically need a different frame.
126-
127-
Watch stdout for the diagnostic line:
128-
```
129-
RenderDoc: <full-path-to-loaded-renderdoc.dll> (API X.Y.Z, FileVersion ...)
130-
BGFX Loading RenderDoc... # only if bgfx hadn't seen it; with --renderdoc-dll
131-
# it usually prints "RenderDoc is already loaded."
132-
BGFX [ ] Capture
133-
```
134-
The `RenderDoc:` line confirms the DLL load and version. The `BGFX [ ] Capture`
135-
line confirms the trigger fired. If you see neither, check the DLL location --
136-
the resolution order is documented in Step 1.
13792

138-
The capture file lands at:
139-
```
140-
<cwd>/temp/bgfx_frame<N>.rdc
141-
```
142-
(`temp/bgfx` is the default `BGFX_CONFIG_RENDERDOC_LOG_FILEPATH`, relative
143-
to the working directory. The `<N>` here is bgfx's internal frame counter,
144-
NOT the value you passed to `--capture` -- both can be present.)
93+
The capture file lands at `<cwd>/temp/bgfx_frame<N>.rdc` (where `<N>` is
94+
bgfx's internal frame counter, NOT the value passed to `--capture` -- both
95+
can be present).
14596

14697
> **Do not** edit `config.json`'s `"capture"` / `"renderCount"` for capture
14798
> work -- `--capture=N` supersedes that legacy path. The config-driven path
14899
> still works (kept for backwards compat) but it triggers at the test's
149100
> last frame and may not finalize.
150101
151-
### Step 3 -- Open + inspect
102+
### Step 2 -- Open + inspect
152103

153104
```powershell
154105
$env:RENDERDOC_PYTHON_PATH = "<renderdoc-py dir>"
@@ -163,7 +114,7 @@ rdc.exe buffer <id> -o file.bin # raw buffer bytes
163114
rdc.exe mesh <eid> --stage vs-out --json # post-VS positions
164115
```
165116

166-
### Step 4 -- Clean up
117+
### Step 3 -- Clean up
167118

168119
`--capture` and the rendered extra frames don't persist -- nothing to revert.
169120
Just delete `<cwd>/temp/` if you want to free disk.
@@ -217,10 +168,9 @@ Symptom: pixel `(13,13,15)` instead of expected `(255,0,0)`. Both cubes
217168
render black; reference shows red+green from per-instance color buffer.
218169

219170
```powershell
220-
.\Playground.exe --headless --once --test-index=286 --include-excluded `
221-
--capture=5 --renderdoc-dll="C:\Users\bkaradzic\Private\util\renderdoc-py" `
171+
& $rdcmd capture -w .\Playground.exe `
172+
--once --test-index=286 --include-excluded --capture=5 `
222173
app:///Scripts/validation_native.js
223-
# -> RenderDoc: C:\...\renderdoc-py\renderdoc.dll (API 1.6.0, FileVersion 1.41.0.0)
224174
# -> "First pixel off at 286108: Value: (13, 13, 15) - Expected: (255, 0, 0)"
225175
# -> "Pixel difference: 53296 pixels."
226176
# -> exit -1

‎Apps/Playground/Shared/CommandLine.cpp‎

Lines changed: 5 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@
55
#include <cstdio>
66
#include <cstdlib>
77
#include <cstring>
8-
#include <filesystem>
98
#include <sstream>
109
#include <string>
1110
#include <string_view>
@@ -109,52 +108,6 @@ namespace
109108
return true;
110109
}
111110

112-
// Resolves a --renderdoc-dll value to the absolute path of an existing
113-
// renderdoc.dll. Accepts a file path (used as-is) or a directory path
114-
// containing renderdoc.dll. Returns false with err set on failure.
115-
bool ResolveRenderDocDllPath(std::string_view raw, std::string& outAbsPath, std::string& err)
116-
{
117-
if (raw.empty())
118-
{
119-
err = "empty path";
120-
return false;
121-
}
122-
123-
std::error_code ec;
124-
std::filesystem::path p{std::string{raw}};
125-
126-
if (!std::filesystem::exists(p, ec) || ec)
127-
{
128-
err = "path does not exist: '" + std::string{raw} + "'";
129-
return false;
130-
}
131-
132-
if (std::filesystem::is_directory(p, ec))
133-
{
134-
std::filesystem::path dll = p / "renderdoc.dll";
135-
if (!std::filesystem::exists(dll, ec))
136-
{
137-
err = "directory has no renderdoc.dll: '" + std::string{raw} + "'";
138-
return false;
139-
}
140-
p = dll;
141-
}
142-
else if (!std::filesystem::is_regular_file(p, ec))
143-
{
144-
err = "path is not a regular file: '" + std::string{raw} + "'";
145-
return false;
146-
}
147-
148-
std::filesystem::path absPath = std::filesystem::absolute(p, ec);
149-
if (ec)
150-
{
151-
err = "could not make path absolute: '" + std::string{raw} + "'";
152-
return false;
153-
}
154-
outAbsPath = absPath.string();
155-
return true;
156-
}
157-
158111
// Returns true if arg matches "--longName" / "--longName=value" or
159112
// shortName. For ValueRequired flags missing an "=", consumes the next
160113
// arg as the value.
@@ -376,21 +329,6 @@ namespace CommandLine
376329
}
377330
if (!err.empty()) { opt.ParseError = true; opt.ErrorMessage = err; return opt; }
378331

379-
if (auto m = match("--renderdoc-dll", "", FlagKind::ValueRequired); m.matched)
380-
{
381-
std::string resolved;
382-
std::string rerr;
383-
if (!ResolveRenderDocDllPath(m.value, resolved, rerr))
384-
{
385-
opt.ParseError = true;
386-
opt.ErrorMessage = "invalid --renderdoc-dll: " + rerr;
387-
return opt;
388-
}
389-
opt.RenderDocDll = resolved;
390-
continue;
391-
}
392-
if (!err.empty()) { opt.ParseError = true; opt.ErrorMessage = err; return opt; }
393-
394332
if (auto m = match("--test", "-t", FlagKind::ValueRequired); m.matched)
395333
{
396334
opt.TestFilters.push_back(m.value);
@@ -460,13 +398,11 @@ namespace CommandLine
460398
" <cwd>/temp/bgfx_frame<N>.rdc.\n"
461399
" Combine with --once / --test / --test-index\n"
462400
" to limit to a single capture.\n"
463-
" --renderdoc-dll=PATH Pin which renderdoc.dll bgfx loads. PATH may\n"
464-
" be the DLL itself or a directory containing\n"
465-
" renderdoc.dll. Avoids PATH-ordering surprises\n"
466-
" and version mismatches with rdc-cli. If not\n"
467-
" given, BN_RENDERDOC_DLL and\n"
468-
" RENDERDOC_PYTHON_PATH env vars are consulted\n"
469-
" when --capture is active.\n"
401+
" Requires renderdoc.dll to be loaded into\n"
402+
" the process. Easiest: launch via\n"
403+
" `renderdoccmd capture -w Playground.exe ...`\n"
404+
" or `rdc capture --trigger -w -- Playground.exe ...`\n"
405+
" (those inject the paired DLL before main).\n"
470406
" -t, --test=PATTERN Run tests whose title contains PATTERN.\n"
471407
" May be specified multiple times (OR).\n"
472408
" --test-index=LIST Run only the listed indices. LIST may be a\n"

‎Apps/Playground/Shared/CommandLine.h‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,14 +23,11 @@ struct PlaygroundOptions
2323

2424
// 1-based frame index at which to call TestUtils.captureNextFrame()
2525
// (RenderDoc capture trigger). When set, the runner extends each test's
26-
// render budget so the .rdc finalizes.
26+
// render budget so the .rdc finalizes. Requires renderdoc.dll to be
27+
// injected externally (e.g. via `renderdoccmd capture` or
28+
// `rdc capture --trigger`); bgfx auto-adopts an already-loaded DLL.
2729
std::optional<int> CaptureFrame;
2830

29-
// Absolute path to a renderdoc.dll to preload before bgfx::init.
30-
// Validated at parse time (must point to an existing file, or to a
31-
// directory containing renderdoc.dll which is then resolved to that file).
32-
std::optional<std::string> RenderDocDll;
33-
3431
std::vector<std::string> TestFilters;
3532
std::vector<int> TestIndices;
3633

0 commit comments

Comments
 (0)