Skip to content

Commit 9ca491b

Browse files
committed
Add OOM auto-dump test and pre-release validation report
The phpt suite covered rdump_dump() and the return value of rdump_set_oom_dump(), but nothing actually triggered the memory_limit path, so the headline OOM auto-dump feature was untested. Add tests/oom_dump.phpt: it spawns a child that exhausts memory_limit with rdump.oom_dump set and asserts the extension auto-wrote a well-formed RDUMP file plus its .done marker. Also add docs/validation.md, recording an end-to-end shakedown of ext-rdump + reli against real-world memory issues sourced from GitHub (less.php, HTMLPurifier, PhpSpreadsheet, Doctrine ORM, and the OOM auto-dump reproducing PhpSpreadsheet#1094's 128MB fatal). https://claude.ai/code/session_0173cJdvKHwRMbKuMLMRxuqD
1 parent cdf2561 commit 9ca491b

2 files changed

Lines changed: 223 additions & 0 deletions

File tree

docs/validation.md

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# Pre-release validation: ext-rdump + reli on real-world memory issues
2+
3+
This is a record of an end-to-end shakedown of `ext-rdump` against `reli`
4+
before release. The goal was not a synthetic benchmark but to answer a
5+
practical question: **if you take a real PHP program that people actually
6+
report running out of memory on, can `ext-rdump` capture a dump cleanly, and
7+
does `reli`'s report tell you something useful about why?**
8+
9+
Real cases were sourced by searching GitHub issues for `"allowed memory size
10+
of"` and `"php memory usage"` (934 issues match the first query alone), then
11+
reproducing representative ones locally.
12+
13+
**Verdict: yes, end to end.** Every workload captured a valid RDUMP, `reli`
14+
analysed all of them, and in each case the top findings pointed straight at the
15+
real cause — the parser AST, the accumulated array, the `Cell` objects, the
16+
Doctrine UnitOfWork. The headline feature — the automatic dump on
17+
`memory_limit` exhaustion — fired correctly and produced a report that
18+
reconstructs the exact call stack at the moment of death.
19+
20+
## Environment
21+
22+
| | |
23+
|---|---|
24+
| PHP | 8.4.19 (NTS), x86-64 Linux |
25+
| ext-rdump | this branch, `phpize && ./configure --enable-rdump && make` |
26+
| reli | `reliforp/reli-prof` 0.13.x-dev |
27+
| Pipeline | `rdump_dump()` / OOM auto-dump → `inspector:memory:analyze -f rmem``inspector:memory:report` |
28+
29+
`ext-rdump`'s own test suite (now 7 tests, see [below](#test-added)) is green.
30+
31+
## Results at a glance
32+
33+
| Case | Real issue | Workload | PHP heap | Dump | Graph | reli's top finding |
34+
|---|---|---|---|---:|---:|---|
35+
| 1. less.php | [wikimedia/less.php#104](https://github.com/wikimedia/less.php/issues/104) | compile 1.4 MB LESS | 212 MB | 328 MB | 6.5M nodes | `$parser->rules` holds 165 MB via 4,004 children; AST node classes named |
36+
| 2. HTMLPurifier | [ezyang/htmlpurifier#270](https://github.com/ezyang/htmlpurifier/issues/270) | purify 6,000 docs | 9.8 MB | 22 MB | 60K nodes | `$clean` array 7.4 MB via 6,000 children; `ZendString` 69.7% of heap |
37+
| 3. PhpSpreadsheet | [PHPOffice/PhpSpreadsheet#1094](https://github.com/PHPOffice/PhpSpreadsheet/issues/1094) | read a 2 MB / 360k-cell xlsx | 214 MB | 743 MB | 7.6M nodes | `Cell` × 360,000 (58% of objects); worksheet↔cell cycle 188 MB |
38+
| 4. Doctrine ORM | canonical batch OOM (e.g. [sentry-laravel#178](https://github.com/getsentry/sentry-laravel/issues/178)) | 40k-order batch, no `clear()` | 198 MB | 225 MB | 4.3M nodes | `$em->unitOfWork->{entityIdentifiers,originalEntityData,identityMap}`; Order↔OrderItem cycle 188 MB |
39+
| 5. **OOM auto-dump** | reproduces #1094's *exact* 128 MB fatal | read xlsx under `memory_limit=128M` | 125 MB (97.7% of limit) | 971 MB | 4.3M nodes | `near_memory_limit` HIGH + **exact call stack at the fatal** |
40+
41+
In every case the analysed-heap coverage was 94–98.5%.
42+
43+
## Per-case detail
44+
45+
### 1. less.php — cumulative parser growth (#104)
46+
47+
Compiling a 1.4 MB generated stylesheet left the parser holding **212 MB**
48+
(peak 321 MB) — the cumulative-growth shape the issue describes. `reli` reported:
49+
50+
- `bottleneck_path`: `$parser->rules[...]->rules->referenced` (190 MB)
51+
- `choke_point` / `large_array`: `$parser->rules` — 165 MB via **4,004**
52+
children (exactly the 4,000 components compiled), plus
53+
`$parser->cachedEvaldRules` (21 MB)
54+
- `companion_cluster` / `expensive_property`: the AST node classes by name —
55+
`Less_Tree_Ruleset` ×28,000, `Less_Tree_Declaration::$value`, etc.
56+
57+
That is a precise map of where less.php's memory goes, down to the named
58+
properties on its tree nodes.
59+
60+
### 2. HTMLPurifier — definition cache (#270)
61+
62+
`#270` is a **PHP 7.4-specific `unserialize` engine regression**, so it does
63+
**not** reproduce on 8.4 — an honest reproducibility caveat. Memory stayed flat
64+
at ~10 MB. The case still validated `reli` on a real library's live heap: it
65+
correctly isolated the accumulated `$clean` array (7.4 MB via 6,000 children),
66+
flagged `ZendString` as 69.7% of the heap, and resolved deep HTMLPurifier
67+
internals by PHP-syntax path
68+
(`$config->definitions['HTML']->info`,
69+
`$purifier->strategy->strategies[1]->zipper->referenced->front->...`).
70+
71+
### 3. PhpSpreadsheet — large-file read OOM (#1094)
72+
73+
A **2 MB** xlsx (360k cells) expanded to **214 MB** of heap on load — the
74+
classic "small file, huge memory" report. `reli`:
75+
76+
- named the cause: `PhpOffice\PhpSpreadsheet\Cell\Cell`**360,000 instances**,
77+
58% of object memory
78+
- traced the bottleneck path to `…->cellCollection` (188 MB) and detected the
79+
worksheet↔cell reference **cycle** (15 classes, 188 MB retained)
80+
- surfaced a genuine inefficiency: every `Cell` carries an `IgnoredErrors`
81+
object — 360,000 of them for **41 MB**, almost all empty
82+
(`structural_duplicate` flags 41 MB of identical shapes)
83+
84+
### 4. Doctrine ORM — the framework batch OOM
85+
86+
The single most common real-world "allowed memory size of" in Symfony/Laravel
87+
batch commands: a long loop that `persist()`s and `flush()`es but never
88+
`clear()`s, so the `EntityManager` retains every entity. 40,000 orders /
89+
**160,000 managed entities** → 198 MB. `reli` pointed straight at the
90+
`UnitOfWork`:
91+
92+
- `$em->unitOfWork->entityIdentifiers`, `originalEntityData` (67 MB **each**
93+
the change-tracking snapshot, the non-obvious cost of skipping `clear()`),
94+
`identityMap['App\Order']` (40,000), `identityMap['App\OrderItem']` (120,000),
95+
`entityStates` (160,000)
96+
- the bidirectional `Order``OrderItem` reference **cycle** (188 MB retained)
97+
98+
Any Doctrine developer reading this report would immediately recognise the
99+
missing `$em->clear()`.
100+
101+
### 5. OOM auto-dump — the headline feature
102+
103+
This is the capability that matters most in production: capture the moment of
104+
`memory_limit` death **with no code change**, driven only by INI. Re-running the
105+
PhpSpreadsheet read under `memory_limit=128M` reproduced #1094's exact fatal
106+
("Allowed memory size of 134217728 bytes exhausted") and the extension wrote
107+
`oom-<pid>.rdump` automatically. From that dump `reli` produced a full
108+
post-mortem:
109+
110+
- `near_memory_limit` **HIGH** — "Peak usage is 97.7% of memory_limit — only
111+
2.91 MB headroom" (125 MB / 128 MB), confirming the snapshot is at the
112+
exhaustion point
113+
- the **exact call stack at the fatal**:
114+
`preg_match → Coordinate::coordinateFromString:37 →
115+
Xlsx::loadSpreadsheetFromFile:913 → BaseReader::load:290 → <main>:18`,
116+
matching the engine's "Coordinate.php on line 37"
117+
- the cause at that instant: 167,134 `Cell` objects (it died roughly half-way
118+
through the 360k), the cell cache holding 73.85 MB, plus the in-flight reader
119+
arrays
120+
121+
Auto-dump was validated on three independent paths:
122+
123+
- `rdump.oom_dump` via INI (with `%p` PID expansion) — fires
124+
- `rdump_set_oom_dump()` at runtime, no INI — fires
125+
- `rdump.oom_dump_marker=1` — the `<path>.rdump.done` marker is written
126+
- `rdump.oom_dump_max_total` — with the budget set below the directory's
127+
existing `*.rdump` footprint, the next OOM dump is **skipped** (the process
128+
still dies, but no file is written), as designed
129+
130+
## Notable real-world insight: OOM dumps can be ~1 GB even at a 128 MB limit
131+
132+
The case-5 dump was **971 MB** despite the 128 MB PHP limit, because the dump
133+
includes the glibc `[heap]` and anonymous mmaps, and PhpSpreadsheet's
134+
libxml2/libzip allocate the XLSX parse tree in **C memory that `memory_limit`
135+
never counts**. `reli` shows it plainly: `RSS: 985 MB` vs `Heap: 119 MB`. This
136+
is exactly the confusing real-world situation where `memory_get_usage()` looks
137+
fine but the process is huge — and `ext-rdump` captures it faithfully.
138+
139+
The practical consequence: on a worker that OOMs repeatedly, an unguarded
140+
auto-dump could fill the disk fast. The existing guards
141+
(`oom_dump_max`, `oom_dump_min_interval`, `oom_dump_max_total`) are the right
142+
mitigation, and `oom_dump_max_total` is verified above. Worth keeping this
143+
prominent in the README (it already is).
144+
145+
## Rough edges / recommendations
146+
147+
1. **Analysis is heavy on large dumps.** A 200 MB heap becomes a 4–8M-node
148+
graph; `inspector:memory:analyze` took ~1.5 min and ~2 GB RSS on the 1 GB
149+
OOM dump, more on less.php. This is reli's side, not ext-rdump's, but it is
150+
the practical ceiling for "capture in prod, analyse later." For first-pass
151+
triage on big dumps, reach for `--no-full-analysis` and `--memory-limit`.
152+
2. **Capture worked on every shape tried** — AST-heavy (less.php), object-heavy
153+
(PhpSpreadsheet, Doctrine), string-heavy (HTMLPurifier), and mid-syscall at
154+
OOM — with no crashes and 94–98.5% analysed coverage. No correctness issues
155+
found in `ext-rdump` itself.
156+
3. **Test gap closed.** Before this run, the phpt suite exercised
157+
`rdump_dump()` and the *return value* of `rdump_set_oom_dump()`, but nothing
158+
actually triggered the `memory_limit` path — the headline feature was
159+
untested. Added `tests/oom_dump.phpt` (below).
160+
161+
## Test added
162+
163+
`tests/oom_dump.phpt` spawns a child that exhausts a 16 MB `memory_limit` with
164+
`rdump.oom_dump` set, then asserts from the parent that the extension
165+
auto-wrote a well-formed RDUMP file plus its `.done` marker — i.e. that the OOM
166+
hook fires end to end, not just that the setter returns `true`.

tests/oom_dump.phpt

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
--TEST--
2+
rdump.oom_dump writes a well-formed RDUMP dump when memory_limit is exhausted
3+
--SKIPIF--
4+
<?php
5+
if (!extension_loaded("rdump")) die("skip rdump not loaded");
6+
if (PHP_OS_FAMILY !== "Linux") die("skip Linux only");
7+
if (!is_file(dirname(__DIR__) . "/modules/rdump.so")) die("skip built modules/rdump.so not found");
8+
if (!function_exists("shell_exec") || !shell_exec("echo ok")) die("skip shell_exec disabled");
9+
?>
10+
--FILE--
11+
<?php
12+
// A memory_limit error is fatal, so it can only be observed from a separate
13+
// process: spawn a child that exhausts memory with rdump.oom_dump set, then
14+
// verify from here that the extension auto-dumped at the moment of death.
15+
$so = dirname(__DIR__) . "/modules/rdump.so";
16+
$php = getenv("TEST_PHP_EXECUTABLE");
17+
if ($php === false || $php === "") {
18+
$php = PHP_BINARY;
19+
}
20+
21+
$dump = sys_get_temp_dir() . "/rdump_oom_" . getmypid() . ".rdump";
22+
$child = sys_get_temp_dir() . "/rdump_oom_child_" . getmypid() . ".php";
23+
@unlink($dump);
24+
@unlink($dump . ".done");
25+
file_put_contents($child, '<?php $a = []; while (true) { $a[] = str_repeat("x", 4096); }');
26+
27+
$cmd = escapeshellarg($php)
28+
. " -n"
29+
. " -d extension=" . escapeshellarg($so)
30+
. " -d memory_limit=16M"
31+
. " -d rdump.oom_dump=" . escapeshellarg($dump)
32+
. " -d rdump.oom_dump_marker=1"
33+
. " " . escapeshellarg($child)
34+
. " 2>&1";
35+
$output = (string) shell_exec($cmd);
36+
37+
// The child died on the memory_limit error...
38+
var_dump(strpos($output, "Allowed memory size") !== false);
39+
40+
// ...and the OOM hook wrote the dump without any rdump_dump() call.
41+
var_dump(is_file($dump));
42+
43+
// The dump is a well-formed RDUMP file.
44+
var_dump(file_get_contents($dump, false, null, 0, 8) === "RDUMP\0\0\0");
45+
46+
// The completion marker was written (rdump.oom_dump_marker=1).
47+
var_dump(is_file($dump . ".done"));
48+
49+
@unlink($dump);
50+
@unlink($dump . ".done");
51+
@unlink($child);
52+
?>
53+
--EXPECT--
54+
bool(true)
55+
bool(true)
56+
bool(true)
57+
bool(true)

0 commit comments

Comments
 (0)