Skip to content

Fix parser segfault on malformed Properties + plug memory leaks - #30

Merged
jameskermode merged 3 commits into
masterfrom
fix/parser-crashes-and-leaks
Jun 9, 2026
Merged

jameskermode merged 3 commits into
masterfrom
fix/parser-crashes-and-leaks

Conversation

@jameskermode

Copy link
Copy Markdown
Member

Summary

The third-party Rust port extxyz/extxyz (crate extxyz-ng) markets itself against this library, citing a memory leak (with a valgrind trace) and segfaults on "misaligned input" — but filed no issues. I independently verified each claim. Three are real and fixed here; two are overstated/false and now have regression tests.

Fixes

1. Segfault on malformed Properties (NULL deref)

An info line whose Properties value doesn't match name:type:count (e.g. Properties=species) fails the grammar's strict rule, falls through to the generic key=value rule, and is stored as a plain string. The strtok-based Properties parser then dereferenced a NULL token (strtok(NULL, ":") → strlen(NULL)) → SIGSEGV. Now every strtok result is NULL-checked and reported as a clean ExtXYZError.

2. Grammar memory leak

The compiled cleri grammar was never freed — 240 leaks / 13.5 KB under macOS leaks, matching the valgrind trace in the Rust port's README. Now cleri_grammar_free is exposed to the Python binding and the cached grammar is freed via atexit; the C test driver frees it (and the dicts) too.

3. Error-path dict leaks

Mid-parse failures freed line/re_str/props but leaked the partially built *info/*arrays (17 leaks / 720 B on a truncated file). Added free_partial_dicts() on every error path; string columns are now calloc'd so freeing a partially filled array is safe.

4. C test driver — fixes #20

test_C_main.c passed &comment (a single char, non-NULL) and a single-char error_message, so the reader treated the entire 2nd line as one comment key and risked a buffer overflow on errors. Now it passes comment = NULL and a real error_message buffer, and tears everything down. Verified against the example in #20 — Lattice, Properties, pbc, etc. now parse as separate keys.

Not bugs (regression-tested)

  • "Array rows with space padding cause segfault" — listed only as an unverified TODO in their README. Padded/tabbed data rows parse fine; locked in.
  • Leading whitespace on the info line — already handled correctly on both backends.

Tests

  • tests/test_robustness.py — reproduces each crash/leak in a subprocess (so a SIGSEGV can't take pytest down) and covers the two non-bugs above. 18 cases, both backends.
  • tests/test_bench_vs_rust.py — skip-by-default head-to-head benchmark vs extxyz-ng (runs it in a separate venv via subprocess, since both packages import as extxyz).

Verification

  • meson compile — all targets, 0 warnings.
  • macOS leaks — 0 leaks on valid, truncated, malformed-Properties, and multi-property files (was 240 + 17 leaks).
  • pytest tests/ — 20 passed, 2 skipped (bench).

Benchmark (measured, current code with PCRE2 JIT)

  • Single 20k-atom frame: ours 7.4 ms vs extxyz-ng 4.5 ms → 1.6× faster for them (not the README's "~4×", which was vs the pre-JIT legacy C).
  • 200×500-atom trajectory: extxyz-ng fails to read it — its multi-frame reader desyncs after a few non-trivial frames. Ours reads all 200 frames in ~48 ms.

Closes #20.

🤖 Generated with Claude Code

jameskermode and others added 3 commits June 9, 2026 18:06
…leaks

Independently confirmed three real defects (and debunked two overstated ones)
raised by the third-party Rust port `extxyz/extxyz` (`extxyz-ng`), whose README
criticises this C library but filed no issues.

Fixes:

- Segfault on malformed `Properties`: an info line whose `Properties` value does
  not match `name:type:count` (e.g. `Properties=species`) slips past the grammar
  as a plain string, after which the strtok-based parser dereferenced a NULL
  token (`strlen(NULL)`). Guard every `strtok(NULL, ":")` and error cleanly.

- Grammar memory leak: the compiled cleri grammar was never freed (240 leaks /
  13.5 KB under macOS `leaks`; matches the valgrind trace in the Rust port's
  README). Expose `cleri_grammar_free` to the Python binding and free the cached
  grammar via `atexit`; free it (and the dicts) in the C test driver.

- Error-path dict leaks: mid-parse failures freed `line`/`re_str`/`props` but
  leaked the partially built `*info`/`*arrays`. Add `free_partial_dicts()` on
  every error path and `calloc` the string column so partial frees are safe.

- C test driver (`test_C_main.c`): pass `comment = NULL` and a proper
  `error_message` buffer (were single `char`s), so it parses the info line into
  separate keys instead of reading the whole 2nd line as one `comment`, and add
  full teardown. Fixes #20.

Tests: `tests/test_robustness.py` reproduces each crash/leak in a subprocess and
locks in that space-padded data rows and leading-whitespace info lines (two
non-bugs the Rust port alleged) parse correctly on both backends.
`tests/test_bench_vs_rust.py` is a skip-by-default head-to-head benchmark.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ence gcc warning

The subprocess-based robustness tests forced the repo `python/` dir onto
sys.path, but that source tree lacks the build-generated `extxyz._version`, so
they failed under CI (where extxyz is installed as a wheel) with
`ModuleNotFoundError: No module named 'extxyz._version'`. Let the child inherit
the parent's environment instead, so it imports the same extxyz the suite does
(installed in CI, PYTHONPATH source tree locally).

Also initialise `cur_array` at its declaration to silence a GCC
-Wmaybe-uninitialized false positive introduced by the new NULL guards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Windows extension only exports symbols listed in _extxyz.def, so
`cleri_grammar_free` (a libcleri symbol) was missing there, breaking import of
cextxyz and thus all test collection. Add it to the .def, and make the Python
binding degrade gracefully (`_have_grammar_free`) if a build ever lacks it.

Also build the test's CRASH_SIGNALS set from only the signals that exist on the
platform -- `signal.SIGBUS` is absent on Windows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jameskermode
jameskermode merged commit fbaa859 into master Jun 9, 2026
42 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C example reads the whole 2nd line as a comment

1 participant