Skip to content

perf: contiguous fixed-width buffer for per-atom string columns - #35

Merged
jameskermode merged 3 commits into
masterfrom
perf/string-buffer
Jun 9, 2026
Merged

jameskermode merged 3 commits into
masterfrom
perf/string-buffer

Conversation

@jameskermode

Copy link
Copy Markdown
Member

What

A per-atom string column was stored as char ** — an array of N pointers, each a separate malloc+strcpy on read, then decoded one-by-one in a Python loop in c_to_py_dict. This replaces it with one contiguous fixed-width buffer (numpy S{width} layout):

  • C read: a single calloc per column (growing the cell width on demand), one memcpy per field — no per-atom malloc.
  • Python: the whole column becomes a numpy array via one np.frombuffer(...).astype(str) instead of a per-atom decode loop.

Measured ~18% faster on a 200k-atom read (clean A/B on the same machine: read_dicts 84.9 → 69.3 ms; 64k: 27.2 → 22.1 ms). The per-atom malloc count and the Python decode loop dominated at scale. Combined with the earlier copy-removal (#32), the benchmark table moves from ~2.2×→2.8× (plugin) and ~2.5×→3.2× (parser) over ASE's built-in; README table + plot refreshed.

How — width carried in n_in_row, no struct change

DictEntry.n_in_row was unused for finalized per-atom columns, so it carries the cell width — negated (-width) as the "contiguous" marker. (A positive width would be ambiguous: parse_tree leaves a non-negative n_in_row item-count on finalized scalar info entries, which would make free_data skip the inner-string free — a leak I hit and fixed during development.) Every string consumer dispatches on the sign:

  • extxyz_read_ll fill (contiguous, grow-on-demand)
  • free_data (one free if n_in_row < 0, else the legacy per-cell loop)
  • extxyz_write_ll per-atom output
  • c_to_py_dict (np.frombuffer vs decode loop)
  • fextxyz.f90 read path

Info-line strings and the Python write path keep the legacy char ** (n_in_row >= 0) and are untouched. No DictEntry/ctypes struct change.

Safety / verification

  • Leaks: contiguous = one alloc / one free, so error paths are simpler and leak-safe. leaks --atExit = 0 on valid, malformed-Properties, truncated, multi-column-string, long-string, and read→write inputs.
  • Tests: new tests/test_string_buffer.py (multi-column S:2 matrix path; long strings forcing a width grow; round-trip) on both backends. Core suite (40) + ase-extxyz suite (38) green.
  • Fortran: the fextxyz.f90 read path is updated and is exercised by CI's USE_FORTRAN=T round-trip tests (CI builds QUIP).

Note on the benchmark refresh

The committed table's ASE built-in baseline reproduces on this machine (~226 vs 233 ms at 200k), so the cextxyz/read_dicts gains are real, not a hardware artifact. (The dev meson build defaults to -O0; these numbers are from a -Dbuildtype=release build, matching how wheels are built.)

🤖 Generated with Claude Code

jameskermode and others added 3 commits June 9, 2026 20:37
A string column was an array of N malloc'd pointers (one malloc+strcpy per
cell on read, and a per-cell Python decode loop in c_to_py_dict). Replace it
with a single contiguous fixed-width buffer (numpy 'S{width}' layout): the C
read does one allocation (growing the cell width on demand) and Python builds
the array with one np.frombuffer(...).astype(str) instead of a 20k-iteration
decode loop.

The cell width is carried in the previously-unused DictEntry.n_in_row, NEGATED
(-width) to mark the column contiguous — parse_tree leaves a non-negative
n_in_row on finalized scalar info entries (an item count), so a positive width
would be ambiguous and would make free_data skip the inner string free (a leak).
Every string consumer dispatches on the sign: free_data (one free vs per-cell),
extxyz_write_ll per-atom output, c_to_py_dict, and the Fortran read path in
fextxyz.f90. Info-line strings and the Python write path keep the legacy char**
(n_in_row >= 0), so they are untouched. No DictEntry/ctypes struct change.

Result: the species string column drops from ~2.2 ms to ~1.1 ms of a 20k-atom
read; contiguous = one alloc/one free, so error paths are leak-safe too. New
tests cover multi-column and width-growing string columns on both backends;
core + ase-extxyz suites and `leaks` are clean. The Fortran read path is
exercised by CI's USE_FORTRAN=T round-trip tests (CI builds QUIP).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regenerated benchmarks/results.csv and read_speedup.png with an optimized
build on the current read path (redundant-copy removal + contiguous string
columns). The ASE built-in baseline is unchanged (~226 ms at 200k atoms,
matching the previous table), while cextxyz/read_dicts are faster: the plugin
now runs ~2.8x over the built-in end-to-end (was ~2.2x) and ~3.2x for the
parser alone (was ~2.5x). README table and prose updated to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The contiguous-string-column change made extxyz_write_ll and free_data
dispatch on DictEntry.n_in_row (<0 = contiguous fixed-width buffer). The
Fortran write-build raw-malloc's its DictEntry nodes and never set n_in_row,
so it held garbage; a random negative value made the C writer misread a
per-atom species char** column as a contiguous buffer and emit corrupt bytes,
non-deterministically failing the USE_FORTRAN=T round-trip tests
(UnicodeDecodeError on re-read). Zero n_in_row per node so Fortran-built dicts
are unambiguously the legacy char** representation, matching the Python
binding's zeroed ctypes buffers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jameskermode
jameskermode merged commit 8de6220 into master Jun 9, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant