Repository navigation
perf: contiguous fixed-width buffer for per-atom string columns - #35
Merged
Merged
Conversation
A string column was an array of N malloc'd pointers (one malloc+strcpy per
cell on read, and a per-cell Python decode loop in c_to_py_dict). Replace it
with a single contiguous fixed-width buffer (numpy 'S{width}' layout): the C
read does one allocation (growing the cell width on demand) and Python builds
the array with one np.frombuffer(...).astype(str) instead of a 20k-iteration
decode loop.
The cell width is carried in the previously-unused DictEntry.n_in_row, NEGATED
(-width) to mark the column contiguous — parse_tree leaves a non-negative
n_in_row on finalized scalar info entries (an item count), so a positive width
would be ambiguous and would make free_data skip the inner string free (a leak).
Every string consumer dispatches on the sign: free_data (one free vs per-cell),
extxyz_write_ll per-atom output, c_to_py_dict, and the Fortran read path in
fextxyz.f90. Info-line strings and the Python write path keep the legacy char**
(n_in_row >= 0), so they are untouched. No DictEntry/ctypes struct change.
Result: the species string column drops from ~2.2 ms to ~1.1 ms of a 20k-atom
read; contiguous = one alloc/one free, so error paths are leak-safe too. New
tests cover multi-column and width-growing string columns on both backends;
core + ase-extxyz suites and `leaks` are clean. The Fortran read path is
exercised by CI's USE_FORTRAN=T round-trip tests (CI builds QUIP).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regenerated benchmarks/results.csv and read_speedup.png with an optimized build on the current read path (redundant-copy removal + contiguous string columns). The ASE built-in baseline is unchanged (~226 ms at 200k atoms, matching the previous table), while cextxyz/read_dicts are faster: the plugin now runs ~2.8x over the built-in end-to-end (was ~2.2x) and ~3.2x for the parser alone (was ~2.5x). README table and prose updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The contiguous-string-column change made extxyz_write_ll and free_data dispatch on DictEntry.n_in_row (<0 = contiguous fixed-width buffer). The Fortran write-build raw-malloc's its DictEntry nodes and never set n_in_row, so it held garbage; a random negative value made the C writer misread a per-atom species char** column as a contiguous buffer and emit corrupt bytes, non-deterministically failing the USE_FORTRAN=T round-trip tests (UnicodeDecodeError on re-read). Zero n_in_row per node so Fortran-built dicts are unambiguously the legacy char** representation, matching the Python binding's zeroed ctypes buffers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A per-atom string column was stored as
char **— an array of N pointers, each a separatemalloc+strcpyon read, then decoded one-by-one in a Python loop inc_to_py_dict. This replaces it with one contiguous fixed-width buffer (numpyS{width}layout):callocper column (growing the cell width on demand), onememcpyper field — no per-atommalloc.np.frombuffer(...).astype(str)instead of a per-atom decode loop.Measured ~18% faster on a 200k-atom read (clean A/B on the same machine: read_dicts 84.9 → 69.3 ms; 64k: 27.2 → 22.1 ms). The per-atom
malloccount and the Python decode loop dominated at scale. Combined with the earlier copy-removal (#32), the benchmark table moves from ~2.2×→2.8× (plugin) and ~2.5×→3.2× (parser) over ASE's built-in; README table + plot refreshed.How — width carried in
n_in_row, no struct changeDictEntry.n_in_rowwas unused for finalized per-atom columns, so it carries the cell width — negated (-width) as the "contiguous" marker. (A positive width would be ambiguous:parse_treeleaves a non-negativen_in_rowitem-count on finalized scalar info entries, which would makefree_dataskip the inner-string free — a leak I hit and fixed during development.) Every string consumer dispatches on the sign:extxyz_read_llfill (contiguous, grow-on-demand)free_data(onefreeifn_in_row < 0, else the legacy per-cell loop)extxyz_write_llper-atom outputc_to_py_dict(np.frombuffervs decode loop)fextxyz.f90read pathInfo-line strings and the Python write path keep the legacy
char **(n_in_row >= 0) and are untouched. NoDictEntry/ctypes struct change.Safety / verification
leaks --atExit= 0 on valid, malformed-Properties, truncated, multi-column-string, long-string, and read→write inputs.tests/test_string_buffer.py(multi-columnS:2matrix path; long strings forcing a width grow; round-trip) on both backends. Core suite (40) + ase-extxyz suite (38) green.fextxyz.f90read path is updated and is exercised by CI'sUSE_FORTRAN=Tround-trip tests (CI builds QUIP).Note on the benchmark refresh
The committed table's ASE built-in baseline reproduces on this machine (~226 vs 233 ms at 200k), so the cextxyz/read_dicts gains are real, not a hardware artifact. (The dev
mesonbuild defaults to-O0; these numbers are from a-Dbuildtype=releasebuild, matching how wheels are built.)🤖 Generated with Claude Code