Repository navigation
perf: drop redundant per-frame array copies; streaming/perf analysis - #32
Merged
Merged
Conversation
`read_frame_dicts` already frees the C buffers before returning, so the numpy arrays it hands back are independently owned. `_read_frame_dict` was then copying every column into a structured array and copying it straight back out again (`data[name] = value` then `.copy()`) — two redundant copies per column. Use the returned arrays directly for the C-extension path; the pure-Python path still needs the structured array from np.fromregex/genfromtxt and is unchanged. Measured ~3% faster on a 20k-atom single-frame read (7.21 -> 6.99 ms min) and a simpler code path. Profiling shows Python post-processing is only ~5% of the total — the rest is the C parser (pcre2_match per line) — so this is the only worthwhile Python-side win; see PR description for the full analysis. Core suite (both backends) and the ase-extxyz plugin suite pass unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
out of curiosity, what do your quoted results imply about the 4x faster claim? |
Member
Author
|
I recently added JIT compilation of the regex which make us closer. It was more like 1.6x faster, but with the further optimisations I'm merging now (single string buffer, faster float parser for easy cases) we'll be quite a big faster than the extxyz-ng and ca. 4x faster than the built in ASE extcyz. I will stop there, I think. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background — investigating the extxyz-ng performance comparison
The Rust port (
extxyz-ng) markets ~4× speed and "streaming for large trajectories". I profiled our read path to see what we can actually learn. Two findings reframe the question:1. We already match it on streaming / memory.
iread_dictsopens the file once and yields oneFrameat a time; the Cextxyz_read_llallocates exactly one frame and frees it right after conversion. Reading an N-frame trajectory holds one frame in memory at a time — constant memory. (extxyz-ng's multi-frame reader, by contrast, desyncs after a few non-trivial frames and can't read a real trajectory at all — see PR #30's benchmark.)2. The single-frame gap is the C parser, not Python. On a 20k-atom frame:
read_dictsoverallread_frame_dicts(parse + convert)Controlled experiments isolate the costs further:
pos:R:3only (floats, no strings): 4.74 ms — already ≈ extxyz-ng's total (4.5 ms). The dominant cost ispcre2_matchper line.species:S:1: 6.93 ms — the string column adds ~2.2 ms (per-cellmalloc+strcpyin C and per-celldecodein Python), vs only ~0.37 ms for an equivalent int column.This PR
Removes the only worthwhile Python-side win the profile revealed:
_read_frame_dictcopied each column into a structured array and back out again (two redundant copies). Sinceread_frame_dictsfrees the C buffers before returning, the arrays it hands back are already owned — so we use them directly. ~3% faster (7.21 → 6.99 ms) and a simpler code path. Pure-Python path unchanged.Verified: core suite (both backends) + ase-extxyz plugin suite green.
What we are not doing (and why)
The remaining levers don't pass the profile's cost/benefit bar pre-release:
atof/STR_INCRmicro-opts — negligible for typical short numeric fields.np.frombufferinstead of decoding 20k cells in a loop). That touches the C data model,free_dict, the writer, the buffer-aliasing path, andpy_to_c_dict— high blast radius for ~1.5–2 ms.pcre2_matchwould mean a hand-rolled per-line tokenizer — a substantial rewrite.Given we already win on correctness and memory and match streaming, these larger rewrites aren't justified for this release; documenting them here as future options.
🤖 Generated with Claude Code