Skip to content

perf(cache): hash NumPy arrays without JSON expansion#4883

Open
dexhunter wants to merge 1 commit into
ManimCommunity:mainfrom
dexhunter:perf/cache-ndarray-hashing
Open

perf(cache): hash NumPy arrays without JSON expansion#4883
dexhunter wants to merge 1 commit into
ManimCommunity:mainfrom
dexhunter:perf/cache-ndarray-hashing

Conversation

@dexhunter

@dexhunter dexhunter commented Jul 11, 2026

Copy link
Copy Markdown

Overview: What does this pull request change?

This replaces large plain-ndarray JSON representations with deterministic SHA-256 digests over their exact logical bytes. It also includes pixel_array when hashing animations and mobjects, while continuing to exclude camera framebuffers.

Fixes #2924. Related to #2710.

Motivation and Explanation: Why and how do your changes improve the library?

The previous encoder resized arrays larger than 1,000 elements to 100 x 100 and serialized their representation. That had two consequences:

  • changes outside the sampled values could reuse a stale scene cache entry, including visible ImageMobject pixel changes;
  • building large array representations dominated scene-cache hashing CPU time.

The new array digest includes shape, dtype, field names, and every logical value. It normalizes byte order and memory layout, hashes structured fields separately so record padding is ignored, and preserves the existing fallback for ndarray subclasses and object-dtype arrays.

On the deterministic scene-hash fixture used for autoresearch (120-136 mobjects plus a 1,920 x 1,072 RGBA image), eight paired randomized samples produced these medians:

Revision Hashing CPU time
Current main 811.60 ms
This change 49.06 ms

The paired-ratio metric was 60,543 ppm, a 93.95% reduction in local hashing CPU time. The evaluator rejects candidates unless cache signatures remain stable across fresh processes and equivalent layouts, visible pixel mutations invalidate the hash, camera framebuffer aliases remain excluded, structured dtype names, titles, offsets, item sizes, alignment, overlapping fields, and padding semantics are preserved, and distinct top-level mobjects with colliding Python hashes remain distinct.

A separate seven-sample benchmark using real Manim objects makes the workload trade-off explicit:

Workload Current main This change Difference
120 VMobjects with 333 x 3 point arrays 381.93 ms 7.77 ms -98.0%
Eight 512 x 512 RGBA ImageMobjects 1.49 ms 6.63 ms +5.14 ms

The image workload is intentionally slower because current main omits those visible pixels from the cache key. This change spends about 5 ms across eight 1 MiB images to make their cache invalidation correct; it does not claim that every scene hashes faster.

Autoresearch trajectory: https://dashboard.weco.ai/share/Ezfm3RFgvWPOdMcNVY9JBWwBxS-Tu18V

Validation

  • uv run python -m pytest -q tests/module/utils/test_hashing.py (16 passed)
  • uv run ruff check manim/utils/hashing.py tests/module/utils/test_hashing.py
  • uv run ruff format --check manim/utils/hashing.py tests/module/utils/test_hashing.py
  • git diff --check

Links to added or changed documentation pages

No documentation changes are required; this preserves the public hashing API.

Further Information and Comments

The final scene cache key remains the existing CRC32-based format. SHA-256 is used only to compact each array before it enters that existing serialization and cache-key pipeline.

Reviewer Checklist

  • The PR title is descriptive enough for the changelog, and the PR is labeled correctly
  • If applicable: newly added non-private functions and classes have a docstring including a short summary and a PARAMETERS section
  • If applicable: newly added functions and classes are tested

Replace lossy array reprs with deterministic logical-byte digests and include ImageMobject pixels in cache invalidation.

Refs ManimCommunity#2710

Fixes ManimCommunity#2924

Co-Authored-By: Aiden <aiden@weco.ai>
@dexhunter
dexhunter force-pushed the perf/cache-ndarray-hashing branch from 8abd050 to df4ae34 Compare July 14, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ImageMobjects are not properly hashed

1 participant