Paper: Caching for fast notebook restarts - #1262
Conversation
|
Curvenote Preview
|
272c417 to
163867d
Compare
dmadisetti
changed the title
dmadisetti
changed the title
163867d to
b7da545
Compare
|
CC @scipy-conference/2026-proceedings (@rowanc1) for a draft tag please :) |
|
I'm a reviewer on the paper. |
|
Hello, all. I'm Chris Holloman (@chrisholloman), from the Scipy Proceedings committee. I will serve as the editor on this paper. Please let me know if I can help in anyway. Thanks @ycwkatie for agreeing to review this paper. |
|
Hello @chrisholloman - thank you! I have minor edits (CI fix and phrasing suggestions from a colleague) and an update- the implementation of this paper is now off experimental and in marimo 0.24.14 (https://github.com/marimo-team/marimo/releases/tag/0.23.14#cached-wasm-exports) I have these changes staged- just wondering if there is a submission freeze and whether I should wait for feedback to fold this in (I wouldn't want someone to review an old edit) (edit: maybe this question is better suited for @ycwkatie ?) |
|
Hi, I am a reviewer for this paper too. |
Hi @dmadisetti! There is no submission freeze. You should go ahead and fold this in so that reviewers are looking at the most recent edit. |
| Cache keys are derived recursively from the reactive DAG, content-addressing reference values and substituting parent-cell hashes where direct content addressing is not possible. | ||
| The recurrence forms a Merkle structure that an edit invalidates at the subtree granularity. | ||
| Cached values cross process and session boundaries through a lazy stub mechanism and participate in marimo's static WASM/HTML export, so heavy computations and trained models can ship with a notebook to readers with a browser-embedded Python runtime. | ||
| Empirically the cache lookup is as fast or faster than representative baselines on microbenchmarks of variably sized payloads. |
There was a problem hiding this comment.
As fast or faster" is only true on M4 Mac. On Linux x86-64 the advantage compresses to ~1.2× as mentioned in the paper in §5
Would recommend this
| Empirically the cache lookup is as fast or faster than representative baselines on microbenchmarks of variably sized payloads. | |
| Empirically the cache lookup is as fast or faster than representative baselines on microbenchmarks of variably sized payloads, though the magnitude of the advantage is hardware-dependent. |
| The recurrence forms a Merkle structure that an edit invalidates at the subtree granularity. | ||
| Cached values cross process and session boundaries through a lazy stub mechanism and participate in marimo's static WASM/HTML export, so heavy computations and trained models can ship with a notebook to readers with a browser-embedded Python runtime. | ||
| Empirically the cache lookup is as fast or faster than representative baselines on microbenchmarks of variably sized payloads. | ||
| However, unlike other mechanisms, marimo's caching is native to the reactive notebook, adds little user-facing overhead in its utilization, and allows for cross platform reuse. |
There was a problem hiding this comment.
The abstract claims that marimo's caching "allows for cross platform reuse." But as I understand it, the persistent cache relies on Python bytecode and platform-specific codecs, making it generally unsafe to share cache directories across different operating systems or Python versions. (I noticed you implicitly acknowledge this by using _HOST_FINGERPRINT in the benchmarks).
If "cross platform reuse" refers exclusively to the static WASM/HTML export, it would be great to make that explicit so users don't erroneously assume the local disk cache is portable across environments.
My suggestion: Rephrase the abstract to clarify: "and allows for cross-platform reuse via static WASM exports."
| For computational caching, false positive hits are unacceptable and false negatives merely undesirable. | ||
| For a useful cache, marimo's caching requires a stable key derivation that is sensitive to value changes and robust to superficial notebook edits. | ||
| At runtime a marimo notebook exposes a reactive DAG over its cells together with the ref values bound in memory, so a naive approach may be to construct the cache key by hashing every `ref`s' value directly. | ||
| However, this does not survive Python's exposure of mutation and FFI. |
There was a problem hiding this comment.
minor nitpick : Would recommend to expand the FFI acronym on first use. While the SciPy audience is highly technical, standard academic style suggests expanding it to "Foreign Function Interface (FFI)" on first use.
| Storage and loading are therefore decoupled from lookup. | ||
| The default `PickleLoader` in marimo serializes the full `Cache` envelope as one pickle blob, re-materializing every `def` eagerly on lookup. | ||
|
|
||
| The newer, opt-in mechanism, the `LazyLoader` writes a JSON manifest of per-def references alongside individual blob files. |
There was a problem hiding this comment.
Non-blocking / Food for thought
The LazyLoader and JSON manifest design looks great. Out of curiosity, have we thought about the scaling limits of parsing the manifest on cache misses?
If we ever move this to remote storage (or if the manifest gets massive), we might want to consider putting a Bloom filter in front of the lookup. It could instantly tell us if a cache key doesn't exist, saving us from expensive disk/network checks on cache misses.
Definitely not needed for this PR, especially if the manifest easily fits in memory right now, but might be worth keeping in mind for future optimization if we start seeing latency on misses.
| mandala derives its key via `joblib.hash`, which serializes through pickle before hashing [@makelov2024mandala]; marimo's `data_to_buffer` views the ndarray's contiguous bytes through Python's buffer protocol and hashes them directly. | ||
| On the Apple M4 Max used for the camera-ready figures, hashing is fast and the pickle pass costs mandala roughly 3× end to end; on a Linux x86-64 server, memory bandwidth makes the pickle pass nearly free, value load dominates instead, and the penalty compresses to roughly 1.2×. | ||
| marimo's value load tracks the `diskcache (fixed key)` floor, isolating the structural gap in key derivation rather than storage. | ||
| Panel (c) exposes per-call variance. |
There was a problem hiding this comment.
The main text here just states that "Panel (c) exposes per-call variance," but it misses the critical detail shown in the figure itself: diskcache.memoize failing due to the SQLite blob ceiling. Consider expanding the main text to explicitly mention this limitation, as it's a major takeaway from the plot.
|
|
||
| Panel (b) decomposes the largest-payload hit into key derivation and value load, alongside the overhead a miss adds (key derivation and value save). | ||
| mandala derives its key via `joblib.hash`, which serializes through pickle before hashing [@makelov2024mandala]; marimo's `data_to_buffer` views the ndarray's contiguous bytes through Python's buffer protocol and hashes them directly. | ||
| On the Apple M4 Max used for the camera-ready figures, hashing is fast and the pickle pass costs mandala roughly 3× end to end; on a Linux x86-64 server, memory bandwidth makes the pickle pass nearly free, value load dominates instead, and the penalty compresses to roughly 1.2×. |
There was a problem hiding this comment.
The explanation of mandala 's hardware variance is thorough, but we need to bring the narrative back to marimo here. After explaining that the penalty compresses to 1.2x on Linux, consider adding a concluding sentence summarizing exactly how marimo 's architecture bypasses this entirely to achieve its final speedup.
| # Bench-key components folded into every cached measurement: a host | ||
| # fingerprint (system + arch + Python version) and a methodology | ||
| # version string. Bumping `_BENCH_VERSION` invalidates every persisted | ||
| # sweep on every host, which is the right move whenever `_e2e_setup`, | ||
| # `compute.measure_hit`, `compute.measure_miss`, codec choices, or | ||
| # payload generation change. Without this, `@mo.persistent_cache` | ||
| # would key only on `sizes_mb` and silently reuse cross-host or | ||
| # pre-edit measurements. | ||
| _HOST_FINGERPRINT = ( | ||
| f"{platform.system()}|{platform.machine()}|" | ||
| f"{platform.python_implementation()}|{platform.python_version()}" | ||
| ) | ||
| _BENCH_VERSION = "2026-06-miss-overhead-and-cold-timing" |
There was a problem hiding this comment.
Looking at the benchmark script, I noticed that _HOST_FINGERPRINT and _BENCH_VERSION are used to ensure cache measurements are strictly isolated and aren't silently reused across different environments or after methodology changes. This is a strong methodological detail that seems currently omitted from the paper. Adding a brief mention of this in the evaluation setup would significantly reinforce the experimental rigor and reproducibility of the results.
| // Cache key dispatch — flat three-way fallback, top-down flow. | ||
| // Rendered to figs/fig1_dispatch_flow.png via `make flowchart`, then | ||
| // composed with the rasterized derivation listing (figs/alg_hash.typ) | ||
| // into figs/fig1_dispatch.png by figs/compose_fig1.py. | ||
| // | ||
| // A per-ref dispatch tries three categories in order: | ||
| // 1. Pure — immutable primitives, hash the value directly. | ||
| // 2. ContentAddressed — buffer-protocol bytes, hash the buffer. | ||
| // 3. ExecutionPath — fall back to the producing cell's stored H. | ||
| // All three feed the same sha256 combiner at the bottom, which | ||
| // concatenates the per-ref keys with the cell's compiled bytecode to | ||
| // produce H(cell). | ||
|
|
There was a problem hiding this comment.
I noticed a structural mismatch between the caching implementation provided in the code snippet and the conceptual diagram in this figure.
The figure and its supporting text describe the cache key dispatch as a "flat three-way fallback," explicitly listing three categories: Pure , ContentAddressed, and ExecutionPath.
However, when reading the description in the text, I saw that it actually introduces ContextExecutionPath as a distinct fourth category.
Because ContextExecutionPath is missing from the diagram entirely, I found it confusing to map the text to figure.
Suggestion:
I suggest updating figure to explicitly include the ContextExecutionPath branch so it accurately reflects the provided implementation.
|
I noticed the curvenote check GitHub Action is currently failing on this PR. Please check the CI logs and address the failing checks when you have a moment. |
|
Thank you for the submission! The caching mechanism described is a valuable contribution, and the review above assesses its architectural comparison against baselines. The comments below focus on minor clarifications to strengthen the narrative. Specifically, they aim to align terminology with the diagrams, clarify the benchmark methodology (like host fingerprinting), and contextualize the final performance claims. These are provided as line-level suggestions for consideration in the next revision. |
53d7939 to
f25914e
Compare
for more information, see https://pre-commit.ci
6 participants
First-draft submission for SciPy Proceedings 2026
This paper is fully exportable to static pages, per #1207 we could investigate a full static dump (about 700 hundred files of generated assets). As is, I have a jupyterbook mirror at https://dmadisetti.github.io/scipy_proceedings