Conversation
|
Thank you @Stanpol ! I'll be reviewing this PR. |
belveryin
left a comment
There was a problem hiding this comment.
@Stanpol thank you so much for your contribution. I have two comments to the PR:
-
Record WAL range before post-commit sealing in fast mode —
src/transaction/mod.rs:679
In fast mode, WAL range publication is deferred to async finalize, but sealing now runs immediately after commit. That allows a segment to be sealed before its WAL range is attached, which breaks WAL floor/GC bookkeeping assumptions. WAL range should be attached before any operation that can seal. -
Keep seal policy configuration build-time; avoid exposing runtime setter yet —
src/db/mod.rs:336,src/db/builder.rs:950DbBuilder::with_seal_policy(...)already provides the needed configuration path at database creation time.
The new publicDB::set_seal_policy(...)has surprising semantics: it only works with an exclusive handle (Arc::get_mut) and otherwise becomes a no-op (false). SinceDBis cloneable, this is easy to misuse and leaks internal ownership behavior into the public API.
Recommendation: for this PR, maybe keep seal policy as builder configuration and remove (or make internal/test-only) the public runtime setter. If runtime reconfiguration is needed later, introduce a shared-safe API with explicitResulterrors.
I found an issue where transactional writes remained in memory indefinitely, failing to trigger memtable sealing or SSTable flushing. This is particularly impactful for S3-backed storage where data visibility and persistence rely on regular SSTable creation. I added the ability to configure sealing policies via DbBuilder and improve WASM compatibility.
Key Changes
src/transaction/mod.rs)maybe_seal_after_insert()andmaybe_run_minor_compaction()immediately after a transaction commit.db.ingest()calls triggered maintenance. Transactional writes stayed in the mutable memtable regardless of sealing policies, leading to data loss on web page refresh/process exit in serverless environments.src/db/builder.rsandsrc/db/mod.rs)DbBuilder::with_seal_policy()to allow configuring memtable behavior during database initialization.DB::set_seal_policy()public.BatchesThreshold { batches: 1 }) for example in wasm web applications.src/db/mod.rsandsrc/ondisk/sstable.rs)std::time::Instantwithexecutor::Instantin the L0StatsCache.std::time::Instantis often unreliable or panics in WASM environments, using the executor's clock ensures correct time-based sealing logic in the browser.Happy to answer questions, fix something or discuss the issues.