Remove blocking IO from mvcc bootstrap and recovery paths#7349
Open
PThorpe92 wants to merge 5 commits into
Open
Remove blocking IO from mvcc bootstrap and recovery paths#7349PThorpe92 wants to merge 5 commits into
PThorpe92 wants to merge 5 commits into
Conversation
Collaborator
Author
|
Will kick off an Antithesis run from this branch once the CI is green.
|
Merging this PR will not alter performance
Comparing Footnotes
|
PThorpe92
commented
Jun 4, 2026
| Ok(Some((value, bytes, len))) => (value, bytes, len), | ||
| Ok(None) => return Ok(PayloadParseResult::Eof), | ||
| Err(err) => return Err(err), | ||
| match return_if_io!(self.consume_varint_bytes()) { |
Collaborator
Author
There was a problem hiding this comment.
note to self: looks to be that this function is not safely re-entrant
it was re-entrant it was just stupid because it re-parsed the whole frame if any frame yielded mid-way, so a real state machine was added
SJYX
added a commit
to SJYX/Momo-Study-Agent
that referenced
this pull request
Jun 6, 2026
When wait > 300s, now checks: - Active TCP connections at connect() return → server-side long-poll - No connections + network_sent ≈ 0 → local blocking (checkpoint/reconcile/schema) Previous report incorrectly said 'long-poll' when data shows no TCP connections and zero bytes sent — the 318s is pyturso internal blocking IO (cf. tursodatabase/turso#7349), not waiting for server.
…chine to avoid duplicating work
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lift blocking IO part 2: schema reparse + MVCC bootstrap/recovery
Removes synchronous
io.block/wait_for_completionfrom the MVCC open path so it no longer hangs on backends without a synchronous IO pump (e.g. WASM, whereio.step()is a no-op). The whole bootstrap → checkpoint-reconcile → schemareparse → metadata-table init → logical-log replay sequence now yields IO cooperatively via
IOResultinstead of blocking.Major changes
Non-blocking statement runner
Statement::run_ignore_rows_nonblock/run_with_row_callback_nonblock:drive a statement to completion returning
IOResult, surfacing the pendingcompletion (via
take_io_completions) instead of pumpingio.step(). TheVDBE interpreter already yields
StepResult::IO; these just bridge it to theIOResultmodel. No interpreter changes.Schema reparse
reparse_schema/reparse_schema_with_cookielifted to aReparseSchemaStatemachine (read cookie → scan
sqlite_schema→ load custom types → refreshstats), carrying the half-built schema + captured table-valued functions +
reparse guard across yields.
parse_schema_rows,refresh_analyze_stats, andread_current_schema_cookielifted to
IOResultaccordingly.MVCC bootstrap
bootstrap_nonblockreworked into a linear state machine that drives, allnon-blocking: interrupted-checkpoint reconciliation, schema reparse, the
persistent_tx_ts_maxread, metadata-table create/seed, and the metadata IOchain. Dead blocking shims removed.
Logical-log recovery
maybe_recover_logical_loglifted to aRecoverLogicalLogStatemachine. Setupphases (header / tx-ts / cookie /
sqlite_schemascan) yield IO; the replayloop carries its accumulators in
RecoverCtxacross per-framenext_frameyields. The per-frame replay body is unchanged in behavior. Driven from a new
bootstrap
Recoverphase.Checkpoint hang fix (
mvcc/database/checkpoint_state_machine.rspath)maybe_complete_interrupted_checkpoint'sDriveEarlyTruncatewas recreatingits
CheckpointResulton every re-entry, sotruncate_wal(which trackstruncate/sync progress through that struct) never completed and spun under
io.block. The result now persists in the state variant.