Experiment: atomic buf page fields - #6130
Draft
polchawa-percona wants to merge 9 commits into
Draft
Conversation
Concurrent workloads that repeatedly hit the same few pages (point lookups/updates on hot rows) contend on buf_page_get_mutex() even though most of the work done under it while checking/refreshing a page is just reading or writing these two scalar fields. buf_fix_count and io_fix were already made atomic upstream; state and access_time were the two BPageMutex-documented fields still plain. Several call sites already read state/access_time without holding the block mutex (buf_LRU_free_from_common_LRU_list, buf_page_peek_if_too_old, the read-ahead heuristics in buf0rea.cc, I_S buffer_page population), which was an existing unsynchronized data race on these fields; making them std::atomic (via the existing copyable_atomic_t<T> helper) turns those reads well-defined. buf_page_set_accessed() now uses a compare_exchange from the zero time_point instead of a mutex-guarded load-then-store, so it no longer requires the caller to hold buf_page_get_mutex(). All loads/stores use std::memory_order_relaxed, matching the existing io_fix convention: ordering is still supplied by the surrounding mutex/latch protocol, this only removes the requirement to hold block->mutex purely to touch these two fields. Not changed: buf_page_optimistic_get() still takes buf_page_get_mutex() for its state-check + buf-fix step. Removing it would race with buf_LRU_block_remove_hashed(), which checks buf_fix_count == 0 once under the block mutex with no re-validation before freeing/reusing the block; making that path lock-free needs its own fix/evict protocol redesign, out of scope here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the fix_epoch seqlock design from buf_page_optimistic_get_lockfree_design.md: adds buf_page_t::fix_epoch, a per-block monotonic counter bumped odd->even by the evictor around the buf_fix_count==0 gate and state transition in buf_LRU_free_page() and buf_LRU_remove_all_pages() (the two buf_LRU_block_remove_hashed() callers that can reach a BUF_BLOCK_FILE_PAGE block a fast-path pointer could be guessing at; the third caller, buf_LRU_free_one_page() via buf_read_page_handle_error(), operates on a block freshly claimed for the current read attempt and needs no bracket -- see the design doc section 6). buf_page_optimistic_get() no longer takes buf_page_mutex_enter()/exit() for its state-check + buffer-fix + access-time read. It reads fix_epoch before and after buf_block_buf_fix_inc(); any mismatch (or an odd value up front) means a concurrent evictor's bracket overlapped the fix, so it backs out and falls through to the always-correct slow path. See the design doc sections 4-5 for the full protocol and correctness proof. This is explicitly a PoC for measuring the performance impact of removing that mutex on hot point-lookup/point-update workloads. It skips the staged rollout (kill-switch, instrumentation-only landing) the design doc recommends for a production change, and has not yet had the TSan/Helgrind stress soak the doc calls for as a prerequisite to shipping -- do not merge as-is. Smoke-tested: mysqld built with this change initializes, starts, and serves point SELECT/UPDATE against a freshly created table correctly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Moves the Case A/B/C correctness argument from buf_page_optimistic_get_lockfree_design.md into a self-contained comment on buf_page_t::fix_epoch (buf0buf.h), so a reviewer reading only the diff has the full proof without needing the external doc. Every call site that touches fix_epoch (buf_page_optimistic_get, buf_LRU_free_page, buf_LRU_remove_all_pages) now points at that comment instead of re-deriving or only citing the doc. Also documents, at the two sites it applies to, why they're excluded from the protocol rather than leaving that only in the design doc: buf_LRU_free_one_page() (the third buf_LRU_block_remove_hashed() caller, which doesn't need the bracket) and buf_relocate() (zip_mutex path, never touched by the lock-free fast path). No functional change -- comments only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
While setting up the TSan stress test, traced fil_discard_tablespace() and found it calls fil_delete_tablespace() with BUF_REMOVE_NONE, which never invokes buf_LRU_flush_or_remove_pages()/buf_LRU_remove_all_pages() at all. DISCARD instead reclaims pages lazily via the fil_space_t version/"stale" mechanism (buf_page_t::is_stale(), buf_page_free_stale() -> buf_LRU_free_page(), i.e. site 2). The only caller of BUF_REMOVE_ALL_NO_WRITE found in the tree is undo-tablespace cleanup at server startup (srv0start.cc:1026), before normal query traffic exists. The bracket at this site stays (defensive, no cost to keeping it, and nothing proves it will always remain unreachable at runtime), but the comment claiming "DISCARD TABLESPACE" as the reason was checked and is false -- corrected here and in buf_page_optimistic_get_lockfree_design.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prompted by a review question: could a block being freed and reused between a fixer's e1 and e2 reads cause fix_epoch to land back on a value the fixer already saw, defeating the e1==e2 check (ABA on the epoch itself, not just on the payload fields)? Traced it: fix_epoch is already never reset after a block's first-ever initialization (verified via every caller of buf_chunk_init() -- initial pool creation, and buf_pool_resize()'s growth path restricted to the newly-added chunk range -- both cases memory no fixer could hold a stale e1 for), and checked there's no stray memset reaching it either. So the ABA scenario doesn't occur as implemented; a 64-bit monotonic counter would need ~2^63 eviction attempts on one exact block to wrap back to a prior value. That reasoning previously lived only in conversation. Elevates it to a REQUIRED INVARIANT comment on buf_page_t::fix_epoch (buf0buf.h), matching the existing one for buf_fix_count's memory order, plus a pointer at the one legitimate init site (buf0buf.cc) -- so a future "reset stale state on free" cleanup doesn't silently reopen this. No functional change -- comments only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prompted by a review question: shouldn't the modify_clock check in buf_page_optimistic_get() be covered by the fix_epoch bracket too, the way it used to be covered by block_mutex? Traced modify_clock's documented invariant (buf0buf.h) and every real writer (btr0btr.cc, page0page.cc, page0cur.cc under the block's x-latch; buf0lru.cc:2410 under LRU_list_mutex with buf_fix_count==0): none of them were ever synchronized via block_mutex, before or after this PoC. So the first modify_clock comparison in buf_page_optimistic_get was already an unsynchronized heuristic read relative to its writers even in the original mutex-wrapped code -- block_mutex just happened to also be held there for the fields it does protect. The actual correctness-bearing check is the second one, further down, taken after this thread holds the real rw-latch every writer requires -- unchanged by this patch. Documents this at the check site so a future reader doesn't wonder the same thing and "fix" it by (incorrectly) folding modify_clock into the fix_epoch bracket. No functional change -- comment only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Review confirmed every caller of buf_page_optimistic_get() holds a buffer-fix on the block for the entire call (btr0cur.cc, btr0load.cc's Page_load, and row0sel.cc/gis0sea.cc via Block_hint::run_with_hint). Since buf_page_can_relocate() requires buf_fix_count == 0 (seq_cst), eviction of that block is already excluded before fix_epoch is ever consulted, making the seqlock redundant. Reverts fix_epoch (field, init, evictor brackets in buf0lru.cc, and the lock-free fast path in buf_page_optimistic_get()) back to the original block_mutex-protected code. Keeps the independently justified state/access_time atomic conversion. Design doc updated with the finding and a pointer to Block_hint::buffer_fix_block_if_still_valid() as prior art for the harder (no pre-existing fix) case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that fix_epoch is gone and buf_page_optimistic_get() relies on block_mutex again for ordering, these fields' relaxed loads/stores were only safe by riding the mutex's implicit acquire/release fence at their call sites. Bumping to seq_cst removes that implicit dependency: these fields' cross-thread visibility no longer depends on which lock (if any) happens to be held wherever they're touched, which matters given several call sites read/write them without block_mutex already (buf_page_is_accessed(), buf_page_set_accessed(), buf_page_is_private(), watch/sentinel handling, JSON dump). Applies uniformly to both the buf0buf.ic accessors and every direct bpage->state/access_time touch elsewhere in buf0buf.cc, buf0lru.cc, buf0flu.cc, fil0fil.cc, buf0buf.h. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The state/modify_clock check and buf_fix_inc at the top of this function no longer need block_mutex. Every caller already holds a buffer-fix on the block for the whole call (btr_cur_optimistic_latch_ leaves(), Page_load::latch(), and the row0sel.cc/gis0sea.cc callers via Block_hint::run_with_hint()), and buf_LRU_block_remove_hashed() -- the only function that frees/reuses a block -- is reachable only through buf_page_can_relocate(), which requires buf_fix_count == 0 (seq_cst). A standing fix therefore always excludes eviction of this exact block for the call's duration. state is now a seq_cst atomic; modify_clock was never protected by block_mutex to begin with (see its own invariant), so this read's race profile is unchanged, backed by the existing recheck under a real latch further down. Added a release-build ut_a(buf_fix_count > 0) at entry: this precondition is now load-bearing, and a future caller that violates it must fail loudly, not silently corrupt the pool. Second block_mutex bracket further down (access_time refresh) is untouched. Verified with a 5-minute TSan stress run (6 concurrent hot-row point-access threads against a 6M buffer pool holding ~18MB of data, forcing continuous LRU eviction). No race lands on the changed region (the state/modify_clock check or buf_fix_inc); the races that do mention buf_page_optimistic_get or buf_LRU_block_remove_hashed are pre-existing and unrelated: buf_pool->freed_page_clock and the per-block `old` LRU-repositioning hint, both read via buf_page_make_young_if_needed()/buf_page_peek_if_too_old(), a call path that was never under block_mutex before this change either; and rw_lock's own internal lock-word, unrelated to block_mutex entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.