[feature](file cache) Add asynchronous file cache writes

[bobhan1]

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

On a file-cache miss, CachedRemoteFileReader currently performs two different kinds of work on the query thread:

1. read bytes from remote storage to satisfy the query;
2. append and finalize those bytes into the local file cache for future queries.

The first operation is on the critical path. The second is best-effort cache population, but local filesystem latency and backpressure are still charged to foreground scan latency because the two operations are coupled.

This PR implements phase 1 of asynchronous file-cache writes. For ordinary reads, the query thread returns after the caller's buffer is complete and hands eligible cache-miss blocks to per-disk background workers. The feature is opt-in and disabled by default. Explicit cache-population operations, including warm-up and prefetch/download paths, retain synchronous completion semantics.

Simply moving append() to another thread is unsafe. The design also has to preserve cache-block ownership, avoid duplicate queued writes, survive clear/remove races, bound background work, and keep existing warm-up and dry-run behavior. This PR introduces dedicated components for those responsibilities instead of transferring existing FileBlock objects across threads.

Goals

• Remove best-effort local cache persistence from the ordinary remote-read critical path.
• Reuse remote bytes that have already been fetched but are still waiting for local persistence.
• Avoid duplicate background writes for the same cache block.
• Bound the buffer-capacity ownership of accepted queued and active tasks without making queries wait for disk persistence; when saturated, prefer newly read blocks by evicting only the oldest queued task.
• Prevent stale tasks from recreating cache data after clear/remove operations.
• Preserve the existing synchronous behavior for explicit cache-population callers.
• Provide query-profile, per-cache-disk core, and BE-global cache-probe observability for the new path.

Non-goals of this phase

• This is not remote-read singleflight. Two cold readers that overlap before either publishes its completed remote buffer can still issue separate remote reads.
• The asynchronous write path does not consume peer-cache reads. A call that selects peer cache keeps synchronous cache-write semantics; after a call selects asynchronous write, it pins direct remote storage for the rest of that call even if the online peer-read policy changes.
• This does not change local file-cache durability guarantees; persistence remains best effort.
• This does not make warm-up or prefetch completion asynchronous.
• The asynchronous path does not support file_cache_query_limit_bytes or file_cache_query_limit_percent; deployments that depend on either query-scoped admission limit must keep asynchronous file-cache writes disabled.
• This does not enable the feature by default.

Performance validation

A storage-compute separated PL1 benchmark was run on commit b586dd1f429 with one 32-core, 123 GiB BE, two PL1 file-cache disks, 16 background write workers per disk, and enable_adaptive_scan=false. Each round restarted the BE with an empty file cache. The workload was a full scan of ssb.lineorder: about 6 billion rows and 17 columns, with 135.8 GB reported by SHOW DATA.

Pending task ownership versus the active working set

At 64 target scanners, the estimated active buffer working set was 16 instances x 4 scanners x 17 columns x 1 MiB = 1,088 MiB, or about 544 MiB per cache disk.

Mode	Effective pending ownership per cache instance	Query time	Remote I/O requests
Sync	0	171 s	138,290
Async	256 MiB	136 s	261,980
Async	512 MiB	104 s	186,861
Async	1,024 MiB	90 s	150,509
Async	2,048 MiB	89 s	144,457

With the pending ownership limit below the working set, older queued buffers can be evicted before later scans reuse them or before persistence completes, causing repeated remote reads. Once the pending ownership limit covers the working set, reuse improves and foreground scans no longer wait for local persistence. At 1,024 MiB per disk, query time fell from 171 s to 90 s, a 47.4% reduction. Increasing the limit to 2,048 MiB per disk improved the result by only one additional second. The synchronous baseline already drove one cache disk to 94.4% utilization, confirming that local cache writes were the bottleneck in this workload.

Scanner scaling after working-set coverage

The pending ownership limit was then increased with scanner concurrency so that each run continued to cover its estimated working set. Remote I/O stayed within 6.2% of the synchronous cold-cache baseline.

Peak active scanners	Effective pending ownership per cache instance	Query time	Network receive	Combined disk write	Remote I/O delta
63	2,048 MiB	90 s	1,677.4 MiB/s	634.6 MiB/s	+4.4%
94	3,072 MiB	64 s	2,398.7 MiB/s	634.5 MiB/s	+6.2%
128	4,096 MiB	56 s	2,725.8 MiB/s	635.8 MiB/s	+5.6%

As peak scanner concurrency increased from 63 to 128, query time fell from 90 s to 56 s and network receive throughput increased, while combined cache-disk write throughput remained approximately 635 MiB/s. This shows that asynchronous persistence removed the saturated local disks from the foreground critical path for this workload. Gains began to narrow at 128 scanners, indicating that the next bottleneck was shifting toward the network, remote storage, or CPU.

These results are workload-specific rather than a universal speedup claim. The benefit requires both a foreground bottleneck in local file-cache writes and enough effective accepted-task ownership on each cache instance to cover the active scanner working set. The tables report the effective per-instance shares used by the benchmark. With the current configuration, async_file_cache_write_max_pending_bytes is one BE-wide total divided equally among successfully initialized cache instances; reproducing a two-disk row therefore requires a BE-wide total twice the listed per-instance value. This remains an ownership limit, not an aggregate hard limit on all live async buffer memory.

Architecture

The query thread owns read planning and result assembly. It performs one whole-range inflight lookup and, unless inflight buffers cover every aligned block, at most one read-only probe for the complete aligned request. It then represents each aligned block with a small source classification. Background workers own cache-cell acquisition and persistence. The two sides exchange immutable, reference-counted buffers through an inflight index and a per-disk manager whose admission is bounded by accepted queued and active task ownership.

flowchart LR
    Caller[Scan / query caller]
    Reader[CachedRemoteFileReader]
    Remote[Direct remote FileReader]

    subgraph CacheDisk[One file-cache disk]
        Probe[BlockFileCache read-only probe]
        Index[InflightWriteBufferIndex]
        Manager[AsyncCacheWriteManager]
        Queue[Locked FIFO deque with byte admission]
        Workers[Resizable write workers]
        Storage[BlockFileCache and local storage]
    end

    Metrics[Runtime profile, per-disk bvars, and global probe bvars]

    Caller -->|read request| Reader
    Reader -->|lookup completed remote buffers| Index
    Reader -->|observe cache state without ownership| Probe
    Reader -->|one remaining middle-range read| Remote
    Remote -->|aligned bytes| Reader
    Reader -->|complete caller buffer| Caller
    Reader -->|publish true MISS buffers| Index
    Reader -->|submit without disk IO wait| Manager
    Manager --> Queue
    Queue --> Workers
    Workers -->|revalidate, acquire, append, finalize| Storage
    Workers -->|conditional cleanup| Index
    Storage -.->|clear/remove advances write epoch| Manager
    Reader -.-> Metrics
    Index -.-> Metrics
    Manager -.-> Metrics

Loading

Component responsibilities

Component	Responsibility	Important guarantee
CachedRemoteFileReader	Resolves synchronous versus asynchronous mode for every read, pins asynchronous calls to direct remote storage, performs one whole-range inflight lookup and at most one whole-range cache probe, classifies aligned blocks, performs the required remote read, fills the caller buffer, and submits only true misses.	The caller never observes an incomplete buffer. Cache-write submission is not allowed to turn a successful remote read into a query failure.
BlockFileCache::probe	Reports downloaded, downloading, empty, deleting, and missing ranges without creating cache cells or taking downloader ownership.	Planning does not mutate cache state or transfer thread-affine FileBlock downloader ownership. LRU touch happens only after a block is actually consumed.
InflightWriteBufferIndex	Stores shard-protected references to completed remote buffers that are pending persistence, keyed by cache key and block offset and tagged with a write epoch.	Insert-if-absent selects one background-write owner. Conditional removal prevents an old completion callback from removing a newer entry.
AsyncCacheWriteManager	Owns the per-disk mutex-protected FIFO deque, queued/active/pending count and byte accounting, fixed drop-oldest admission, tracked-buffer accounting, resizable workers, shutdown, and manager-level metrics.	Submission does not wait for disk I/O. At the byte limit it accepts the new fixed-size task by evicting only the oldest queued task; if there is no queued victim or one task exceeds the limit, it rejects and rolls back the new inflight entry. Active tasks are never evicted.
Async write worker	Rechecks the write epoch and current cache state, calls the normal cache admission/cell-creation path, skips blocks already downloaded or owned by another downloader, and appends/finalizes only eligible empty blocks.	Background work follows current cache state; it does not blindly overwrite or resurrect blocks.
Cache write epoch	Represents the generation of work accepted by a cache instance. Clear/remove paths advance it.	A task accepted under an older generation is dropped before or during persistence.
Profile and bvar integration	Records query-local submission/rejection/inflight reuse, five core per-cache-disk bvars, and four BE-global probe counters.	Operators can observe query-path outcomes, worker availability, live async-buffer memory, FIFO eviction, and cache-probe results without multiplying low-value per-disk series.

Important flows

1. Ordinary read with mixed coverage

Each aligned block is classified in this order:

1. completed remote buffers in InflightWriteBufferIndex;
2. existing cache blocks through the read-only probe;
3. remaining blocks as misses.

The lookup covers the complete aligned request. A full inflight hit skips the file-cache probe; otherwise one probe covers the same whole range, so the planner does not build or repeatedly probe individual uncovered runs. It identifies the first and last real cache miss, materializes only inflight/cache blocks outside those boundaries, and retains the existing wait behavior for a DOWNLOADING block outside the remote span. If misses remain, everything from the first through the last miss is fetched with one aligned direct-remote read. Existing cache, inflight, or downloading blocks inside that span are intentionally covered by the same remote result instead of being filled or waited one by one. Only the blocks that were real misses are published and submitted for asynchronous persistence.

sequenceDiagram
    participant C as Caller
    participant R as CachedRemoteFileReader
    participant I as Inflight index
    participant F as BlockFileCache
    participant O as Direct remote storage
    participant S as Async write manager
    participant W as Worker

    C->>R: read_at(offset, size)
    R->>I: lookup aligned blocks for current epoch
    I-->>R: reusable pending buffers
    R->>F: at most one read-only probe for the aligned request
    F-->>R: downloaded / downloading / miss coverage
    R->>R: materialize blocks outside the first-to-last miss span

    alt all requested bytes are covered
        R-->>C: return completed caller buffer
    else a middle range remains
        R->>O: one aligned middle-range read
        O-->>R: remote bytes
        R->>R: copy requested overlap into caller buffer
        loop each true MISS block in the remote span
            R->>I: insert completed block buffer if absent
            alt another owner already published it
                I-->>R: existing entry, skip duplicate write
            else this reader owns persistence
                R->>S: try_submit without waiting
                alt fits pending byte limit
                    S-->>R: accepted at FIFO tail
                else full with a queued victim
                    S-->>R: accepted and oldest queued task evicted
                    S->>I: conditional victim cleanup outside queue lock
                else task exceeds byte limit, no queued victim, or manager stopped
                    S-->>R: rejected
                    R->>I: conditional rollback
                end
            end
        end
        R-->>C: return completed caller buffer
        S->>W: dequeue in background
        W->>F: revalidate epoch and current block state
        W->>F: append and finalize eligible empty blocks
        W->>I: conditional completion cleanup
    end

Loading

2. Later reader reuses a pending buffer

After the first remote read publishes a buffer and before its worker finishes persistence, a later reader can copy the requested bytes directly from the inflight entry. It does not wait for the local file to be finalized and does not submit another write task for that block.

This index deduplicates pending cache writes and bridges the interval between remote-read completion and local persistence. It intentionally does not claim to deduplicate remote requests that began before publication.

3. Pending-ownership admission and asynchronous failure

Each cache instance's share of the BE-wide pending-byte limit accounts the full buffer-capacity bytes owned by accepted queued and active tasks. The manager protects a FIFO deque and all queued/active/pending ownership transitions with one mutex, maintaining both pending_count = queued_count + active_count and pending_bytes = queued_bytes + active_bytes.

This is an admission and ownership-accounting limit, not a hard upper bound on every live async buffer allocation. A buffer is allocated before try_submit(). After a replacement or worker completion, the task can also remain live while its callback removes the inflight entry and while other shared references are released, even though it no longer contributes to pending_bytes. The manager memory tracker and inflight-index buffer gauges expose those live allocations separately.

Production submits one task per aligned cache block. Every task allocates exactly file_cache_each_block_size bytes; write_size is the valid prefix and may be shorter only for the physical EOF block. Admission and byte accounting use the full buffer capacity, including for that short tail block. Therefore a configured remainder smaller than one cache block is intentionally unusable, and replacing one queued task with another preserves pending bytes exactly.

• If one task's fixed buffer is larger than the effective per-instance limit, the task is rejected.
• While pending_bytes + task_buffer_bytes fits, the new task is appended to the FIFO tail and increases queued and pending count/byte state.
• Without capacity but with at least one queued task, the manager appends the new task and removes exactly the oldest queued task from the head. The new submission succeeds; queued and pending count/byte state stays unchanged, and active tasks are never selected as victims.
• Without capacity and with an empty queue, every pending task is active, so the new task is rejected because there is no queued victim.
• After a runtime limit decrease, replacement continues while a queued victim exists even if current pending bytes remain above the new limit. This does not increase accounted pending ownership; active work drains toward the new limit.

Evicted-task metrics, callback execution, conditional inflight-index removal, and buffer release happen outside the queue mutex. An evicted task is best-effort cache population that never reaches a worker; it does not affect the query that already received its remote bytes.

Likewise, an append/finalize failure is logged and counted by the worker; it does not retroactively fail a query whose remote read succeeded. A future read simply observes another cache miss and may retry population.

4. Clear/remove and stale-task handling

Cache invalidation advances the manager's write epoch. Both inflight lookup/replacement and worker processing compare their entry or task against the current epoch. Workers check the epoch before acquiring cache blocks and again while processing them. This closes the window where old queued work could recreate blocks after a clear/remove operation.

Completion cleanup uses pointer-conditional removal. Therefore, even if a newer generation has published an entry for the same block offset, an older task cannot erase it.

5. Mode selection and compatibility

CacheWriteMode is resolved for every read so an online configuration switch affects existing readers. Ordinary reads may select the asynchronous path when the feature is enabled. Peer-cache reads are not supported by the asynchronous path: a read that observes peer mode during resolution selects synchronous cache write. Once asynchronous mode has been selected, the reader copies the call's IOContext and sets bypass_peer_read on that private copy, pinning direct remote storage for the rest of the call even if the online peer-read policy changes; the caller's context is not mutated. Dry-run, warm-up, segment-index population, downloader, and explicit prefetch paths also select or override synchronous mode because their completion semantics depend on the cache being populated before returning.

The default remains synchronous because enable_async_file_cache_write defaults to false.

The asynchronous path currently does not support file_cache_query_limit_bytes or file_cache_query_limit_percent. Background tasks do not retain the query-owned RemoteScanCacheWriteLimiter or the query-context-holder lifetime required by those admission policies, so their limits are not guaranteed for asynchronous persistence. Keep asynchronous file-cache writes disabled when either query-level limit is required.

6. Worker lifecycle

Each BlockFileCache creates its inflight index and async-write manager state during initialization. When asynchronous writeback is disabled, the manager does not create persistent worker loops. Enabling the feature online explicitly starts all initialized per-disk managers.

Workers pop one task at a time from the FIFO head under the same queue mutex used by producers and atomically move it from queued to active. There is no batch-size setting or bulk dequeue. When an active task finishes, active and pending count/byte state is decremented under the queue mutex; lifecycle metrics and the task callback run after releasing it.

Each worker object owns its stop request and completion signal. Worker count can be resized online without exposing numeric worker IDs as lifecycle state. Before submitting long-running Worker tasks, the manager raises the pool's minimum thread count to the configured worker count, so an OS-thread creation failure is returned before any new Worker task can be accepted and stranded in the pool queue. Shutdown first stops new submitters, waits for already registered submitters, drains accepted tasks, joins workers, and only then destroys state referenced by task-finalization callbacks. There is no second persistence pool or cross-thread FileBlock ownership transfer.

Configuration

Configuration	Default	Purpose
enable_async_file_cache_write	false	Enables the ordinary asynchronous write path.
async_file_cache_write_workers_per_disk	16	Number of persistence workers for each cache disk; supports online resize. Workers are started only when asynchronous writeback is enabled.
async_file_cache_write_max_pending_bytes	-1 (automatic)	BE-wide accepted queued + active task buffer-capacity ownership limit. A positive value is the exact BE-wide total; -1 resolves to max(1 GiB, 1% of the BE memory limit) before the total is split equally across successfully initialized cache instances. Supports online update.
enable_async_file_cache_write_inflight_write_buffer_index	true	Enables pending-buffer reuse and duplicate-write suppression.
async_file_cache_write_inflight_write_buffer_index_shard_count	64	Controls index lock sharding at cache initialization; it is not mutable online.

async_file_cache_write_max_pending_bytes is one BE-wide total. After cache initialization, FileCacheFactory divides it by the number of successfully initialized cache instances and applies the same floor-divided share to every per-disk manager. Cache keys are hash-sharded across those instances, so this keeps the aggregate ownership budget stable as the cache-disk count changes; integer-division and sub-cache-block remainders are intentionally unused. The value is an upper bound on accepted queued + active task ownership, not a reservation or a hard upper bound on all live async buffer memory. Task buffers are allocated lazily before admission, and finalizing or externally referenced buffers can outlive pending ownership. The queue-full behavior is fixed to FIFO drop-oldest and has no separate policy configuration. Accepted tasks are not discarded by an age watchdog.

Observability

Query runtime profiles expose asynchronous submissions, rejections, buffer-allocation failures, stale-epoch drops, and inflight hits/misses.

Each BlockFileCache instance exposes five bvars prefixed by its cache base path:

• async_cache_write_running_worker_count: currently running persistence workers;
• async_cache_write_buffer_memory_bytes: all live AsyncCacheWriteBuffer capacity charged to the manager memory tracker;
• async_cache_write_submitted_task_count: accepted asynchronous write tasks;
• async_cache_write_evicted_oldest_task_count: queued tasks evicted by fixed FIFO drop-oldest admission;
• inflight_write_buffer_index_buffer_bytes: buffer capacity currently retained by inflight-index entries.

The BE exposes four global cache-probe bvars:

• cached_remote_file_reader_probe_count: whole-range cache probes performed while building asynchronous read plans;
• cached_remote_file_reader_probe_hit_downloaded_count: probed logical blocks already downloaded;
• cached_remote_file_reader_probe_hit_downloading_count: probed logical blocks currently downloading;
• cached_remote_file_reader_probe_miss_count: probed logical blocks classified as real misses.

Detailed queue and pending accounting, rejection-reason breakdowns, operation latencies, write-epoch details, state-based skips, failures, and inflight-index operation breakdowns are not exposed as named bvars. Query-relevant outcomes remain available through runtime profiles, while focused tests inspect manager diagnostics directly without adding scrape series.

async_cache_write_buffer_memory_bytes is the more complete live-memory signal: task buffers are allocated before admission and can outlive pending ownership while callbacks and other shared references release them. inflight_write_buffer_index_buffer_bytes reports only capacity retained by index entries, so the two gauges can differ transiently and when the index is disabled.

Tests

• Latest bvar-surface reduction: ./build.sh --be -j100, build-support/clang-format.sh, and build-support/check-format.sh passed. Focused unit tests were not rerun because the final commits only removed metric exposure and collection; the behavioral tests below predate that observability-only cleanup.

• Latest review fixes: 7/7 targeted ASAN BE unit tests passed.

  • asynchronous reads remain pinned to direct remote storage after an online peer-policy change;
  • tracked-buffer allocator failures are caught and converted to a status;
  • worker resize reserves backing pool threads before publishing long-running Worker tasks, while existing stop/resize ownership tests continue to pass.

• Current locked-FIFO/drop-oldest manager: 21/21 AsyncCacheWriteManagerTest cases passed after the worker-lifecycle encapsulation.

  • FIFO order, active-only saturation, one-for-one oldest eviction, epoch-safe callback cleanup, runtime byte-limit convergence, direct count/byte conservation, worker resize, and shutdown/replacement races.

• Current one-task-at-a-time queue and read integration: 36/36 focused manager, inflight-index, and async-reader tests passed after removing the ineffective batch-size path.

• Pending-ownership-bounded admission and read/task contracts: 37/37 focused manager and reader tests passed.

  • fixed cache-block buffer capacity with a short physical EOF write_size;
  • queued + active byte limits, one-for-one oldest replacement, runtime limit decrease, inflight cleanup, and direct count/byte accounting.

• Latest BE-wide pending-limit update: 26/26 focused manager and configuration BE unit tests passed.

  • positive BE-wide totals, the 1 GiB automatic floor, 1% memory scaling, and invalid sentinels;
  • one- and two-cache creation, equal per-instance splitting, and online switching between a fixed BE-wide total and -1 automatic sizing.

• Latest read-planning refactor: 6/6 targeted ASAN BlockFileCacheTest cases passed.

  • inflight-buffer reuse and downloaded-block reads;
  • DOWNLOADING wait outside the remote span;
  • cached prefix/suffix assembly;
  • one first-to-last-miss remote read with an existing middle block;
  • true-miss-only submission and backpressure rollback;
  • per-read cache-write mode resolution.

• Docker cloud regression: 1/1 suite passed.

  • a cold query submits asynchronous cache writes;
  • after the queue drains, a second query observes probe hits;
  • switching the feature off restores synchronous behavior without asynchronous submissions.

• Build and style:

  • ./build.sh --be --fe --cloud -j100
  • ./build.sh --be -j100
  • ./build.sh --be --file-cache-microbench -j100 and an all-mode microbenchmark smoke run completed without hanging
  • build-support/check-format.sh
  • clang-tidy was intentionally not run for this change.

Release note

Add an opt-in asynchronous file-cache write path controlled by enable_async_file_cache_write. Each cache disk uses a locked FIFO whose accepted queued + active task buffer ownership is byte-bounded; when the effective per-instance limit is reached, a new fixed-block task replaces the oldest queued task, while active tasks are never evicted. This is not a hard upper bound on every live async buffer allocation. async_file_cache_write_max_pending_bytes defaults to -1, which resolves to a BE-wide max(1 GiB, 1% of the BE memory limit); the resolved total is split equally across successfully initialized cache instances. The asynchronous path does not support peer-cache reads, file_cache_query_limit_bytes, or file_cache_query_limit_percent. The feature remains disabled by default.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (Docker cloud regression and runtime configuration switch)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes. When explicitly enabled, ordinary file-cache misses persist eligible cache blocks through background workers using per-disk FIFO drop-oldest admission whose byte limits are equal shares of one BE-wide pending-ownership budget. Default behavior and explicit cache-population completion remain synchronous.

• Does this need documentation?

  • No. The feature is experimental and disabled by default.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[bobhan1]

/review

[github-actions]

Request changes: the async-write design is broadly implemented and the happy-path coverage is substantial, but nine blocking issues remain.

• Reader correctness: legal noncanonical/restored cache cells can abort probe(), and a live peer-cache enable transition can make the async path abort the BE with a null remote buffer.
• Concurrency and lifecycle: accepted-but-unstarted persistent worker loops can deadlock shrink; invalidation can enqueue a generation-blind delete that removes a newer refill.
• Configuration and errors: runtime/startup validation does not validate a coherent incoming option set; concurrent callbacks can apply stale snapshots; online enablement reports success after service-start failure.
• Admission: async handoff drops both file_cache_query_limit_bytes and the lifetime needed to enforce file_cache_query_limit_percent.
• Performance: the default idle workers poll every millisecond, around 16,000 timed wakeups per second per cache disk, while the current notification protocol is not safe for a naive indefinite wait.

Critical checkpoint conclusions:

• Goal and test proof: ordinary aligned misses do return promptly and persistence is handed to a bounded service. Unit/regression coverage is broad, but it omits the negative and concurrency cases identified inline.
• Scope, parallel paths, and compatibility: downloader, index-loader, prefetch, dry-run, warm-up, and stable peer paths retain synchronous completion. No FE/BE wire, transaction, EditLog, MoW, or visible-version contract changes were found. Existing writer/restored cache layouts are not compatible with the new exact-boundary probe.
• Concurrency, lifecycle, configuration, and errors: shutdown/destruction order, thread context, tracker ownership, lock order, and pointer-conditional inflight cleanup are sound. The worker-start/shrink, generation deletion, validation, snapshot ordering, and enablement failures remain blocking.
• Data and persistence: source data is immutable/remote and no transactional persistence boundary changes. The generation race can delete a newer derived cache file/meta and force self-heal, but does not corrupt the remote source.
• Observability and performance: new query/service metrics are propagated through merge/diff/reporting and scanner detection. The 1 ms idle wake protocol is the remaining observability/performance concern.
• Tests and validation: static review only, as the review contract forbids builds and source changes. .worktree_initialized and thirdparty/installed are also absent. Existing review threads/comments were empty, and all three round-two reviewers returned NO_NEW_VALUABLE_FINDINGS against this exact nine-comment set.
• User focus: no additional focus was supplied; the whole PR was reviewed.

[bobhan1]

run buildall

[bobhan1]

run buildall

[bobhan1]

run buildall

[bobhan1]

/review

[github-actions]

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-20T03:35:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/32326126737

Please trigger /review again after that time.

[hello-stephen]

TPC-H: Total hot run time: 17419 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2fe45a97ba588677b90f450d2fc244ebf233305e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17583	3117	3074	3074
q2	1899	240	150	150
q3	10440	913	518	518
q4	4673	242	200	200
q5	7681	576	392	392
q6	140	118	94	94
q7	532	507	388	388
q8	9240	891	879	879
q9	3427	2437	2438	2437
q10	6515	840	714	714
q11	452	266	241	241
q12	704	392	331	331
q13	17862	1528	1181	1181
q14	161	143	144	143
q15	q16	425	398	363	363
q17	816	735	815	735
q18	3105	2245	2261	2245
q19	1251	897	783	783
q20	682	546	475	475
q21	5654	1842	1852	1842
q22	335	260	234	234
Total cold run time: 93577 ms
Total hot run time: 17419 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3503	3405	3400	3400
q2	216	215	155	155
q3	2180	2347	2203	2203
q4	1181	1165	905	905
q5	2167	2118	2143	2118
q6	169	121	87	87
q7	1059	924	856	856
q8	1611	1437	1421	1421
q9	3133	3111	3111	3111
q10	1864	1783	1629	1629
q11	355	275	259	259
q12	461	422	338	338
q13	1479	1537	1160	1160
q14	163	182	163	163
q15	q16	396	390	357	357
q17	1061	1051	1040	1040
q18	4906	4405	4750	4405
q19	854	849	841	841
q20	966	952	813	813
q21	3787	3192	3316	3192
q22	413	366	322	322
Total cold run time: 31924 ms
Total hot run time: 28775 ms

[hello-stephen]

TPC-DS: Total hot run time: 83291 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2fe45a97ba588677b90f450d2fc244ebf233305e, data reload: false

query5	4289	431	335	335
query6	413	162	158	158
query7	4840	448	270	270
query8	297	129	122	122
query9	8701	2878	2946	2878
query10	431	267	219	219
query11	5386	1052	925	925
query12	117	69	71	69
query13	1194	457	313	313
query14	6137	2223	2085	2085
query14_1	1974	1988	1950	1950
query15	175	118	107	107
query16	922	389	346	346
query17	792	451	363	363
query18	2325	339	273	273
query19	166	137	114	114
query20	71	70	70	70
query21	215	116	102	102
query22	5519	5444	5293	5293
query23	6601	6146	5976	5976
query23_1	6115	6012	6028	6012
query24	7306	1095	762	762
query24_1	777	751	770	751
query25	417	297	259	259
query26	1255	276	165	165
query27	2700	431	278	278
query28	4677	1499	1487	1487
query29	955	431	350	350
query30	276	176	151	151
query31	849	420	354	354
query32	104	49	48	48
query33	472	218	168	168
query34	1003	833	498	498
query35	401	397	338	338
query36	578	558	518	518
query37	122	83	71	71
query38	1008	842	822	822
query39	505	493	470	470
query39_1	438	461	482	461
query40	226	129	114	114
query41	59	56	57	56
query42	84	86	89	86
query43	248	240	214	214
query44	1043	558	540	540
query45	111	111	102	102
query46	768	849	560	560
query47	763	768	705	705
query48	310	315	234	234
query49	552	235	198	198
query50	798	350	258	258
query51	8015	8047	8014	8014
query52	76	74	71	71
query53	208	214	167	167
query54	241	220	187	187
query55	79	64	58	58
query56	267	238	234	234
query57	740	666	637	637
query58	262	210	187	187
query59	1232	1239	1084	1084
query60	270	203	192	192
query61	112	116	118	116
query62	384	200	172	172
query63	193	160	151	151
query64	2746	710	602	602
query65	1671	1583	1641	1583
query66	1830	307	312	307
query67	9589	9584	9609	9584
query68	3021	1245	748	748
query69	347	228	201	201
query70	658	594	613	594
query71	287	254	267	254
query72	2347	1720	1681	1681
query73	659	575	363	363
query74	2022	1210	1139	1139
query75	1222	1148	1016	1016
query76	2371	758	543	543
query77	258	246	204	204
query78	3934	3721	3220	3220
query79	2793	819	579	579
query80	1584	395	343	343
query81	529	190	175	175
query82	656	131	105	105
query83	326	257	234	234
query84	320	119	102	102
query85	897	453	382	382
query86	466	173	166	166
query87	1005	989	880	880
query88	3003	2157	2140	2140
query89	320	225	202	202
query90	2028	150	143	143
query91	157	149	123	123
query92	67	48	44	44
query93	2052	1127	831	831
query94	650	262	223	223
query95	607	388	422	388
query96	796	609	257	257
query97	1049	1010	998	998
query98	181	134	140	134
query99	410	356	310	310
Total cold run time: 179825 ms
Total hot run time: 83291 ms

[hello-stephen]

ClickBench: Total hot run time: 14.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2fe45a97ba588677b90f450d2fc244ebf233305e, data reload: false

query1	0.01	0.00	0.01
query2	0.07	0.04	0.04
query3	0.26	0.11	0.11
query4	1.60	0.09	0.09
query5	0.17	0.16	0.16
query6	1.27	0.67	0.70
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.22
query10	0.35	0.36	0.39
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.30	0.30	0.30
query14	0.45	0.45	0.45
query15	0.36	0.35	0.34
query16	0.22	0.21	0.23
query17	0.67	0.68	0.70
query18	0.18	0.16	0.17
query19	1.18	1.18	1.15
query20	0.02	0.02	0.01
query21	15.43	0.17	0.11
query22	5.09	0.04	0.04
query23	16.15	0.25	0.11
query24	3.00	0.29	0.28
query25	0.10	0.03	0.04
query26	0.77	0.17	0.11
query27	0.04	0.03	0.04
query28	3.59	0.55	0.28
query29	12.45	3.12	2.59
query30	0.26	0.12	0.12
query31	2.75	0.37	0.18
query32	3.52	0.31	0.23
query33	1.43	1.52	1.51
query34	15.38	2.18	1.74
query35	1.74	1.77	1.71
query36	0.46	0.29	0.30
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.10	0.08	0.07
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.33 s
Total hot run time: 14.8 s

[bobhan1]

run cloud_p0

[bobhan1]

/review

[github-actions]

Request changes: one new P1 startup-compatibility regression remains in the opt-in asynchronous file-cache write feature.

Review completion: converged after three complete rounds. The final normal configuration/lifecycle pass, normal reader/data/test pass, and separate adversarial risk pass all returned NO_NEW_VALUABLE_FINDINGS. MAIN-001 was independently revalidated; every other candidate was dismissed with code evidence or duplicate-fenced against live threads and landed review bodies.

Critical checkpoint conclusions:

• Goal, scope, focus, clarity, and proof: the cohesive 48-file change implements opt-in query-side asynchronous cache persistence with bounded per-disk workers, inflight reuse, dynamic options, metrics, unit tests, a cloud regression, and a benchmark. Happy-path proof is broad, but the unconditional startup gate violates opt-in compatibility and lacks a feature-disabled upgrade test. No additional user focus was supplied.
• Concurrency, locks, and lifecycle: query submitters, persistent workers, config threads, concurrent readers, invalidation, and cleanup were traced through lifecycle, queue, shard, cache, and block locks plus stop/epoch atomics. Manager, worker, buffer, inflight, epoch, probe, block, and reader ownership were checked through start, resize, invalidate, exceptions, shutdown, and teardown. No new distinct deadlock or lifecycle issue survived the existing live-thread fences.
• Configuration and compatibility: enable/disable, worker/pending limits, validators, callbacks, per-disk option fan-out, and HDFS/S3 writer sizing were checked. The attached P1 makes a previously valid 640 KiB cache-block/default-writer configuration startup-fatal on upgrade while the dependent feature is disabled. Existing runtime-validation, partial-application, stale-snapshot, enablement, and noncanonical-layout defects were not duplicated. There is no FE/BE wire, Thrift, EditLog, transaction-visibility, or durable user-data format change.
• Parallel paths and special conditions: Doris/external, direct/inflight/probe/remote, peer, EOF, downloader, prefetch, segment-index, warmup/dry-run, S3, HDFS, and packed paths were traced. Completion-sensitive population callers remain synchronous. Dormant UNALIGNED and NO_WRITE states have no current production producer.
• Tests and results: manager/index/probe/reader/writer tests, the cloud regression/output, and benchmark plumbing were inspected, including negative, EOF, rejection, exception, invalidation, resize, shutdown, and mixed-read cases. The feature-disabled startup-compatibility case is missing. No build or test command was run because the authoritative review prompt requires static review only.
• Observability, persistence, and performance: profiles, metrics, bvars, logs, bounded admission, worker pools, and inflight deduplication were reviewed. The changed persistence is expendable BE cache state rather than database transaction or user-data durability state. All other surviving observability, accounting, direct-map, wakeup, benchmark, append/finalize, invalidation, and cleanup concerns already have live review owners.
• FE/BE variables and other checks: all new settings and reader contexts are BE-local, with no cross-process propagation path. All 48 authoritative changed files matched the bundle and were swept; every candidate was accepted, dismissed with evidence, or deduplicated.