Skip to content

Don't log NotFound as a critical error in delete_evicted_splits - #6622

Open
joepio wants to merge 1 commit into
quickwit-oss:mainfrom
joepio:fix/split-cache-evict-notfound-noise
Open

Don't log NotFound as a critical error in delete_evicted_splits#6622
joepio wants to merge 1 commit into
quickwit-oss:mainfrom
joepio:fix/split-cache-evict-notfound-noise

Conversation

@joepio

@joepio joepio commented Jul 24, 2026

Copy link
Copy Markdown

Description

SearchSplitCache::delete_evicted_splits logs every remove_file failure as a "critical" error implying an uncounted disk-space leak, but it doesn't special-case io::ErrorKind::NotFound and discards the actual io::Error, so a genuine failure and "someone already removed it" are indistinguishable from the logs.

The sibling .temp cleanup a few lines above in the same file already handles this correctly:

if let Err(io_err) = std::fs::remove_file(&path)
    && io_err.kind() != io::ErrorKind::NotFound
{
    error!(path=?path, "failed to remove temporary file");
}

This PR mirrors that pattern for delete_evicted_splits, and includes the real io_err in the log for the cases that remain.

Why this matters in practice

On a self-hosted single-node deployment (0.8.1) with a very short commit_timeout_secs (aggressive backfill ingest, lots of small-split churn), we run an external sidecar that also removes orphaned split-cache files directly (diffing the cache directory against the metastore's Published set) as a workaround for cache-pollution buildup. When that sidecar wins the race against delete_evicted_splits for the same file, Quickwit's own eviction logs the "critical... disk space leak" message above even though the outcome (file gone) is exactly what was wanted. There's no way to tell that apart from an actual leak without attaching a debugger, since the real io::ErrorKind is discarded.

Even without an external deleter in the picture, any two callers racing eviction for the same split (e.g. overlapping LRU decisions) would hit the same false alarm.

How was this PR tested?

Small, mechanical change mirroring an existing pattern in the same file; no behavior change for genuine (non-NotFound) failures beyond now logging the real error.

remove_file() failures here were logged as "critical... disk space
leak" unconditionally, including on plain NotFound, and the actual
io::Error was discarded so a real failure and "someone already removed
it" were indistinguishable. The sibling .temp cleanup a few lines
above already special-cases NotFound; this mirrors that and surfaces
the real error for the cases that remain.
@joepio
joepio requested a review from a team as a code owner July 24, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant