Add an optional Foyer-backed data-file cache by zhangstar333 · Pull Request #73 · lance-format/lance-c

zhangstar333 · 2026-09-03T05:01:46Z

#71 not merge to main branch, only as patch.

lance-gatekeeper

⚠️ Gate recommendation: approve with a non-blocking risk.

The eager-stream blocker is fixed: cached gets now return before payload I/O, fetch at most one cache block per poll, and stop reading when the consumer drops the stream. The bounded-read-ahead, repeated-scan, and prior regression coverage passes at this head.

The remaining risk is operational: a cold multi-block stream issues one sequential origin range request per cache block, and concurrent cold readers still do not coalesce misses. Normal large scanner reads retain the existing coalesced get_ranges path, but cold streamed reads should be benchmarked with the chosen block size before broad rollout; keep sessions long-lived and cached bytes disposable.

### What problem does this PR solve? Issue Number: None Related PR: lance-format/lance-c#73 Problem Summary: Refresh the bundled Lance-C PR apache#73 patch from head a39bc8b08cd1367b7445cb9acf3309faa1f3d610 to a4c71309ddb76ad79808e3e8f7797bcd9bfc174a. The updated cache implementation extends coverage to whole-object and single-range reads while preserving the existing v0.1.9 plus PR apache#73 then PR apache#74 patch order. ### Release note None ### Check List (For Author) - Test: No need to test (the refreshed PR apache#73 and existing PR apache#74 patches apply in order to a clean v0.1.9 archive with zero fuzz and no rejects or backup files; compilation and tests were not requested) - Behavior changed: Yes. The bundled PR apache#73 data-cache patch is refreshed to its latest upstream head. - Does this need documentation: No

Issue Number: None Related PR: lance-format/lance-c#73 Problem Summary: Refresh the bundled Lance-C PR apache#73 patch from head a39bc8b08cd1367b7445cb9acf3309faa1f3d610 to a4c71309ddb76ad79808e3e8f7797bcd9bfc174a. The updated cache implementation extends coverage to whole-object and single-range reads while preserving the existing v0.1.9 plus PR apache#73 then PR apache#74 patch order. None - Test: No need to test (the refreshed PR apache#73 and existing PR apache#74 patches apply in order to a clean v0.1.9 archive with zero fuzz and no rejects or backup files; compilation and tests were not requested) - Behavior changed: Yes. The bundled PR apache#73 data-cache patch is refreshed to its latest upstream head. - Does this need documentation: No

add case [chore](lance) Upgrade Lance-C to v0.1.8 Issue Number: None Related PR: lance-format/lance-c#69 Problem Summary: Lance-C v0.1.8 now contains the upstream changes previously carried by the v0.1.7 PR apache#64 and runtime-filter patches. Upgrade the bundled source archive and checksum, remove both obsolete v0.1.7 patches, and retain only the current two-commit snapshot of upstream PR apache#69 for prepared FTS INDEX_ONLY plan scoping and explicit _rowid behavior. Upgrade the bundled Lance-C dependency to v0.1.8 and apply upstream PR apache#69. - Test: Manual test - Verified the official archive MD5, applied PR apache#69 to a clean v0.1.8 archive with zero fuzz, compared every tracked file byte-for-byte with PR head 6f0fae4, checked shell syntax, Rustfmt, stale references, and upstream CI status - Behavior changed: Yes. Prepared FTS INDEX_ONLY scans use the PR apache#69 plan rewrite without scanner-wide fast_search - Does this need documentation: No add fts more [chore](lance) Upgrade Lance-C to v0.1.9 Issue Number: None Related PR: lance-format/lance-c#73, lance-format/lance-c#74 Problem Summary: Pick the Lance-C third-party state from commit 417cd56 into the branch-4.1 Lance FTS branch. Upgrade the bundled release from v0.1.8 to v0.1.9, replace the obsolete PR apache#69 patch with PR apache#73, and rename the existing byte-identical PR apache#74 patch for the v0.1.9 base. Keep the branch-specific third-party changes outside the Lance-C section unchanged. None - Test: No need to test (verified shell syntax, patch blob equality, stale-reference removal, and scoped staged paths; compilation and tests were not requested) - Behavior changed: Yes. The bundled Lance-C version and patch baseline are updated. - Does this need documentation: No

zhangstar333 marked this pull request as ready for review September 3, 2026 05:02