Skip to content

Commit b96e41e

Browse files
authored
perf: Lazily load fast fields dictionaries. (#2842)
## What Lazily load fast field dictionaries from buffers: see paradedb/tantivy#55 ## Why A customer reported slower-than-expected paging on a string/uuid column. 85% of the time for that query was being spent in _opening_ a fast fields string/bytes column, with a large fraction of that time spent fully consuming the column's `Dictionary`. ## Tests See the attached benchmark results: * [`docs` dataset](#2842 (review)) * No regressions. * 2x faster for `top_n-score` * 1.4x faster for `highlighting` * [`logs` dataset](#2842 (review)) * No regressions. * 4.5x faster for `paging-string-max` * 1.7x faster for `paging-string-median` * 1.6x faster for `paging-string-min` The `paging-string-*` benchmarks were added in #2834 to highlight this particular issue.
1 parent 5191c6a commit b96e41e

2 files changed

Lines changed: 11 additions & 11 deletions

File tree

Cargo.lock

Lines changed: 9 additions & 9 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ lto = "thin"
2323
codegen-units = 32
2424

2525
[workspace.dependencies]
26-
tantivy = { git = "https://github.com/paradedb/tantivy.git", package = "tantivy", rev = "d37e61e6b8ddb105cbdc3ab391633ca108de6b73", features = [
26+
tantivy = { git = "https://github.com/paradedb/tantivy.git", package = "tantivy", rev = "16740add9447330b133348448ce4988edcc8174f", features = [
2727
"quickwit", # for sstable support
2828
"stopwords",
2929
"lz4-compression",
@@ -34,4 +34,4 @@ tantivy-jieba = "0.11.0"
3434

3535
[patch.crates-io]
3636
rust_icu_sys = { git = "https://github.com/google/rust_icu.git", rev = "53e98c8" }
37-
tantivy-tokenizer-api = { git = "https://github.com/paradedb/tantivy.git", package = "tantivy-tokenizer-api", rev = "d37e61e6b8ddb105cbdc3ab391633ca108de6b73" }
37+
tantivy-tokenizer-api = { git = "https://github.com/paradedb/tantivy.git", package = "tantivy-tokenizer-api", rev = "16740add9447330b133348448ce4988edcc8174f" }

0 commit comments

Comments
 (0)