Search overhaul: matching accuracy, ranking, results UI, jump to match - #10633
Open
perfectra1n wants to merge 40 commits into
Open
Search overhaul: matching accuracy, ranking, results UI, jump to match#10633perfectra1n wants to merge 40 commits into
perfectra1n wants to merge 40 commits into
Conversation
Restore the documented label-equality semantics for the = and != operators: they now compare the whole normalized value for strict equality instead of word/phrase matching. #capital=Vienna matches label value "vienna" but no longer "Vienna Austria". The former word/phrase-match logic moves to a new internal "word=" operator (not user-typable, absent from parse's OPERATORS) which the leading-"=" fulltext title comparison now uses so its user-visible behavior is unchanged. Refs #9422
text/html notes previously lost their link previews and reference links to search: stripTags deleted the data-* metadata of <section class="link-embed"> and <span class="link-mention"> elements, and reference-link anchors carry stale/empty text (the title is resolved at render time). preprocessContent now, before stripping tags, extracts data-url/title/ description/site-name from link previews (entity-decoded) and appends the resolved title of each distinct internal-link target via an injected resolver, keeping the preprocessor pure (no becca import). The one caller passes becca.notes[id]?.title. Resolved titles become ordinary content words that Task 2's tier classifier ranks automatically. Refs #10616
Add HighlightedTokenInfo to commons and a regexTokens set + getHighlightedTokenInfos() on SearchContext, tagging %= operator values (parseLabel and note content fulltext) as regex tokens. searchFromNote's result gains an additive highlightedTokenInfos array (highlightedTokens stays untouched — public scripting API). Make the snippet/highlight machinery regex-aware and consolidate its position math on a new length-preserving core normalizer (normalizePreservingLength) so match offsets found on normalized text map 1:1 onto the original when slicing/inserting markers — no index drift or mid-marker splits across ligatures (ß, æ) and NFD-decomposed content. Regex tokens compile to /token/gi (invalid patterns skipped) with a per-field wrap cap; plain tokens behave as before.
Extract buildSearchResultDetails(results, searchContext) — the snippet extraction + highlight + wire-shape map duplicated between the quickSearch route and searchNotesForAutocomplete — and rebuild both call sites on it. It runs the regex-aware snippet/highlight machinery with the context's structured highlight tokens and now also includes noteId in the wire shape (SearchResultDetails in commons). Split searchFromNote into searchFromNoteWithContext(note), which returns the raw SearchResults plus the producing SearchContext (null for script-based searches), with searchFromNote reimplemented on top. Public behavior unchanged.
Add POST /api/search-note/:noteId/result-details, which re-runs a saved search and returns snippet + highlight details only for a requested page of result noteIds (max 100), in requested order. Requested ids are filtered against the actual result set so the endpoint can't be used as a snippet oracle for arbitrary notes; script-based searches return titles/icons with empty snippets and no token infos. Stateless (re-runs per request); a future LRU keyed by (search noteId + searchString) is noted but intentionally not built. Wire types SearchResultDetailsRequest/Response live in commons alongside HighlightedTokenInfo/SearchResultDetails.
Add a synced searchResultsPageSize option (default 20) for the full-search results view's page size: interface entry in commons, default in trilium-core options_init, and the ALLOWED_OPTIONS allowlist so the API accepts changes. Inert until the client consumes it.
useImperativeSearchHighlighlighting now accepts structured HighlightedTokenInfo tokens (plain/regex) alongside legacy string[]. Plain tokens use mark.js's term API with diacritics:true so an unaccented query like "ktory" still highlights "ktorý" in result previews (#10616); regex tokens (from %= searches) are compiled and applied via markRegExp instead of being regex-escaped as literals, so patterns like 'pat.*' actually highlight matches (#5332). Plumbs highlightedTokenInfos from the search-note endpoint through FNote and froca (falling back to mapping the legacy highlightedTokens as plain when absent), and widens the highlightedTokens prop type through ViewModeProps/NoteLink/NoteList/the legacy list-grid view/the dashboard view so the wider type compiles end to end. All widening is additive; existing string[] and null/undefined callers are unchanged.
Add an additive `defaultPageSize` parameter to `usePagination` so the search results view can drive paging from the synced `searchResultsPageSize` option while an explicit `#pageSize` label still wins. Clamp the current page back onto the last valid page when the page size grows (or the result set shrinks) so the slice can't yield an empty page. Existing list/grid callers are unaffected.
Render search-note results as Google-style snippet cards for the "list" view type (issues #5667, #6225); other view types keep the legacy SearchNoteList path so the ViewTypeSwitcher still works. Each card shows the live title/icon from froca, a server-built highlighted snippet, an ancestor-path breadcrumb, and outline badges for matched attributes, linking by bare noteId. Snippet/highlight details are fetched lazily one page at a time via `useSearchResultDetails`, with a sequence-ref stale-response guard and a `searchRefreshed` refetch. The collection bar gains an always-visible result count and a 10/20/50/100 page-size selector bound to `searchResultsPageSize`. Also pass `highlightedTokenInfos ?? highlightedTokens` from `renderCollection` so included/rendered search collections get regex-aware highlighting.
Quick-search dropdown highlight <b> tags now point at --note-list-view-search-result-highlight-background/-color, the same vars the snippet-card search results view uses, with fallbacks that preserve the previous look on themes that don't define them. Also adds highlightedTokens to the quickSearch route response (it was the one snippet-response consumer Task 4's shared-builder refactor missed) so the dropdown can stash the tokens for jump-to-match.
note_autocomplete's suggestion template can now render the server's highlightedContentSnippet, gated behind a new showContentSnippets option (default false) so link dialogs, relation editors and @-mentions stay compact. Only the jump-to-note dialog opts in. The jump-to-note dialog also carries the typed search string into navigation as viewScope.searchTerms when a note is opened from a text search (not the command palette, not an untyped recent-notes pick), via a locally-widened ViewScope type — same forward-compatible pattern SearchResultCard.tsx uses ahead of Task 8 promoting searchTerms onto ViewScope proper.
actualText.current (the gate for viewScope.searchTerms in jump_to_note.tsx) only updated on the native "input" DOM event, but several paths change the autocomplete's value programmatically without firing one: the mid-session "show recent notes" button, the "clear text" button (both in note_autocomplete.ts), and the >120s-later dialog reopen resetting to recent-notes (jump_to_note.tsx's openDialog). Each left the ref holding a previous session's typed query, so picking an untyped/recent-notes suggestion could still attach searchTerms from an earlier search. - note_autocomplete.ts: clearText/setText/showRecentNotes/showAllCommands now re-trigger "input" after setting the value programmatically, so any consumer tracking the live query (not just this dialog) stays accurate. - jump_to_note.tsx: openDialog syncs actualText.current with the freshly-computed initialText immediately, since the "recent-notes" branch bypasses NoteAutocomplete's text-prop path entirely. - Extracted the gate itself into a pure, exported deriveSearchViewScope() and added a focused spec (jump_to_note.spec.ts) covering typed/blank/ command-mode/cleared-after-typed inputs.
Promote `searchTerms` from a temporary local intersection type onto the real ViewScope interface. calculateHash serializes non-empty search terms as a `searchTerms` hash param (double-encoded so commas inside a token can't be confused with the token separator); parseNavigationStateFromUrl decodes and splits it back into an array, dropping any malformed percent-encoded token instead of throwing. SearchResultCard and jump_to_note now use the real ViewScope field and drop their task-8-pending local aliases (SearchResultViewScope / JumpToNoteViewScope).
…bookmarks Add expandCollapsedAncestors(el), which opens every closed <details> ancestor of an element (CKEditor collapsibles and raw imported <details> alike). Wire it in before every scroll-to-match/bookmark point that can land inside a collapsed block, since scrollIntoView is a no-op for content with no layout box: - find_in_html.ts jumpTo() (read-only text/render/doc find, incl. find-next cycling) — fixes Ctrl+F being unable to reveal matches inside collapsed blocks (#10616). - find_in_text.ts (editable CKEditor) after "find" and after "findNext"/"findPrevious", re-scrolling explicitly afterwards since CKEditor's own debounced scroll can't reach a still-hidden range. - ReadOnlyText.tsx / EditableText.tsx bookmark-anchor scroll (?bookmark=...).
Extend the findInText command with an optional searchTerms payload. When present, FindWidget seeds the find bar with the first token, jumps to the first match, and (read-only HTML notes only) highlights the remaining tokens with a muted secondary style so every matched term is visible while Enter/F3 keeps cycling only the seed matches. The bar opens whenever there is at least one match so non-Electron users (no Ctrl+F) can cycle and dismiss it; with zero matches nothing is shown and no marks are left behind. Route the command by explicit ntxId when targeted (the seeded flow carries it) so only the matching split's widget responds, falling back to the active context for manual Ctrl+F. performFind now returns the match counts. Groundwork for #3098 / #10616.
Add consumeSearchTerms: a one-shot consumer that reads viewScope.searchTerms (carried in on note open, mirroring viewScope.bookmark), clears them, and fires a seeded findInText deferred to the next animation frame so it runs after the noteSwitched dispatch and can't be race-closed by FindWidget's own noteSwitched -> closeSearch. Wire it into the text and code type widgets at their content-ready point (the [blob] effect for read-only, onContentChange for editable). A shared useSearchTermsConsumer hook covers the same-note re-click case, where setNote proceeds on viewScope inequality and fires noteSwitched without reloading the blob; it is guarded to the note the widget already shows so a genuine switch is left to the newly mounted widget's content-ready path. Closes part of #3098 / #10616.
Build quick-search result links with calculateHash, attaching the last search's highlighted tokens as viewScope.searchTerms (both the snippet items and the froca-fallback createLink path). Opening a result now jumps to the first match and pre-fills the find bar. Tokens are omitted when empty, so links degrade to a plain #notePath.
Task 11 full integration pass over the search-overhaul branch: - Fix a duplicate @triliumnext/commons import (no-duplicate-imports) in useSearchResultDetails.spec.tsx. - Fix an unsorted import block in find_in_text.ts. - Reword three stale "Task N" comments (SearchResultsList.spec.tsx, quick_search.ts, build_comparator.ts) to be self-contained now that the referenced tasks have landed. No behavioral changes; typecheck, the full client suite, and the touched search spec all still pass.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
dosubot
Bot
added
the
size:XL
Contributor
|
🖥️ App preview is ready! 🔗 Preview URL: https://pr-10633.trilium-app.pages.dev ✅ All checks passed This preview will be updated automatically with new commits. |
Bundle ReportChanges will increase total bundle size by 20.05kB (0.02%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: standalone-esmAssets Changed:
view changes for bundle: client-esmAssets Changed:
Files in
Files in
Files in
Files in
Files in
Files in
|
Contributor
|
📚 Documentation preview is ready! 🔗 Preview URL: https://pr-10633.trilium-docs.pages.dev ✅ All checks passed This preview will be updated automatically with new commits. |
Contributor
Greptile SummaryThis is a large search overhaul that fixes matching accuracy (punctuation-aware exact matching, AUTO-scaled fuzzy distances,
Confidence Score: 5/5Safe to merge. The matching, ranking, and results-UI changes are thoroughly tested (456 core + 2307 client tests, 32 doc examples validated). The strict-equality restoration for attribute = is a deliberate, documented behavior change with a regression test pinning it. The core search logic is well-covered by unit tests including new specs for every major changed path. All findings are minor ranking-accuracy and code-organization observations that do not affect correctness. match_quality.ts for the betterQuality inOrder tie-breaking; note_content_fulltext_preprocessor.ts for the module-level global regex lastIndex pattern. Important Files Changed
Reviews (4): Last reviewed commit: "Merge origin/main into feat/search-overh..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…nst navigation races Review feedback on #10633: the rAF-deferred findInText could target a newer note if the tab navigated within the frame (now aborted via viewScope identity), and overlapping froca.getNotes loads in usePagination could resolve out of order and overwrite the current page (now cancelled on supersession; out-of-range slices are skipped while the page clamp settles).
ReadOnlyText's bookmark effect ([blob] deps) also fired on mount, while the blob was still loading and the content container was empty, and unconditionally cleared viewScope.bookmark — so the post-load run had nothing left to reveal and the collapsible expansion never happened on a cold open. Extract the reveal-and-consume logic into services/bookmark_jump.ts (consumeBookmark), shared by ReadOnlyText and EditableText: a missing container leaves the bookmark unconsumed for the next pass, a dangling anchor is consumed without scrolling, and the target is found by exact id comparison instead of interpolating user text into a CSS attribute selector (ids with quotes/brackets used to break the selector). ReadOnlyText's effect is now gated on the blob being present.
Reconciles the overlap with the collapsible-blocks work that landed in main (#10635, #10628) while this branch was in review: - services/collapsible.ts (main's expandAncestorDetails) is now the single collapsible-expansion helper; this branch's duplicate services/collapsibles.ts is removed and bookmark_jump/find_in_html use main's module. The param is widened to Element for the bookmark path, and the mixed open/closed chain and self-is-details cases from the removed spec are ported into collapsible.spec.ts. - find_in_text's revealSelectedFindResult is dropped: the collapsible plugin now owns find-reveal in the editable editor (transient editing-view open), and a direct DOM toggle would be adopted into the persisted model — a plain Ctrl+F must not rewrite the note's saved open/closed layout. - useImperativeSearchHighlighlighting keeps this branch's token-info rewrite (diacritics, CJK, regex tokens with caps) and gains main's post-mark pass that expands <details> around highlighted results; specs merged accordingly. - ReadOnlyText.spec.tsx combines main's #10575 suite with this branch's ?bookmark= consume-race regression test, ported onto main's harness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This overhauls search matching, ranking, the results UI, and match navigation. It started from the complaint list in #10616 and picks up the related open issues along the way.
Closes #10616, closes #9426, closes #9422, closes #5667, closes #5332, closes #6225, closes #3098. Related: #6991 (this lands a heuristic ranking upgrade rather than BM25; all weights live in one table, which is where configurable weights would plug in later if we want them).
Matching fixes
=prefix) is now punctuation aware.=syncfinds a note containing(sync),sync,or"sync". Previously content was split on whitespace only, so any punctuation next to the word broke the match. Edge punctuation is stripped symmetrically from the query and the content, and underscores,+and inner apostrophes are kept, soc++,_privateandd'Artagnanstill work.syncfrom matchingSendandceckfrom matchingTech(both were 2 edits on a 4 character word), whilecombinefstill findscombined.~=and~*now work on properties and labels (note.title ~= boks,#author ~= tolkein). This was a lexer bug:~was always treated as a relation prefix, so~=never reached the parser even though the comparator supported it (Fuzzy search not working #9426).~*also keeps fragment matching, so~* progrfinds "programming".data-url/data-title/data-description/data-site-nameattributes of link embeds and link mentions are indexed too.Ranking
Content matches previously contributed nothing to the score; only title and path did. That is why a note whose title merely contained
asynccould outrank a note whose body had the exact wordsync, and why scattered words beat an exact phrase. Content matches are now classified into tiers (exact phrase, all words in proximity with an in-order bonus, exact word, prefix, substring, fuzzy) and scored accordingly. The weights are chosen so an exact title match always dominates, and that invariant is enforced by a test.Behavior change reviewers should look at: attribute
=is strict again#capital=Viennaused to match a label value of "Vienna Austria" because the=comparator drifted into word matching at some point. That contradicts the documented semantics (#9422). This branch restores strict full-value equality (case and diacritic insensitive) for attribute and property=/!=, and moves the word-match behavior to an internal operator used only by the leading=fulltext prefix, whose user-visible behavior is unchanged.Who this affects: ETAPI searches, saved searches (and therefore bulk search-and-execute), the script APIs, share search, and
getNotesWithLabel. All of them run withfuzzyAttributeSearch: falseand silently inherited the word-match drift. An ETAPI regression test now pins the strict behavior. Quick search and autocomplete are unaffected (they run with relaxed attribute matching, which is now documented and covered by a test as well).Results UI
POST /api/search-note/:noteId/result-detailsendpoint (max 100 ids per request, ids are validated against the actual result set). Other view types for search notes are untouched; the cards are just the "list" view.searchResultsPageSize(Settings for displaying search results #6225).ktoryhighlightsktorý), and regex searches (note.content %= '...') highlight their real matches instead of nothing ((Feature request) Improve search result highlighting #5332). Server snippet positions are computed on a guaranteed length-preserving normalization, which also let us drop thenormalize-stringsdependency.Jump to match (#3098)
Opening a result from the search page, quick search, or the jump-to-note dialog scrolls to the first match and pre-fills the find bar, so Enter/F3 cycles through matches. The terms travel in a new
viewScope.searchTermsthat round-trips through link hashes, so ctrl+click into a new tab works too. Collapsed<details>blocks (CKEditor collapsibles) are expanded automatically when a find match or a bookmark target sits inside one; previously Ctrl+F would count those matches but could not show them.Docs
Search.md and Quick search.md were rewritten around concrete examples, and every example in them (32 in total, including the pre-existing ones) is validated by a new
docs_examples.spec.tsthat runs the exact query strings against the real engine. Each test names the doc section it validates. This already paid off: it caught two pre-existing doc bugs (the negation example usednote.ancestor.title, which is not a recognized specifier, and quick search's description of=was simply wrong).Note for maintainers: these two files were edited directly as Markdown rather than through
pnpm edit-docs:edit-docs, so a later edit-docs round trip may reformat them. The examples and the spec are kept in lockstep, so a reformat should not change any example text.Testing
Manual checks before merging
These involve real-browser DOM behavior that the unit environment cannot cover:
?bookmark=link whose target sits inside a closed collapsible expands and scrolls to itFollow-ups, will file separately
%=regex patterns (current bounds: pattern length cap plus match count caps).