Sorter for all lines

[natecraddock]

Is your feature request related to a problem? Please describe.
Telescope supports custom sorters through extensions, but I am not aware of a way to use an external process for sorting all of the lines together. The readme (emphasis added) says

A Sorter is called by the Picker on each item returned by the Finder.

Which makes calling a separate process for scoring really inefficient.

Describe the solution you'd like
Support for custom Sorters that accept all lines from the finder, and then return the sorted lines. My use case is a custom fuzzy finder I use like fzf or fzy. I would expect to be able to send the lines from the finder (like file paths, commands, etc.) to my fuzzy finder in an external process over stdin which would process the lines and then output the filtered and sorted results on stdout.

Describe alternatives you've considered
I could rewrite the algorithm in C/Lua like the fzy- and fzf-native projects, but I would prefer to use my one codebase for all of my fuzzy finding if possible. I am not aware of other alternatives, but if one exists I am interested to hear it!

I do like to contribute to projects I benefit from, so if this request is in scope and not too crazy to implement I might attempt a PR for it :)

[fdschmidt93]

https://github.com/nvim-telescope/telescope-fzf-writer.nvim is pretty much what you are looking for I suppose. Not sure if it still works after all the async updates, I've never used it.

More generally on telescope performance: Sorting itself is very fast nvim-telescope/telescope-fzf-native.nvim#32 (that PR is also relevant for piping results across). I think telescope rather has two primary bottlenecks (TJ & Conni, please correct me :)

• Something what you propose: IIRC TJ worked on rewriting finders in Rust (not pushed) on stream - https://github.com/nvim-telescope/telescope-rs.nvim
• Entry manager to sort scored items (atm insertion sort): Give break: c rewrite #1270 a try and you'll see tremendous improvements.

Insertion sort is also under revision as it's needed to draw each result live. With #1491 we could move to different data structures and only sort entries periodically upon refresh which could have a tremendous impact.

I hope that sufficiently answers your question.

[natecraddock]

Thanks for the detailed response!

More generally on telescope performance ...

These are some cool ideas, thanks for sharing! I am not too concerned with performance though. I typically work on small to medium sized projects and telescope has never slowed me down (using fzf-native). The reason I want to integrate my custom fuzzy finder is because it has some behavioral adjustments that I find more intuitive compared to fzf or fzy.

https://github.com/nvim-telescope/telescope-fzf-writer.nvim is pretty much what you are looking for I suppose.

I will keep looking at this. From what I can see this would work, but it would require replacing all the finders I use if I understand correctly. That's why I was hoping to only create an extension for sorters.

[natecraddock]

The longer I think about this, the more I think the proper solution would be at the sorter level. That way an extension can set the sorter as default for all pickers.

I imagine that a sorter could indicate whether it works on one item at a time, or if it filters all lines. Then the finder's result processor could be designed to take all entries and score all at once, instead of scoring for the single entry. Looking through the code I think I could hack a proof of concept together, so I might try that soon.

I understand if this is out of scope for the project, though I would hope otherwise :)

[Conni2461]

If you open a PR i'll take a look at it, but i won't promise anything. If the maintenance burden of the patch is to high, for example clashing with our future plans like the fps mode or other stuff tj and i've talked about, then i won't accept the PR and you most likely have to maintain your own fork.

[natecraddock]

I took a closer look at this, and I think the cost of modifying telescope to sort all lines is greater than modifying my fuzzy finder to integrate with telescope, so I won't be needing this feature anymore.

I'll just close this now. :)