[bug] a few logic and UI issues with new Discover feature

[thuthancs]

I saw a new feature on OP, so gotta try it out :) Discovered a few bugs and UI issues:

• The Explore button did not return the expected result. The first screenshot shows the results when I searched using Explore mode. The second screenshot shows the results when I searched using Scholarly mode. Haven't read the code, but I assume the logic has been flipped (i.e., Scholarly mode is the agentic search).

• The selection bar is cut off, and I could only see the arXiv option when I zoomed out the main screen

• It's interesting that Attention Is All You Need original paper is not even displayed when I searched for the exact phrase: "attention is all you need"

[sabaimran]

Hey @thuthancs ! Thanks for checking it out proactively, was just about to send it your way for feedback haha.

1. The explore issue is actually a backend config problem - I didn't update some env variables correctly, will address it.

2. Good find! We should make that selection filter scrollable.

3. Ah, this is a good point. This is one of those cases where the decompose logic within the discover endpoint ends up adding too much noise to the original query. Will need to play with the logic, but this is a good base case that should be satisfied.

Def feel free to tackle 2 and 3 if you're interested. 3 might demand a little more on taste and vibes - the part to focus on is the decompose prompt passed to the LLM in the discover backend logic. Lmk! :)

[thuthancs]

thank you @sabaimran, for getting back to me so quickly!! i would be very happy to take on 2 & 3 :)

[thuthancs]

hello @sabaimran, a few points regarding the agentic search feature that I'd love to discuss with you before the actual implementation:

1. high variance in output response: right now, the results vary significantly with the same prompts. I know this is inevitable because the responses aren't deterministic, but I'm wondering whether we should set a lower temperature for the model so we get more consistent and relevant search results.

2. caching: in relation to (1), if the user uses the exact same search keyword(s)/phrase(s)/sentence(s), the results should be cached so that they return the same search results. this, however, raises another question: what if the search results are not exhaustive enough on the first try, and we want the system to generate different results on the second try with the same prompt?

w.r.t. points (1) and (2), i think to decide on what kind of improvements to implement, the key question is:

what kind of search results are useful to users? and how do we define usefulness?

as a user of OP, a few metrics i use to define whether the search results are good are:

• how many citations does a paper have?: this serves as a proxy for test-of-time and the impact of the paper
• which journals do the research papers publish in?: for CS researchers, those are IEEE, ACM and arXiv but for researchers of a different field, this might be different for them.
• what is the key conclusion/abstract of the paper?: if you take a look at the screenshot right below, the information displayed (apart from the research title) is not that helpful because it doesn't give me a reason why i would want to read this specific paper. i provided an example of perplexity's search results, which i think is pretty useful given the information that is displayed to me.

3. result display: as a user, I want to have an overview of what subqueries are being generated and the number of results associated with each of the subqueries before I take a look at the search results. right now, i have to scroll all the way down to see all the subqueries + results. my suggestion is to make each subquery a toggle (see screenshot below).

let me know what you think! thank you so much :)

[sabaimran]

1. This is an interesting point. I would say high variance isn't inherently an issue for this type of search and retrieval task, but it might be beneficial for us to further stabilize & guide the outputs with a more structured prompt. I wouldn't manipulate the temperature. I'd focus more on further refinement of the prompt for generating subqueries that adheres to the principles we outline. Sometimes this means adding redundancy to the instructions. Or perhaps adding examples, counterexamples, structure guidelines.
2. The LLM request does have some automatic caching that should kick in, but these prompt sizes are trivial. I think generally the maintenance overhead for retaining cache freshness isn't particularly important when the cost (latency and $) for queries to Exa/OA is relatively low. Plus, our deployed environment uses several backend containers, so keeping the cache in sync would require persistent storage. In short, I think it's not good ROI :)
3. Excellent suggestion on design! Agreed this is an improvement. I would just ensure they default open.

Re: utility at a glance - I totally agree. OA's API provides this relevant metadata, which is why I've set that as the default response (in scholarly mode). Exa is surprisingly lacking. We could leverage its automated AI summary feature, but my reservation is that this would add a big hit to latency. Using more structured retrievals via websets is also cost and latency prohibitive. And it doesn't give us number of citations anyhow.

There could be an option to do on the fly enrichment with the OA API if we find matches to the Exa papers, but I haven't fully evaluated this. Did you have thoughts on good ways to add on that structured info even in the Explore mode? I agree with your feedback generally, just don't see a clear answer to this. Very open to get your thoughts.

I also evaluated firecrawl, but the results were less useful for research publications. Semantic scholar is very promising, but has overly aggressive rate limiting that does not make it usable in prod.

[thuthancs]

thank you for your response, saba! i will give it some more thoughts and will reply with ideas/suggestions EOD Friday PST :)