Add Serply web and Google Scholar retrievers - #2068
Open
googio wants to merge 1 commit into
Open
Conversation
Add two optional retrievers backed by the Serply SERP API, following the
existing per-provider pattern: RETRIEVER=serply for Google web results
(mirroring the serper retriever) and RETRIEVER=serply_scholar for Google
Scholar, which the project did not have. Both read SERPLY_API_KEY, normalize
to the standard {title, href, body} shape, and always return a list. No new
dependency and no change for users who do not opt in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Serply retrievers (web + Google Scholar)
This adds two optional retrievers backed by the Serply SERP API, following the existing per-provider pattern (one directory under
gpt_researcher/retrievers/, registered inretrievers/__init__.py,retrievers/utils.py, andactions/retriever.py):RETRIEVER=serply- Google web results, mirroring theserperretriever it sits next to.RETRIEVER=serply_scholar- Google Scholar results. GPT Researcher already has arXiv, Semantic Scholar, PubMed Central, and OpenAlex, but no Google Scholar, which spans publishers, theses, and citations across every field. This gives the research flow that coverage.Both read
SERPLY_API_KEYfrom the environment and normalize responses to the standard{title, href, body}shape. Like the sibling retrievers,search()always returns a list, neverNone.Why two
The repo already pairs same-vendor SERP wrappers (
serperandserpapi) and ships four academic retrievers, so both variants fit the existing surface. Each is small and self-contained; the file count comes from the per-provider scaffold applied twice, not from a large change to any one file.Optional and non-breaking
Nothing changes for anyone who does not set
RETRIEVER=serplyorserply_scholar. No new dependency is added (requestsis already required). Note: Serply sits behind Cloudflare, which rejects the defaultrequestsUser-Agent, so both retrievers send an explicit one.Testing
api.serply.io, confirming real web and scholar results normalize to the expected shape.tests/test_serply_returns_list.py, mirroringtest_serper_returns_list.py, covering the return-a-list error paths with the network mocked.Docs: added both entries to
docs/docs/gpt-researcher/search-engines/search-engines.md. More at serply.io/docs.Disclosure: I work with Serply. Happy to adjust scope, naming, or split the web and scholar retrievers into separate PRs if you prefer.