fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885)#2944
fix(crawler): anonymous Reddit fallback via JS-challenge solve (#2885)#2944beaglemoo wants to merge 5 commits into
Conversation
…eep-app#2885) Reddit now blocks unauthenticated .json metadata requests. PR karakeep-app#2887 adds an OAuth client-credentials path, but Reddit closed self-serve API app creation, so many self-hosters cannot obtain credentials. This keeps the OAuth path (used when REDDIT_CLIENT_ID/SECRET are set) and adds an anonymous fallback: with a browser User-Agent, Reddit serves a lightweight JS challenge (solution = seed repeated twice) rather than a hard block. The metascraper warms up on the HTML post URL, solves the challenge, submits the hidden form to earn clearance cookies, and reuses them for the .json request. Cookies are threaded manually since fetchWithProxy (node-fetch) has no cookie jar. If the challenge format changes it falls back to the existing DOM scraping.
Greptile SummaryThis PR adds a Reddit metadata fallback for crawls without usable API credentials. The main changes are:
Confidence Score: 4/5This is close, but one fallback path should be fixed before merging.
apps/workers/metascraper-plugins/metascraper-reddit.ts Important Files Changed
Prompt To Fix All With AIFix the following 1 code review issue. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 1
apps/workers/metascraper-plugins/metascraper-reddit.ts:305
**OAuth errors skip fallback**
When Reddit returns a non-OK OAuth response other than 401 or 403, such as a 429 rate limit or transient 5xx from `oauth.reddit.com`, this branch returns that response instead of trying the anonymous challenge path. The cached token is left in place, so later Reddit crawls keep using the same OAuth path until local token expiry and continue returning failed metadata even though anonymous crawling may work.
Reviews (2): Last reviewed commit: "feat: include top Reddit comments in cra..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- Resolve the challenge form action with new URL(action, base) so absolute or protocol-relative actions no longer build a malformed submit URL. - On a 401/403 from the OAuth path, drop the cached token and fall through to the anonymous challenge path instead of failing the crawl.
|
Thanks for the review. Addressed 2 of the 3 in the latest commit:
On redirect cookies dropped: this is a real limitation but I've left it as-is deliberately. Capturing per-hop |
If the JSON body fetch fails (e.g. Reddit rate-limits us), returning no readable content let the generic readability pass scrape the rendered page, which on Reddit captures the cookie-consent banner as the article body and poisoned tagging/summaries. Fall back to the DOM post title instead so the stored content stays meaningful and a later re-crawl can recover the body.
Many Reddit posts (link/discussion posts) have an empty body, so the useful context lives in the comments. Parse the comments listing and fold the top ~10 comments (by score, with author) into readableContentHtml after the post body, so the reader view and the tagger get real content instead of just the title. HTML is sanitized downstream by DOMPurify.
| // A rejected token (revoked, early-expired, or insufficient scope) should | ||
| // not fail the crawl - drop the cached token and fall through to the | ||
| // anonymous challenge path, which works without credentials. | ||
| if (response.status !== 401 && response.status !== 403) { |
There was a problem hiding this comment.
When Reddit returns a non-OK OAuth response other than 401 or 403, such as a 429 rate limit or transient 5xx from oauth.reddit.com, this branch returns that response instead of trying the anonymous challenge path. The cached token is left in place, so later Reddit crawls keep using the same OAuth path until local token expiry and continue returning failed metadata even though anonymous crawling may work.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/workers/metascraper-plugins/metascraper-reddit.ts
Line: 305
Comment:
**OAuth errors skip fallback**
When Reddit returns a non-OK OAuth response other than 401 or 403, such as a 429 rate limit or transient 5xx from `oauth.reddit.com`, this branch returns that response instead of trying the anonymous challenge path. The cached token is left in place, so later Reddit crawls keep using the same OAuth path until local token expiry and continue returning failed metadata even though anonymous crawling may work.
How can I resolve this? If you propose a fix, please make it concise.The OAuth path only fell back to the anonymous challenge on 401/403, so a 429 rate limit or transient 5xx from oauth.reddit.com returned the failing response and kept reusing the cached token. Fall back on any non-OK status; only drop the cached token on 401/403 (a rate limit or server error does not mean the token is invalid).
|
Good catch on the OAuth fallback - fixed in the latest commit. The OAuth path now falls back to the anonymous challenge on any non-OK response (including 429 rate limits and 5xx), not just 401/403. The cached token is only dropped on 401/403, since a rate limit or transient server error doesn't mean the token itself is invalid. |
|
Hi @beaglemoo, looking forward to getting this merged. In the meantime, do you have a docker registry with this version on the web ? Would love to try it. |
|
Hey @ncareau — sure thing. I pushed an unofficial build of this PR to GHCR so you can try it before it merges: It's the standard all-in-one image, just built from this PR branch. Swap your existing karakeep A couple of caveats:
Note that AI tagging still needs your usual inference config; the fix itself just restores Reddit title/body/comments. Let me know how it goes! |
2 participants
Fixes #2885.
Reddit now blocks unauthenticated
.jsonmetadata requests, so Reddit bookmarks fail to crawl. #2887 adds an OAuth client-credentials path, but Reddit closed self-serve API app creation under its Responsible Builder Policy, so many self-hosters can't obtain credentials.This keeps the OAuth path (used when
REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRETare set) and adds an anonymous fallback: with a browser User-Agent, Reddit serves a lightweight JS challenge (solution = seedrepeated twice) instead of a hard block. The metascraper warms up on the HTML post URL, solves the challenge, submits the hidden form to earn clearance cookies, and reuses them for the.jsonrequest. Cookies are threaded manually sincefetchWithProxy(node-fetch) has no cookie jar. If the challenge format changes, it falls back to the existing DOM scraping.Verified end-to-end on a self-hosted instance: Reddit bookmarks crawl with title, author, subreddit, preview image, and content, with no credentials configured.
Note: this deliberately solves Reddit's anti-bot challenge, flagging that explicitly in case it raises ToS concerns for the project.