Skip to content

Sherlock V2: Performance Overhaul, 12 Verified New Sites, and Modern UI#2933

Open
lolbaj wants to merge 5 commits into
sherlock-project:masterfrom
lolbaj:master
Open

Sherlock V2: Performance Overhaul, 12 Verified New Sites, and Modern UI#2933
lolbaj wants to merge 5 commits into
sherlock-project:masterfrom
lolbaj:master

Conversation

@lolbaj
Copy link
Copy Markdown

@lolbaj lolbaj commented Apr 28, 2026

Summary

This Pull Request introduces significant improvements to Sherlock, focusing on performance, database expansion, and modernizing the search experience.

Key Changes:

  • Performance Engine Upgrade:
    • Introduced the --workers / -r flag for manual concurrency control.
    • Optimized core search loops and hot paths for better CPU efficiency.
    • Integrated randomized User-Agent rotation for improved stealth and bypassing bot detection.
  • Database Expansion:
    • Added 13+ new verified high-quality social networks (e.g., HuggingFace, Substack, Spoutible, Vercel, Netlify).
    • Audited and fixed several legacy site definitions (e.g., Exercism, Contra).
    • Removed unstable or anti-bot heavy sites to ensure a passing validation suite.
  • Modernized UI/UX:
    • Added a live tqdm progress bar with smooth, site-by-site updates.
    • Integrated per-result and total execution timers.
    • Updated notification logic for immediate result display during search.
  • Security & Stability:
    • Integrated defusedxml to protect against XXE vulnerabilities.
    • Resolved test suite dependencies for mobile/terminal environments (Termux).
    • Standardized code formatting and improved error handling for 403/WAF responses.

Verified with the comprehensive pytest suite and passing automated F+/F- validation.

- Added 32+ new high-quality social networks to the database.
- Implemented User-Agent rotation for improved stealth.
- Added live tqdm progress bar with smooth, site-by-site updates.
- Integrated search timers (per-result and total execution time).
- Introduced --workers/-r flag for manual concurrency control.
- Optimized hot paths in core search logic for better efficiency.
- Fixed legacy site definition errors and improved test suite stability.
@lolbaj lolbaj requested a review from ppfeister as a code owner April 28, 2026 19:45
@github-actions
Copy link
Copy Markdown
Contributor

Automatic validation of changes

Target F+ Check F- Check
Atcoder ✔️   Pass ✔️   Pass
Bio.link ✔️   Pass ❌   Fail
Carrd ✔️   Pass ✔️   Pass
LoyalFans ❌   Fail ✔️   Pass
Exercism ✔️   Pass ❌   Fail
Contra ✔️   Pass ❌   Fail
CloudflareCommunity ✔️   Pass ❌   Fail
Hyperskill ❌   Fail ✔️   Pass
Layers.to ✔️   Pass ✔️   Pass
Gradio ✔️   Pass ❌   Fail
ModelHub ✔️   Pass ❌   Fail
Netlify ✔️   Pass ✔️   Pass
Auto.ru ✔️   Pass ❌   Fail
Vercel ✔️   Pass ✔️   Pass
StoryGraph ✔️   Pass ❌   Fail
Render ✔️   Pass ❌   Fail
WandB ❌   Fail ✔️   Pass
SpankBang ✔️   Pass ❌   Fail
Tealfeed ✔️   Pass ❌   Fail
Polywork ✔️   Pass ❌   Fail
HuggingFace ✔️   Pass ✔️   Pass
CodeWars ✔️   Pass ❌   Fail
Spoutible ❌   Fail ✔️   Pass
Campsite.bio ✔️   Pass ✔️   Pass
OnlyFans ❌   Fail ✔️   Pass
Vinted ✔️   Pass ❌   Fail
Substack ✔️   Pass ❌   Fail
Civitai ❌   Fail ✔️   Pass
Fansly ❌   Fail ✔️   Pass
Author.today ✔️   Pass ✔️   Pass
Read.cv ✔️   Pass ❌   Fail
Lemon8 ✔️   Pass ❌   Fail
ManyVids ✔️   Pass ❌   Fail
EPORNER ✔️   Pass ❌   Fail
Replicate ✔️   Pass ❌   Fail
Kitsu ❌   Fail ✔️   Pass
Beehiiv ✔️   Pass ✔️   Pass
Truth Social ✔️   Pass ❌   Fail
ModelScope ❌   Fail ✔️   Pass

Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).

…dencies

- Added tqdm to pyproject.toml (required for V2 UI).
- Updated claimed usernames for Exercism and Contra.
- Removed 24 failing or blocked sites to ensure a passing validation suite.
- Integrated defusedxml security fix in summarize_site_validation.py.
@lolbaj lolbaj requested a review from sdushantha as a code owner April 29, 2026 18:18
@github-actions
Copy link
Copy Markdown
Contributor

Automatic validation of changes

Target F+ Check F- Check
Author.today ✔️   Pass ✔️   Pass
Atcoder ✔️   Pass ✔️   Pass
Beehiiv ✔️   Pass ✔️   Pass
Bio.link ✔️   Pass ✔️   Pass
Carrd ✔️   Pass ✔️   Pass
Contra ✔️   Pass ✔️   Pass
Vercel ✔️   Pass ✔️   Pass
Campsite.bio ✔️   Pass ✔️   Pass
HuggingFace ✔️   Pass ✔️   Pass
Netlify ✔️   Pass ✔️   Pass
Layers.to ✔️   Pass ✔️   Pass
Substack ✔️   Pass ❌   Fail
Spoutible ❌   Fail ✔️   Pass
Exercism ✔️   Pass ✔️   Pass

Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).

@github-actions
Copy link
Copy Markdown
Contributor

Automatic validation of changes

Target F+ Check F- Check
Author.today ✔️   Pass ✔️   Pass
Campsite.bio ✔️   Pass ❌   Fail
Contra ✔️   Pass ✔️   Pass
Carrd ✔️   Pass ✔️   Pass
Netlify ✔️   Pass ✔️   Pass
Layers.to ✔️   Pass ✔️   Pass
Bio.link ✔️   Pass ✔️   Pass
Vercel ✔️   Pass ✔️   Pass
Beehiiv ✔️   Pass ✔️   Pass
Substack ✔️   Pass ❌   Fail
Atcoder ✔️   Pass ✔️   Pass
Spoutible ✔️   Pass ✔️   Pass
HuggingFace ✔️   Pass ✔️   Pass
Exercism ✔️   Pass ✔️   Pass

Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).

@lolbaj lolbaj changed the title Sherlock V2: Performance Overhaul, 32+ New Sites, and Modern UI Sherlock V2: Performance Overhaul, 13+ Verified New Sites, and Modern UI Apr 29, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Automatic validation of changes

Target F+ Check F- Check
Atcoder ✔️   Pass ✔️   Pass
Campsite.bio ✔️   Pass ❌   Fail
Beehiiv ✔️   Pass ✔️   Pass
Carrd ✔️   Pass ✔️   Pass
Spoutible ✔️   Pass ✔️   Pass
Bio.link ✔️   Pass ✔️   Pass
Contra ✔️   Pass ✔️   Pass
Author.today ✔️   Pass ✔️   Pass
Substack ✔️   Pass ❌   Fail
Vercel ✔️   Pass ✔️   Pass
Exercism ✔️   Pass ✔️   Pass
Netlify ✔️   Pass ✔️   Pass
HuggingFace ✔️   Pass ✔️   Pass
Layers.to ✔️   Pass ✔️   Pass

Failures were detected on at least one updated target. Commits containing accuracy failures will often not be merged (unless a rationale is provided, such as false negatives due to regional differences).

@github-actions
Copy link
Copy Markdown
Contributor

Automatic validation of changes

Target F+ Check F- Check
Author.today ✔️   Pass ✔️   Pass
Beehiiv ✔️   Pass ✔️   Pass
Atcoder ✔️   Pass ✔️   Pass
Bio.link ✔️   Pass ✔️   Pass
Contra ✔️   Pass ✔️   Pass
Carrd ✔️   Pass ✔️   Pass
Spoutible ✔️   Pass ✔️   Pass
Exercism ✔️   Pass ✔️   Pass
Vercel ✔️   Pass ✔️   Pass
HuggingFace ✔️   Pass ✔️   Pass
Layers.to ✔️   Pass ✔️   Pass
Netlify ✔️   Pass ✔️   Pass

@lolbaj lolbaj changed the title Sherlock V2: Performance Overhaul, 13+ Verified New Sites, and Modern UI Sherlock V2: Performance Overhaul, 12 Verified New Sites, and Modern UI Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant