fix: dedup layer for scrape_batch + field filter (197 tests, 96% pass)#139
Closed
SuarezPM wants to merge 5 commits into
Closed
fix: dedup layer for scrape_batch + field filter (197 tests, 96% pass)#139SuarezPM wants to merge 5 commits into
SuarezPM wants to merge 5 commits into
Conversation
…arch_engine_batch - Add context_cache.js: SHA-256 prefix+length fingerprint for batch dedup - Fix prefix collision bug: now includes content.length in hash - Fix search_engine_batch field filtering for Google organic results - scrape_batch: deduplicate param (default true), format param, fields param - search_engine_batch: fields param for token-efficient responses - All 6 unit tests pass Invariant INV-CF-1: no content block appears twice in batch output DOI: https://doi.org/10.5281/zenodo.20277875
SuarezPM
force-pushed
the
feat/context-forge-nexus-integration
branch
from
5d0e584 to
6a89b6d
Compare
SuarezPM
commented
May 23, 2026
SuarezPM
left a comment
SuarezPM
left a comment
Author
There was a problem hiding this comment.
feat: add dedup layer (INV-CF-1) to scrape_batch + field filter to search_engine_batch
- Add context_cache.js: SHA-256 prefix+length fingerprint for batch dedup
- Fix prefix collision bug: now includes content.length in hash
- Fix search_engine_batch field filtering for Google organic results
- scrape_batch: deduplicate param (default true), format param, fields param
- search_engine_batch: fields param for token-efficient responses
- All 6 unit tests pass
- Add backward-compatible response: flat array by default, opt-in metrics wrapper - Fix hash collision: now uses prefix(2048) + suffix(64) to avoid false duplicates - Add 9 unit tests (was 6): edge cases for collision, length, boundary conditions - Add 8 disaster recovery tests: empty/null/long content, filterFields null handling - Rename forge_metrics to metrics, remove ContextForge branding from tool description Fixes: - False positive dedup when pages share header but differ in body - Breaking API change in response structure (now backward compatible) - Missing edge case coverage Tests: 17 passed, 0 failed Real API verified: 66.7% dedup ratio on example.com
SuarezPM
changed the title
- buildForgeMetrics → buildBatchMetrics - Remove forge_version, invariant, doi fields (ContextForge-specific) - Remove ContextForge from deduplication description - Update tests to match
Hash collision fix:
- Use prefix(2048) + middle(256) + suffix(256) for content > 2048 chars
- Short content uses full hash
- Add clear() method for long-running processes
New test files:
- test_real_websites.js: 10 tests with real HTTP calls (lablab.ai, brightdata.com, wikipedia.org)
- test_filter_fields.js: 20 edge case tests for filterFields
Fixed behavior:
- filterFields now returns {} for null/undefined/non-object items
Total tests: 47 passing
- test_context_cache.js: 9
- test_dedup_edge_cases.js: 8
- test_real_websites.js: 10
- test_filter_fields.js: 20
SuarezPM
changed the title
SuarezPM
changed the title
150 tests covering: - 30 single-page load tests - 40 dedup correctness tests - 20 hash consistency tests - 20 cross-domain isolation tests - 20 edge cases (unicode, boundaries, special chars) - 20 error handling tests Sites tested: github, stackoverflow, wikipedia, medium, reddit, hackernews, npmjs, pypi, crates.io, docs.python.org, mozilla.org, rust-lang.org, golang.org, python.org, youtube, vimeo, amazon, arstechnica, phoronix, stackexchange, wikimedia, twitter, mastodon, twitch, ebay, openai, httpbin.org, and more Results: 144/150 passed (96%) Failures due to site blocking (anti-bot) not code bugs. Also fixes JSON response handling in scrape() for httpbin endpoints.
SuarezPM
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deduplication layer for batch scraping that removes duplicate content blocks across URLs, reducing token usage in LLM pipelines.
Problem
When scraping multiple URLs from the same domain, pages share nav/header/footer HTML. Without dedup, identical content is returned N times, wasting tokens.
Solution
scrape_batch new params:
deduplicate(default: true) — removes duplicate content blocks via SHA-256 fingerprintinginclude_metrics(default: false) — opt-in for{results: [...], metrics: {...}}responsefields— filter response to specific top-level fieldsformat— markdown (default) or rawsearch_engine_batch:
fields— filter result.organic array to requested keysHash Algorithm
This captures shared headers/footers while correctly distinguishing pages with same structure but different body content.
Test Suite
Real API Verification
Backward Compatibility
Default behavior (no params or
include_metrics: false) returns flat array — no breaking changes to existing consumers.Prior Art
Deduplication strategy inspired by ContextForge (https://github.com/SuarezPM/Apohara_Context_Forge, DOI: 10.5281/zenodo.20277875).