URL filters (-fsu) and content similarity help, but the headless graph can still fully explore many near-duplicate locations (product templates, calendar cells). That wastes browser time without new doors.
Add a simhash cluster budget on page states (ExplosionBudget / ExplosionHamming) so only the first N near-dupes are fully explored.
URL filters (-fsu) and content similarity help, but the headless graph can still fully explore many near-duplicate locations (product templates, calendar cells). That wastes browser time without new doors.
Add a simhash cluster budget on page states (ExplosionBudget / ExplosionHamming) so only the first N near-dupes are fully explored.