Skip to content

Bug: Missing duplicate URL detection when adding web pages #2490

@toyamarinyon

Description

@toyamarinyon

Bug: Missing duplicate URL detection when adding web pages

The useAddWebPages hook lacks duplicate URL detection before adding web pages to a node. Unlike useUploadFile which properly tracks duplicates using a batchSeen Set and checks against existing files (lines 38-50 of use-upload-file.ts), this hook allows the same URL to be added multiple times. The loop at line 51 immediately creates and adds a new WebPage entry for each URL without checking if the URL already exists in node.content.webpages or was already processed in the current batch. This causes duplicate entries, unnecessary network requests, and potential data inconsistencies.

Fix in Cursor Fix in Web

Originally posted by @cursor[bot] in #2486 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions