enqueueLinks unprocessedRequests return value is always an empty array

[seho0808]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/core

Issue description

1. Run enqueueLinks function within any crawler
2. unprocessedRequests is always an empty array

// within any crawler
const { processedRequests, unprocessedRequests } = await enqueueLinks({...})
// you would obviously expect unprocessedRequests to hold something at some case

// the return value in the source code (`enqueue_links.ts`)
export async function enqueueLinks(
    options: SetRequired<EnqueueLinksOptions, 'requestQueue' | 'urls'>,
): Promise<BatchAddRequestsResult> {
  // ...
  return { processedRequests: addedRequests, unprocessedRequests: [] }; // seems to be hard coded to be empty
}  

is this an intended behavior?

Code sample

const { processedRequests, unprocessedRequests } = await enqueueLinks({
        globs: sameDomainPatterns(this.domain),
        exclude: staticFilePatterns,
      });
// unprocessedRequests is never a filled array.

Package version

[email protected]

Node.js version

v20.15.0

Operating system

macos Sequoia 15.3.2

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[vladfrangu]

Hey! Indeed, it is expected behavior! Usually you should never encounter unprocessedRequests [either because we handle it internally, or its just not returned from the API]

[seho0808]

ok my main question was why was this field open and mantained in the API spec?
when it is just an empty array..?
If it had to be that way, it is ok for me. I was just expecting to get control of unprocessedRequests to log or use in service logics.

Just to give you some background this was what I was doing (kind of derails from the main topic of the discussion though):

• I have two manager objects each with requestqueue but one is playwright crawler and the other is cheerio.
• i initially run cheerio and if it detects dynamic conditions, it queues to playwright manager's playwright crawler.
• when playwright's scrape is done for a single page, I want the entire valid urls to be passed back into cheerio queue managed by cheerio manager.
• so I was trying to utilize unprocessedRequests, but seems not to be working.
• I figured that even if I was able to use unprocessedRequests, I still don't want to push the requests to the actual queue.
• so what i did now is that I created a mimicking util function for only mimicking filter part of enqueueLinks.

  export function mimickedEnqueueLinksFilter(
    options: SetRequired<EnqueueLinksOptions, "urls">
  ) 
  

• now i am having trouble with getting the urls. within the crawlingcontext. => EDIT: found extractUrlsFromPage function so now im set for what I needed.

I just felt like the library was abstracted quite a lot without a overall code structure overview? I still really appreciate for putting this as open source + a lot of jsdoc documentations.

Just wanted to let you know about my use case and my concerns. thanks for a great library!