fix(enqueueLinks)!: Default strategy now SameDomain

[SalvadorN323]

Summary

This PR introduces a significant change to the default behavior of the enqueueLinks function, aligning it with more common crawling patterns and robustly handling common website redirections. The default enqueueStrategy has been changed from SameHostname to SameDomain.

Key Changes

• Code Change (Breaking Change):
  • The enqueueLinks function's default strategy is now EnqueueStrategy.SameDomain when no globs, regexps, or pseudoUrls are explicitly provided.
• Unit Test Updates:
  • Tests that previously expected SameHostname filtering by default now correctly assert SameDomain behavior (e.g., links across www. and apex domains are now included by default).
• Documentation Updates:
  • Updated "Adding more URLs" Guide: The guide now explicitly states that enqueueLinks defaults to SameDomain (including subdomains).

Breaking Change Details

The default enqueueLinks strategy has shifted from EnqueueStrategy.SameHostname to EnqueueStrategy.SameDomain.

Before:

options.strategy ??= EnqueueStrategy.SameHostname

After:

options.strategy ??= EnqueueStrategy.SameDomain

• Mitigation: To maintain the previous strict SameHostname behavior, users must now explicitly set strategy: EnqueueStrategy.SameHostname in their enqueueLinks options.

// If you want the old strict 'same-hostname' behavior after this update:
await enqueueLinks({
    strategy: 'same-hostname',
});

Contributors:

• Salvador Nunez: @SalvadorN323
• Alexander Manalad: @axmanalad
• Bao Truong: @baotruong04

Closes #2513

[pselvana]

Hi @barjin - The team has implemented your suggestion for #2513 in this PR. Could you provide some guidance on if this meets your expectations and any potential next steps (considering this is a breaking change and the need for waiting for your release process for cutting a 4.x based main).

The team also implemented another variant with a new strategy same-domain in case a breaking change was unwanted -- #3098

[pselvana]

Hi @barjin - pinging you one more time with the hope that you have a few cycles to look this and #3098 over to provide some guidance/next steps. If you aren't the right person, we'd appreciate if you could recommend who would be. The team working on this officially disbands tomorrow but would like to see this through if possible. Thank you for your consideration.

[barjin]

Hello, and sorry for the delay in our response.

After discussing this further with the team, we decided we'd like to keep SameHostname as the default strategy for Crawlee v4. The main argument against switching to SameDomain is avoiding accidental crawler runaway - if the user starts with e.g. blog.apify.com, they likely don't want to crawl console.apify.com and other similar subdomains.

I'll close this PR now. Thank you for your contribution and your understanding!