Skip to content

preRequest function cuts the entire branch instead of a single page #357

Open
@AleksandrBorkun

Description

@AleksandrBorkun

What is the current behavior?

preRequest function cutting a lot of links in case of URL regexp filtering

If the current behavior is a bug, please provide the steps to reproduce

const isVisitedMap = { '\\.com\\/users\\/[\\d]+[\\w]+': false //CONSUMER_PROFILE_REGEXP }

function isVisited(url){
  for(const [key, value] of Object.entries(isVisitedMap)){
      if(new RegExp(key, 'g').test(url)){
          if(value){
              return value;
          }else{
              isVisitedMap[key] = true;
              return false;
          }
      }
  }
  return false;
}

(async () => {
  const crawler = await HCCrawler.launch({
    // Function to be evaluated in browsers
    preRequest: (async (opt) => { return !isVisited(opt.url)}),
    evaluatePage: (() => ({
      pagePath: window.location.href,
    })),
    // Function to be called with evaluated results from browsers
    onSuccess: (result => {
      console.log('pagePath:', result.result.pagePath);
    }),
    exporter
  });

  // Queue a request
   await crawler.queue({
    url: 'https://www.example.com/',
    headless: false,
    maxDepth: 4,
    userAgent: 'DuckDuckBot',
    allowedDomains: [/example\.com$/],
    skipDuplicates: true,
  });
  await crawler.onIdle(); // Resolved when no queue is left
  await crawler.close(); // Close the crawler
})();

What is the expected behavior?
an only single page that match regexp skipped

What is the motivation / use case for changing the behavior?

if you'll try to crawl web sites with public access to the user's profile pages (or any other entities pages), you'll probably want to skip all of the user profile links except one, because all of them are similar.
I thought that I can skip a single page by returning false in preRequest function.

Please tell us about your environment:

  • Version: "headless-chrome-crawler": "^1.8.0",
  • Platform / OS version: Windows 10
  • Node.js version: v10.16.3 npm: '6.9.0',

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions