Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Nov 22, 2025

Adds detection for three additional Bing/Microsoft crawlers per their official crawler list: MSNBot, BingPreview, and AdIdxBot.

Changes

  • Browser detection (parser-browsers.js): Added patterns for MSNBot, BingPreview, and AdIdxBot. Placed before BingCrawler to ensure specific patterns match first.
  • Platform detection (parser-platforms.js): Added bot type classification with appropriate vendors (Microsoft for MSNBot, Bing for BingPreview/AdIdxBot).
  • Constants (constants.js): Added aliases and mappings for the three new bots.
  • Test cases (useragentstrings.yml): Added real user agent strings covering both numeric versions and alpha/beta suffixes.

Version handling

Regex pattern /(\d+(\.\d+)*[a-z]*)/i supports versions like "2.0", "2.1", and "2.0b" (with alpha/beta suffixes).

const parser = Bowser.getParser('msnbot/2.0b (+http://search.msn.com/msnbot.htm)');
parser.getBrowserName();     // "MSNBot"
parser.getBrowserVersion();  // "2.0b"
parser.getPlatformType();    // "bot"

Note: Googlebot and BingCrawler were already supported.

Original prompt

This section details on the original issue you should resolve

<issue_title>Build in recognition of search bot user agents</issue_title>
<issue_description>Feature Suggestion:
It seems that it would be relatively common use case to desire to know if the agent is a search bot, so building in the user agents for common search bots seems like a reasonable feature.

Google: https://support.google.com/webmasters/answer/1061943?hl=en
Bing: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
</issue_description>

Comments on the Issue (you are @copilot in this section)

@lancedikson It's ok. Could you prepare a PR?

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI changed the title [WIP] Add recognition for search bot user agents Add support for MSNBot, BingPreview, and AdIdxBot crawlers Nov 22, 2025
Copilot AI requested a review from naorpeled November 22, 2025 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build in recognition of search bot user agents

2 participants