Add more bot platforms #542

luciomartinez · 2023-11-28T10:02:15Z

Hey,

I thought it might be helpful to have a few more popular bots besides Google bot (already registered as a platform).

This list is not extensive, yet it includes the most popular ones worldwide, including the main Russian and Chinese search engines.

One of the down sides of having this list is its maintenance, so feel free to skip it from merging if you deem it so.

Thanks,
Lucio.

lancedikson · 2024-07-23T08:16:27Z

Hi @luciomartinez! Sorry it took a while to get back to your PR. If you still available, can you please also add tests for these platforms so that we have examples in the tests?

luciomartinez · 2024-08-21T11:44:32Z

src/parser-platforms.js

+
+  /* Baidu */
+  {
+    test: [/baiduspider/i],


https://www.baidu.com/search/robots_english.html

luciomartinez · 2024-08-21T11:44:55Z

src/parser-platforms.js

+
+  /* Bingbot */
+  {
+    test: [/bingbot/i],


https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0

luciomartinez · 2024-08-21T11:49:10Z

src/parser-platforms.js

+
+  /* DuckDuckBot */
+  {
+    test: [/duckduckbot/i],


https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

luciomartinez · 2024-08-21T11:56:19Z

src/parser-platforms.js


+  /* AmazonBot */
+  {
+    test: [/Amazonbot/i],


https://developer.amazon.com/amazonbot

luciomartinez · 2024-08-21T11:56:38Z

src/parser-platforms.js

+
+  /* Internet Archive Crawler */
+  {
+    test: [/ia_archiver/i],


https://web.archive.org/web/20141022124819/http://archive.org:80/about/exclude.php

luciomartinez · 2024-08-21T11:57:09Z

src/parser-platforms.js

+
+  /* Meta Web Crawler */
+  {
+    test: [/facebookexternalhit/i, /facebookcatalog/i],


https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

luciomartinez · 2024-08-21T12:01:15Z

src/parser-platforms.js

+
+  /* Yahoo! Slurp */
+  {
+    test: [/yahoo/i],


https://help.yahoo.com/kb/SLN22600.html

luciomartinez · 2024-08-21T12:04:34Z

src/parser-platforms.js

+
+  /* Yandex */
+  {
+    test: [/yandexbot/i, /yandexmobilebot/i],


https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html

luciomartinez · 2024-08-21T12:08:08Z

src/parser-platforms.js

+
+  /* Pingdom */
+  {
+    test: [/pingdom/i],


Best guest I suppose 🙂

luciomartinez · 2024-08-21T12:26:22Z

CI logs don't show exactly which test is failing.

Trying to build it locally, I get this error:

$ npm run build

> [email protected] build
> webpack --config webpack.config.js

node:internal/crypto/hash:79
  this[kHandle] = new _Hash(algorithm, xofLen, algorithmId, getHashCache());
                  ^

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:79:19)
    at Object.createHash (node:crypto:139:10)
    at module.exports (/workspaces/bowser/node_modules/webpack/lib/util/createHash.js:135:53)
    at NormalModule._initBuildHash (/workspaces/bowser/node_modules/webpack/lib/NormalModule.js:417:16)
    at /workspaces/bowser/node_modules/webpack/lib/NormalModule.js:452:10
    at /workspaces/bowser/node_modules/webpack/lib/NormalModule.js:323:13
    at /workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:367:11
    at /workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:233:18
    at context.callback (/workspaces/bowser/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
    at /workspaces/bowser/node_modules/babel-loader/lib/index.js:55:103 {
  opensslErrorStack: [
    'error:03000086:digital envelope routines::initialization error',
    'error:0308010C:digital envelope routines::unsupported'
  ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v20.16.0

I'm not sure what I'm doing wrong here 😕

laurens94 · 2025-03-18T09:43:02Z

I believe that error occurs because of version mismatch with nodejs and some dependencies: https://stackoverflow.com/questions/69692842/error-message-error0308010cdigital-envelope-routinesunsupported

Copilot

Pull request overview

This PR adds platform detection support for 9 popular bot crawlers (AmazonBot, BingCrawler, BaiduSpider, DuckDuckBot, InternetArchiveCrawler, MetaWebCrawler, YahooSlurp, YandexBot, and PingdomBot) to complement the existing Googlebot support. The implementation follows the existing pattern by adding platform parsers and corresponding test cases.

Key changes:

Adds 9 new bot platform detectors to parser-platforms.js with regex patterns for identification
Adds 163 lines of test cases to useragentstrings.yml covering various User-Agent strings for each bot
Includes support for major search engines (Bing, Baidu, Yandex, Yahoo) and monitoring/archiving services

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/parser-platforms.js	Adds 9 new bot platform descriptors with regex patterns and vendor identification
test/acceptance/useragentstrings.yml	Adds comprehensive test cases for all new bots with expected browser names, versions, and platform metadata

test/acceptance/useragentstrings.yml

src/parser-platforms.js

naorpeled · 2025-11-22T12:53:38Z

Hey @luciomartinez,
first of all thank you for this great contribution.

I hope it's okay with you that I'll address Copilot's comments and then merge this PR 🙏

luciomartinez · 2025-11-22T13:23:21Z

Thank you! It, sounds good with me.:)On 22 Nov 2025, at 13:54, Naor Peled ***@***.***> wrote:naorpeled left a comment (bowser-js/bowser#542) Hey @luciomartinez, first of all thank you for this great contribution. If it's okay with you I'll address Copilot's comments and then merge this PR 🙏 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

src/constants.js

test/acceptance/useragentstrings.yml

src/parser-platforms.js

src/constants.js

test/acceptance/useragentstrings.yml

# Conflicts: # src/constants.js

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

src/parser-platforms.js

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Add bot platforms

e417ea2

luciomartinez commented Aug 21, 2024

View reviewed changes

Add Facebook and IA crawler

d197a44

luciomartinez commented Aug 21, 2024

View reviewed changes

src/parser-platforms.js Outdated

/* AmazonBot */

{

test: [/Amazonbot/i],

Copy link

Contributor Author

luciomartinez Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developer.amazon.com/amazonbot

luciomartinez commented Aug 21, 2024

View reviewed changes

src/parser-platforms.js

/* Yahoo! Slurp */

{

test: [/yahoo/i],

Copy link

Contributor Author

luciomartinez Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://help.yahoo.com/kb/SLN22600.html

Add mobile layout of Yandex bot

d668404

luciomartinez commented Aug 21, 2024

View reviewed changes

src/parser-platforms.js

/* Pingdom */

{

test: [/pingdom/i],

Copy link

Contributor Author

luciomartinez Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best guest I suppose 🙂

Add tests

6647d3b

Fix typo

1e86c2e

naorpeled requested a review from Copilot November 22, 2025 12:28

naorpeled approved these changes Nov 22, 2025

View reviewed changes

Copilot started reviewing on behalf of naorpeled November 22, 2025 12:28 View session

Copilot finished reviewing on behalf of naorpeled November 22, 2025 12:31

Copilot AI reviewed Nov 22, 2025

View reviewed changes

test/acceptance/useragentstrings.yml Show resolved Hide resolved

src/parser-platforms.js Show resolved Hide resolved

src/parser-platforms.js Show resolved Hide resolved

fix identation in settings

da61600

added missing parts

4050a25

naorpeled requested a review from Copilot November 22, 2025 13:32

Copilot started reviewing on behalf of naorpeled November 22, 2025 13:33 View session

Copilot finished reviewing on behalf of naorpeled November 22, 2025 13:36

Copilot AI reviewed Nov 22, 2025

View reviewed changes

naorpeled added 7 commits November 22, 2025 15:49

tests fix pt.1

bb05390

tests fix pt.2

8ff7cdc

tests fix pt.3

38547c8

tests fix pt.4

26aafc6

sort constants

8d47b99

Merge remote-tracking branch 'origin/master' into add-bot-platforms

4cdac37

# Conflicts: # src/constants.js

sort constants

2df1fb1

naorpeled requested a review from Copilot November 22, 2025 14:31

Copilot started reviewing on behalf of naorpeled November 22, 2025 14:32 View session

Copilot finished reviewing on behalf of naorpeled November 22, 2025 14:34

Copilot AI reviewed Nov 22, 2025

View reviewed changes

src/parser-platforms.js Outdated Show resolved Hide resolved

fixed amazon casing

3c451d6

naorpeled requested a review from Copilot November 22, 2025 14:41

Copilot started reviewing on behalf of naorpeled November 22, 2025 14:42 View session

Copilot finished reviewing on behalf of naorpeled November 22, 2025 14:44

naorpeled merged commit 771dfb2 into bowser-js:master Nov 22, 2025
6 checks passed

Copilot AI reviewed Nov 22, 2025

View reviewed changes

luciomartinez deleted the add-bot-platforms branch December 8, 2025 18:07

Uh oh!

Add more bot platforms #542

Add more bot platforms #542

Uh oh!

Conversation

luciomartinez commented Nov 28, 2023

Uh oh!

lancedikson commented Jul 23, 2024

Uh oh!

luciomartinez Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luciomartinez commented Aug 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laurens94 commented Mar 18, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

naorpeled commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luciomartinez commented Nov 22, 2025 via email

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

luciomartinez Aug 21, 2024 •

edited

Loading

luciomartinez commented Aug 21, 2024 •

edited

Loading

naorpeled commented Nov 22, 2025 •

edited

Loading