Skip to content

Commit 60cef1f

Browse files
eliasdabbaseliasdabbas
committed
Add Google-specific user-agent tokens and strings
1 parent 577dd3f commit 60cef1f

5 files changed

+81
-1
lines changed

‎advertools/code_recipes/spider_strategies.py

+20
Original file line numberDiff line numberDiff line change
@@ -453,6 +453,26 @@
453453
Xbox One S Mozilla/5.0 (Windows NT 10.0; Win64; x64; XBOX_ONE_ED) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
454454
Xbox Series X Mozilla/5.0 (Windows NT 10.0; Win64; x64; Xbox; Xbox Series X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36 Edge/20.02
455455
Yahoo! bot Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
456+
Googlebot Smartphone Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
457+
Googlebot Desktop Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
458+
Googlebot-Image Googlebot-Image/1.0
459+
Googlebot-News Googlebot-News
460+
Googlebot-Video Googlebot-Video/1.0
461+
Storebot-Google Desktop Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
462+
Storebot-Google Smartphone Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
463+
Google-InspectionTool Mobile Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)
464+
Google-InspectionTool Desktop Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)
465+
GoogleOther Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
466+
GoogleOther-Image GoogleOther-Image/1.0
467+
GoogleOther-Video GoogleOther-Video/1.0
468+
APIs-Google APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
469+
AdsBot-Google-Mobile Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
470+
AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html)
471+
Mediapartners-Google Mediapartners-Google
472+
Google-Safety Google-Safety
473+
FeedFetcher-Google FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
474+
Google Publisher Center GoogleProducer; (+http://goo.gl/7y4SX)
475+
Google Site Verifier Mozilla/5.0 (compatible; Google-Site-Verification/1.0)
456476
======================================================== =========================================================================================================================================================================
457477
458478
""" # noqa: E501

‎docs/_build/doctrees/advertools.code_recipes.spider_strategies.doctree

10.9 KB
Binary file not shown.

‎docs/_build/doctrees/environment.pickle

14.6 KB
Binary file not shown.

‎docs/_build/html/advertools.code_recipes.spider_strategies.html

+60
Original file line numberDiff line numberDiff line change
@@ -887,6 +887,66 @@ <h2>User-agent strings for use in crawling<a class="headerlink" href="#user-agen
887887
<tr class="row-odd"><td><p>Yahoo! bot</p></td>
888888
<td><p>Mozilla/5.0 (compatible; Yahoo! Slurp; <a class="reference external" href="http://help.yahoo.com/help/us/ysearch/slurp">http://help.yahoo.com/help/us/ysearch/slurp</a>)</p></td>
889889
</tr>
890+
<tr class="row-even"><td><p>Googlebot Smartphone</p></td>
891+
<td><p>Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</p></td>
892+
</tr>
893+
<tr class="row-odd"><td><p>Googlebot Desktop</p></td>
894+
<td><p>Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36</p></td>
895+
</tr>
896+
<tr class="row-even"><td><p>Googlebot-Image</p></td>
897+
<td><p>Googlebot-Image/1.0</p></td>
898+
</tr>
899+
<tr class="row-odd"><td><p>Googlebot-News</p></td>
900+
<td><p>Googlebot-News</p></td>
901+
</tr>
902+
<tr class="row-even"><td><p>Googlebot-Video</p></td>
903+
<td><p>Googlebot-Video/1.0</p></td>
904+
</tr>
905+
<tr class="row-odd"><td><p>Storebot-Google Desktop</p></td>
906+
<td><p>Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36</p></td>
907+
</tr>
908+
<tr class="row-even"><td><p>Storebot-Google Smartphone</p></td>
909+
<td><p>Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36</p></td>
910+
</tr>
911+
<tr class="row-odd"><td><p>Google-InspectionTool Mobile</p></td>
912+
<td><p>Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)</p></td>
913+
</tr>
914+
<tr class="row-even"><td><p>Google-InspectionTool Desktop</p></td>
915+
<td><p>Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)</p></td>
916+
</tr>
917+
<tr class="row-odd"><td><p>GoogleOther</p></td>
918+
<td><p>Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)</p></td>
919+
</tr>
920+
<tr class="row-even"><td><p>GoogleOther-Image</p></td>
921+
<td><p>GoogleOther-Image/1.0</p></td>
922+
</tr>
923+
<tr class="row-odd"><td><p>GoogleOther-Video</p></td>
924+
<td><p>GoogleOther-Video/1.0</p></td>
925+
</tr>
926+
<tr class="row-even"><td><p>APIs-Google</p></td>
927+
<td><p>APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)</p></td>
928+
</tr>
929+
<tr class="row-odd"><td><p>AdsBot-Google-Mobile</p></td>
930+
<td><p>Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)</p></td>
931+
</tr>
932+
<tr class="row-even"><td><p>AdsBot-Google</p></td>
933+
<td><p>AdsBot-Google (+http://www.google.com/adsbot.html)</p></td>
934+
</tr>
935+
<tr class="row-odd"><td><p>Mediapartners-Google</p></td>
936+
<td><p>Mediapartners-Google</p></td>
937+
</tr>
938+
<tr class="row-even"><td><p>Google-Safety</p></td>
939+
<td><p>Google-Safety</p></td>
940+
</tr>
941+
<tr class="row-odd"><td><p>FeedFetcher-Google</p></td>
942+
<td><p>FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)</p></td>
943+
</tr>
944+
<tr class="row-even"><td><p>Google Publisher Center</p></td>
945+
<td><p>GoogleProducer; (+http://goo.gl/7y4SX)</p></td>
946+
</tr>
947+
<tr class="row-odd"><td><p>Google Site Verifier</p></td>
948+
<td><p>Mozilla/5.0 (compatible; Google-Site-Verification/1.0)</p></td>
949+
</tr>
890950
</tbody>
891951
</table>
892952
</section>

‎docs/_build/html/searchindex.js

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)