Skip to content

Commit 9d4717a

Browse files
committed
Updated ai.robots.txt to 1.40
1 parent e75c9c7 commit 9d4717a

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

robots.txt

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# robots.txt
22
# newtonfaq.com
3-
# updated 14 August 2025
3+
# updated 28 August 2025
44

55
Sitemap: https://newtonfaq.com/sitemap.xml
66

@@ -47,6 +47,7 @@ User-agent: FriendlyCrawler
4747
User-agent: Gemini-Deep-Research
4848
User-agent: Google-CloudVertexBot
4949
User-agent: Google-Extended
50+
User-agent: Google-Firebase
5051
User-agent: GoogleAgent-Mariner
5152
User-agent: GoogleOther
5253
User-agent: GoogleOther-Image
@@ -88,6 +89,7 @@ User-agent: SBIntuitionsBot
8889
User-agent: Scrapy
8990
User-agent: SemrushBot-OCOB
9091
User-agent: SemrushBot-SWA
92+
User-agent: ShapBot
9193
User-agent: Sidetrade indexer bot
9294
User-agent: Thinkbot
9395
User-agent: TikTokSpider

0 commit comments

Comments
 (0)