We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
1 parent 66ac4f6 commit acef547Copy full SHA for acef547
1 file changed
src/public/robots.txt
@@ -21,6 +21,8 @@ User-agent: adidxbot
21
# apple.com search engine
22
User-agent: Applebot
23
user-agent: AppleNewsBot
24
+# archive.org web archive
25
+User-agent: archive.org_bot
26
# baidu.com chinese search engine
27
User-agent: Baiduspider
28
User-agent: Baiduspider-image
0 commit comments