"A robots.txt .. disallows all robots from accessing your pastes."

[cloudscape-germany]

Describe the problem/question

The README.md reads:

A robots.txt .. disallows all robots from accessing your pastes.

I am unclear about how relevant a robots.txt is to provide the Internet from accessing pastes. Everyone ignores robots.txt nowadays as we all very well know, and thus I think this is an unsafe feature to block Bots from accessing pastes. Shouldn't pastes be protected differently?

Did you use the FAQ section?

• Yes, I have read the FAQ and I found no solution/answer there.

What you did?

1. Read the README.md file
2. Wondering about reliance on a robots.txt file to prevent access to pastes.

What happens

No response

What should happen

No response

Additional information

No response

Server address

No response

Server OS

No response

Webserver

No response

PrivateBin version

No response

Browser and version

No response

Local operating system and version

No response

Issue reproducibility

No, I cannot reproduce it on https://privatebin.net.

[rugk]

You have a good point here. It is and never was a (reliable) mechanism to technically prevent access rather than just friendly ask for it. Feel free to provide a PR, that would help, thanks!

[cloudscape-germany]

Before we just delete it, what purpose does it serve, I mean the documents (pastes) are stored in the backend anyway?

[elrido]

Please don't delete it, it works very much as intended. When searching for the demo website URL, which does use it:

• Google displays it as the first result, but without additional description and as there is "no information on this page". The project info website is the second result.
• Bing doesn't display it at all, first result is the project info website.

So the robots.txt does clearly still work to keep the instances from getting indexed by the big search engines. It is also one of the documented ways (and an example for that is included) to disable getting indexed by the privatebin directory itself.

It is sad that recently the greedy AI companies no longer respect these and their spiders swamp even small sites that otherwise rely on the robots.txt to prevent their resources being exhausted by too many parallel access in quick succession. That doesn't make it unnecessary and it does still work for the more reputable spiders.

For the bots that do execute JS, we include another user agent based mechanism and we are happy to accept PRs to extend the list of denied agents there:

PrivateBin/js/legacy.js

Lines 53 to 64 in b083043

	/**
	* blacklist of UserAgents (parts) known to belong to a bot
	*
	* @private
	* @enum {Array}
	* @readonly
	*/
	var badBotUA = [
	'Bot',
	'bot'
	];

[rugk]

Yeah, we can reword it though. The robots.txt technically cannot prevent any access. Our JS logic is better here, but even there obviously we cannot prevent it at all.

[rugk]

Created a PR to clarify the installation docs (it BTW was never stated in the Readme.md) in #1755

[cloudscape-germany]

Created a PR to extend the list of Bots to a more complete list and changed enum array to type string in #1759