Relax AI scrape policy

[SuperQ]

Switch to the permissive scrape policy to allow more well behaved AI crawlers.

[SuperQ]

This should implement similar to #451.

[nthmost]

Do we need to make some updates to robots.txt with this?

I'm admittedly getting a little head-swirly with the number of layers of bot detection here.

right now robots.txt only blocks ClaudeBot and Amazonbot, but not GPTBot, Google-Extended, CCBot... etc

[SuperQ]

Currently I only get this:

$ curl https://www.noisebridge.net/robots.txt
User-agent: *
Disallow: /wiki/86
Disallow: /86
Disallow: /index.php?page=86
Noindex: /wiki/86
Noindex: /86
Noindex: /index.php?page=86

[nthmost]

Currently I only get this:

oh i see -- there's an extant playbook for robots_txt that hasn't been run, maybe? from April 2024

(Sorry, still finding a lot of this confusing and didn't actually look at live robots.txt first)

[SuperQ]

Yea, probably just never deployed.

[nthmost]

should we clean up the robots_txt tho? (separately)

[SuperQ]

Yes, we should probably do a round of cleanup of the robots.txt and actually deploy it this time. :)