feat: introduce UA Blocker Middleware

[finxol]

Create a new middleware for AI bots blocking.
Inspired by ai.robots.txt.

This provides several options:

• Block all known AI bots
• Block all bots that we're not sure respect robots.txt
• Create a robots.txt file to be served (also offered as a middleware)

See README for details

The author should do the following, if applicable

• Add tests
• Run tests
• yarn changeset at the top of this repo and push the changeset
• Follow the contribution guide

[changeset-bot]

🦋 Changeset detected

Latest commit: d561bc6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@hono/ua-blocker	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[yusukebe]

Can you add a CI for this middleware, referring to https://github.com/honojs/middleware/blob/main/.github/workflows/ci-hello.yml?

[yusukebe]

Hey @finxol

This middleware is interesting! The idea is good. I've left some comments.

[finxol]

@yusukebe Thanks for the feedback! I'll make the appropriate changes. Let me know if there's anything else.

[finxol]

I've addressed all comments in the latest commit, the architecture and goal of the package is now broader than just an AI bots blocker.

The main export is now a generic User Agent based blocker, with a path export for the robots.json stuff that can be passed to the generic. See README file for examples.
The regex construction is moved out of the middleware, to the uaBlocker function. It isn't done at build time, but enables the consumer to pass any list of user agents to block.

[finxol]

publint fails for @hono/ua-blocker/ai-bots on node10 because of the export path.
Do we need to support this node10 module resolution, or can we ignore it?

[BarryThePenguin]

publint fails for @hono/ua-blocker/ai-bots on node10 because of the export path. Do we need to support this node10 module resolution, or can we ignore it?

Yes, it can be ignored by adding attw --pack --profile node16 to the publint script.

I included this change, along with some other suggestions, in finxol#1. Feel free to include them or not 😅

[BarryThePenguin]

Your Readme and tests are very comprehensive 👏🏻

[yusukebe]

@finxol

Wooow. Making this UA Blocker Middleware for general user agents is a great idea. Including ai-bots is also a good idea. If the CI passes, I'll review again, merge it, and release!

[finxol]

@yusukebe Thank you for the feedback! I just want to make a small change first, I'll push it later today and I'll let you know.

And thank you for the great work on Hono :)

[finxol]

I've moved the generator script back out of get-robots-txt.ts. This allows us to generate from the version-controlled robots.json instead of the upstream file

[finxol]

@yusukebe it's now ready for review :)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.52632% with 47 lines in your changes missing coverage. Please review.

Project coverage is 78.25%. Comparing base (86a4ca4) to head (d561bc6).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/ua-blocker/script/prebuild.ts	0.00%	22 Missing and 1 partial ⚠️
packages/ua-blocker/src/escape.ts	65.00%	13 Missing and 1 partial ⚠️
packages/ua-blocker/script/get-robots-txt.ts	0.00%	6 Missing ⚠️
packages/ua-blocker/src/demo.ts	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1181      +/-   ##
==========================================
- Coverage   79.32%   78.25%   -1.08%     
==========================================
  Files          81       88       +7     
  Lines        2443     2538      +95     
  Branches      633      652      +19     
==========================================
+ Hits         1938     1986      +48     
- Misses        419      464      +45     
- Partials       86       88       +2     

Flag	Coverage Δ
ua-blocker	1.89% <50.52%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[finxol]

Suggested changes are applied, ci should now pass

[yusukebe]

@finxol

I tried using this UA Blocker middleware, and it felt good. The rest is making CI/test pass.

[finxol]

Sorry for the force push, needed to rebase to have CI pass with .prettierignore

[finxol]

generated.ts should be ignored by Prettier, but having it in packages/ua-blocker/.prettierignore doesn't get picked up because of the prettier --check . '!packages' packages/ua-blocker path specification.

[finxol]

@yusukebe Sorry for the ci and test issues. Everything should be fixed now.

[yusukebe]

LGTM!

[yusukebe]

@finxol

Thanks! Let's go with this. I'll merge and release an initial version.