Override conventional public share paths

[qrkourier]

Summary

Enhance zrok so users can override or define responses for conventional URL paths such as /favicon.ico, /robots.txt, and /llms.txt for their public shares.

This would let users express standard conventions (like discouraging LLM crawlers or customizing browser tab icons), even when their backend isn't a static web server, or when they're proxying to a server they can't easily modify without extra layers (e.g., reverse proxy).

Motivation

• When using zrok to proxy a backend server that users don't or can't control directly, it's difficult to customize responses to specific URL paths commonly expected on the public internet (e.g., favicon, robots.txt, llms.txt).
• For example, you may want to block ChatGPT and other LLMs from crawling your share via /llms.txt, but can't easily add this to your app/backend.
• Currently, the only workaround is to add another proxy layer, which increases complexity (e.g., using Caddy).

Proposal

• Allow zrok to define and respond to a configurable set of conventional paths (e.g., /favicon.ico, /robots.txt, /llms.txt) on a per-share basis.
• Response bodies could be specified per share or set to defaults.
• Optionally allow specifying not only responses for these paths but also customizing headers, etc.

Use Cases

• Proxying an application where you don't control the backend to set robots.txt for search crawlers.
• Providing a custom /favicon.ico for a branded share, even if the backend is not a web server.
• Discouraging specific crawlers (e.g., LLMs like ChatGPT) using /llms.txt, regardless of backend type.

Alternatives Considered

• Proxy-layer solutions (e.g., running Caddy/Nginx in front of the backend, just to support these files)

Non-goals

• Static web backends don't have this problem because they can place the desired files easily into the conventional paths, so this isn't aimed at modifying how that works. Still, if these metadata are enabled for web backends, they'd override any static files in the shared file tree.