Remove faceted search — bot magnet tying up server resources

[rdhyee]

Context

Eric flagged that faceted search pages are heavily crawled by bots, tying up server resources. The combinatorial URL space of facets creates a near-infinite crawl surface that bots love to explore.

Problem

Faceted search generates many URL permutations (subject × language × format × ...) that:

• Bots crawl exhaustively, consuming server CPU and DB queries
• robots.txt can't practically block all combinations
• Each facet page triggers DB queries that are expensive at scale

Proposed approach

• Identify which faceted search views/URLs exist
• Evaluate whether any real users depend on them
• Remove or simplify to reduce the crawl surface
• Consider replacing with a simpler search that doesn't generate combinatorial URLs

Related

• Infrastructure: AI bot traffic management (bot blocking, Turnstile, signed URLs) #1078 (bot traffic management)
• Part of the modernization tracked in Migration plan: strip features, then upgrade Django 1.11 → 5.1 #1081

[rdhyee]

Fix: #1116

[rdhyee]

Phase 2 items (deferred from PR #1116)

PR #1116 implements Phase 1 — removing the web-facing facet UI. These items remain for Phase 2:

• Remove core/facets.py — still used by OPDS for single-facet filtering. Replace with simple querysets.
• Remove OPDS facet-parameterized routes — /api/opds/<facet>/ still accepts arbitrary facet strings and executes queries. Either restrict to known facets or serve only the root feed.
• Update api/templates/api_help.html — still documents compound facet examples like doab/-gtbg.
• Update frontend/tests.py — tests at lines 146-153 expect 200 from old faceted URLs, now they return 301.
• Clean up distro/push.py — imports BaseFacet from core/facets.py.
• Evaluate subject browse — subject links in subjects.html and subjectbox.html now go to /free/ without filtering. Could add a simple ?q=subject search redirect as an alternative.

[rdhyee]

Correction: Eric wants single facets preserved, only compound facets removed

Per discussion with Eric, the original PR #1116 went too far. Eric's intent:

• Keep: Single-facet paths like /free/kw.Fiction/, /free/epub/, /free/doab/ — users should still browse by subject, format, or collection
• Remove: Compound facet paths like /free/kw.Fiction/epub/, /free/doab/CC BY/ — the combinatorial explosion (O(facets²)) that bots crawl

The fix is narrower:

1. Keep FacetedView working with single-facet paths
2. Remove the "refine" sidebar (it invites adding a second facet → compound URLs)
3. Remove compound facet navigation from OPDS feeds
4. Set MAX_FACETS = 1 or reject compound paths
5. Keep core/facets.py intact

Reworking PR #1116 accordingly.

[rdhyee]

Revised scope (per Eric's feedback)

What I heard from Eric: subjects shouldn't be combined with other facets. The overarching goal is giving bots fewer opportunities to kill the server.

Revised approach — remove the discovery mechanism, not the paths:

1. Remove refine.html sidebar — this is what generates compound facet links on every page ("Show me only... format / license / collection"). Bots follow these links to discover O(facets²) URLs. Without it, compound paths are undiscoverable.

2. Remove compound facet navigation from OPDS/OPDS-JSON feeds — same discovery mechanism for feed consumers.

3. Keep all single-facet paths working — /free/kw.Fiction/, /free/epub/, /free/doab/ etc. all continue to work.

4. Keep FacetedView and core/facets.py intact — the faceting engine still works, we're just not advertising compound paths.

5. Remove dynamic facet links from explore.html — the per-subject links on work detail pages also generate crawlable URLs.

This is a lighter touch than PR #1116's current state. Reworking the PR.