Skip to content

Commit 486e0b7

Browse files
committed
docs(infrastructure): document Faraday HTML sniffing and Botasaurus validation
Faraday decodes unlabeled gzip/brotli and sniffs HTML bodies. Invalid Botasaurus keys (scroll_to_bottom, array window_size) fail validation; --explain surfaces admission drops.
1 parent 9677b3a commit 486e0b7

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

src/content/docs/troubleshooting/troubleshooting.mdx

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ The `auto` flow is URL-surface sensitive.
3030
- search result pages
3131
- client-rendered app-shell entrypoints
3232

33-
If extraction quality is poor, switch to a more specific listing/update URL before tuning selectors.
33+
If extraction quality is poor, switch to a more specific listing/update URL before tuning selectors. `html2rss auto URL --explain` prints admission drop counts on stderr so you can see why items were dropped (junk titles, non-article destinations) without mixing JSON into the feed.
3434

3535
### Empty Feeds
3636

@@ -95,12 +95,13 @@ Then run with:
9595

9696
Common configuration-related errors:
9797

98-
- **`UnsupportedResponseContentType`:** The website returned content that html2rss can't parse (not HTML or JSON).
98+
- **`UnsupportedResponseContentType`:** The website returned content that html2rss can't parse (not HTML or JSON). Faraday still treats bodies that sniff as HTML (`<!DOCTYPE html` / `<html`) as HTML, and decodes gzip/brotli even when `Content-Encoding` is missing.
9999
- **`UnsupportedStrategy`:** The specified strategy is not available. Use `auto`, `faraday`, `botasaurus`, or `local_file`.
100100
- **`BOTASAURUS_SCRAPER_URL is required for strategy=botasaurus.`:** Set `BOTASAURUS_SCRAPER_URL` to your Botasaurus scrape API base URL when using `--strategy botasaurus`.
101101
- **`BOTASAURUS_SCRAPER_URL is invalid`:** Fix the URL format and retry.
102102
- **`Configuration must include at least 'selectors' or 'auto_source'`:** You need to specify either manual selectors or enable auto-source.
103103
- **`stylesheet.type invalid`:** Only `text/css` and `text/xsl` are supported for stylesheets.
104+
- **`request.botasaurus` validation:** Unknown keys fail (`scroll_to_bottom` is not valid — use `scroll: true`). `window_size` must be `{ width:, height: }`. `wait_timeout_seconds` must be `1..20`.
104105

105106
### Missing Item Parts
106107

0 commit comments

Comments
 (0)