You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(domain): document admission, enhance leftover, and Status drops
Cleanup owns feed-item admission; enhance fills missing card fields and drops listing chrome. Status.admission_drops and auto_feed_result/--explain expose the same telemetry.
`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`).
79
+
`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count).
80
+
81
+
For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr).
Copy file name to clipboardExpand all lines: src/content/docs/ruby-gem/reference/auto-source.mdx
+27-19Lines changed: 27 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,25 +20,33 @@ To enable it, add `auto_source: {}` to your configuration:
20
20
21
21
## How It Works
22
22
23
-
`auto_source` uses the following strategies to find content:
24
-
25
-
1.**`wordpress_api`:** Detects the `<link rel="https://api.w.org/">` tag used by WordPress and pulls posts from the REST API without parsing article HTML. See [WordPress API](/ruby-gem/reference/wordpress-api/).
26
-
2.**`sitemap`:** Automatically locates XML sitemap documents (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering entries by priority and recency, with support for Google News tags (`<news:news>`).
27
-
3.**`meta_oembed`:** Extracts single-item articles, video pages, and media updates from OpenGraph/Twitter meta tags and resolves JSON oEmbed endpoints (`<link rel="alternate" type="application/json+oembed">`).
28
-
4.**`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)), extracting authors and publishers.
29
-
5.**`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`).
7.**`json_state`:** Single-page applications often stash pre-rendered article data in `<script type="application/json">` tags or global variables
32
-
such as `window.__NEXT_DATA__`, `window.__NUXT__`, or `window.STATE`. The JSON-state scraper walks those blobs, finds arrays with
33
-
`title`/`url` pairs, and converts them into feed items.
34
-
8.**`xhr_articles`:** When Botasaurus runs the browser tier, it always captures JSON XHR/fetch response bodies. This scraper reuses those
35
-
captured bodies (no extra HTTP requests) and extracts article-like arrays the same way `json_state` does. Empty for Faraday or Botasaurus HTTP-request tiers.
36
-
9.**`semantic_html`:** Searches for semantic HTML5 tags like `<article>`, `<main>`, and `<section>`, with fallback clustering for anchorless lists.
37
-
10.**`html`:** Analyzes HTML structure to find frequently occurring item selectors, with tag-structure and class-clustering fallbacks.
38
-
39
-
`auto_source` also automatically extracts and prepends kicker/teaser labels to article titles when present, and discovers channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
40
-
41
-
You can also restrict the maximum number of discovered articles via the `limit:` keyword argument in Ruby (`Html2rss.auto_source(url, limit: 10)`) or the `--limit` CLI flag.
23
+
One fetch, then scrapers run in **tiers**. Within a tier, results merge. The pipeline stops when Cleanup has admitted enough articles (`limit`, default `25`).
-**`json_state`:** Walks in-page JSON (`<script type="application/json">`, `window.__NEXT_DATA__`, `window.__NUXT__`, `window.STATE`) for arrays with `title`/`url` pairs.
36
+
-**`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during a Botasaurus **browser** scrape (no extra HTTP). Empty for Faraday and Botasaurus HTTP-request tiers.
37
+
-**`wordpress_api`:** Detects `<link rel="https://api.w.org/">` and pulls posts from the REST API. See [WordPress API](/ruby-gem/reference/wordpress-api/).
38
+
-**`sitemap`:** Locates XML sitemaps (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering by priority and recency, with Google News tags (`<news:news>`).
39
+
-**`meta_oembed`:** OpenGraph/Twitter meta tags plus JSON oEmbed (`<link rel="alternate" type="application/json+oembed">`).
40
+
-**`semantic_html`:** Semantic HTML5 (`<article>`, `<main>`, `<section>`), with fallback clustering for anchorless lists.
After scrapers, **Cleanup** is the admission gate: merge, dedupe, hard-exclude non-article destinations, drop junk titles, trim to `limit`. Scoring ranks and demotes; it does not refill `limit` after Cleanup.
44
+
45
+
Drop reasons (counts) land on `Html2rss::Status#admission_drops`. Print them with `html2rss auto URL --explain` (JSON on stderr; the feed stays on stdout). Prefer `Html2rss.auto_feed_result` in Ruby when you need that status object.
46
+
47
+
`auto_source` also prepends kicker/teaser labels to titles when present, and discovers channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
48
+
49
+
Ruby `limit:` and CLI `--limit` cap kept articles (default `25`).
42
50
43
51
**`json_state` / `xhr_articles` Limitations:** both scrapers require discoverable arrays of hashes containing clear `title` and `url` fields. Minified or
44
52
obfuscated payloads, heavily encoded values, or blobs that require executing embedded functions are ignored. `xhr_articles` only sees JSON bodies captured during the current Botasaurus browser scrape (capped by the scrape API).
Copy file name to clipboardExpand all lines: src/content/docs/ruby-gem/reference/selectors.mdx
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,7 +28,11 @@ At a minimum, you need an `items` selector to define the list of articles and a
28
28
29
29
## Automatic Item Enhancement
30
30
31
-
To simplify configuration, `html2rss` can automatically extract the `title`, `url`, and `image` from each item. This feature is enabled by default.
31
+
With `enhance: true` (the default), `html2rss` fills **missing** fields on each matched item from the card HTML: `title`, `url`, `description`, `published_at`, `image`, `categories`, and enclosures. Explicit selectors still win when they produce a value.
32
+
33
+
Description leftover text drops listing chrome (CTA lines such as “read more”, date-shaped chips, type chips, title echoes, section names). When the `items` selector is a heading or wrapping link with no date or description, extraction can climb to the parent card — it aborts if that parent looks like a listing (multiple headings or article links).
0 commit comments