Skip to content

Commit 7fc7d0f

Browse files
committed
docs(domain): document admission, enhance leftover, and Status drops
Cleanup owns feed-item admission; enhance fills missing card fields and drops listing chrome. Status.admission_drops and auto_feed_result/--explain expose the same telemetry.
1 parent cb68e56 commit 7fc7d0f

3 files changed

Lines changed: 35 additions & 21 deletions

File tree

src/content/docs/ruby-gem/guides/managing-feed-configs.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,9 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F
7676
- `to_rss` / `to_json_feed(feed_url:)` — render formats
7777
- `status` — scrape telemetry (`Html2rss::Status`)
7878

79-
`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`).
79+
`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`), and `admission_drops` (Cleanup reason → count).
80+
81+
For URL-only auto discovery, `Html2rss.auto_feed_result(url)` returns the same `FeedResult` (CLI `html2rss auto --explain` prints `status.to_h` on stderr).
8082

8183
<Code
8284
code={`

src/content/docs/ruby-gem/reference/auto-source.mdx

Lines changed: 27 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -20,25 +20,33 @@ To enable it, add `auto_source: {}` to your configuration:
2020

2121
## How It Works
2222

23-
`auto_source` uses the following strategies to find content:
24-
25-
1. **`wordpress_api`:** Detects the `<link rel="https://api.w.org/">` tag used by WordPress and pulls posts from the REST API without parsing article HTML. See [WordPress API](/ruby-gem/reference/wordpress-api/).
26-
2. **`sitemap`:** Automatically locates XML sitemap documents (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering entries by priority and recency, with support for Google News tags (`<news:news>`).
27-
3. **`meta_oembed`:** Extracts single-item articles, video pages, and media updates from OpenGraph/Twitter meta tags and resolves JSON oEmbed endpoints (`<link rel="alternate" type="application/json+oembed">`).
28-
4. **`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)), extracting authors and publishers.
29-
5. **`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`).
30-
6. **`microformats2`:** Parses Microformats2 `h-entry` markup, extracting `p-name`, `e-content`, `u-url`, `dt-published`, `p-author`, `p-category`, and `u-photo` / `u-featured` media.
31-
7. **`json_state`:** Single-page applications often stash pre-rendered article data in `<script type="application/json">` tags or global variables
32-
such as `window.__NEXT_DATA__`, `window.__NUXT__`, or `window.STATE`. The JSON-state scraper walks those blobs, finds arrays with
33-
`title`/`url` pairs, and converts them into feed items.
34-
8. **`xhr_articles`:** When Botasaurus runs the browser tier, it always captures JSON XHR/fetch response bodies. This scraper reuses those
35-
captured bodies (no extra HTTP requests) and extracts article-like arrays the same way `json_state` does. Empty for Faraday or Botasaurus HTTP-request tiers.
36-
9. **`semantic_html`:** Searches for semantic HTML5 tags like `<article>`, `<main>`, and `<section>`, with fallback clustering for anchorless lists.
37-
10. **`html`:** Analyzes HTML structure to find frequently occurring item selectors, with tag-structure and class-clustering fallbacks.
38-
39-
`auto_source` also automatically extracts and prepends kicker/teaser labels to article titles when present, and discovers channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
40-
41-
You can also restrict the maximum number of discovered articles via the `limit:` keyword argument in Ruby (`Html2rss.auto_source(url, limit: 10)`) or the `--limit` CLI flag.
23+
One fetch, then scrapers run in **tiers**. Within a tier, results merge. The pipeline stops when Cleanup has admitted enough articles (`limit`, default `25`).
24+
25+
1. **In-page structured:** `schema`, `microdata`, `microformats2`, `json_state`, `xhr_articles`
26+
2. **Follow-up IO:** `wordpress_api`, `sitemap`, `meta_oembed`
27+
3. **Heuristic:** `semantic_html`
28+
4. **Heuristic:** `html` — skipped when an earlier tier already admitted at least one clean article
29+
30+
What each scraper does:
31+
32+
- **`schema`:** Parses `<script type="json/ld">` tags containing structured data (e.g., [Schema.org](https://schema.org/)), extracting authors and publishers.
33+
- **`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`).
34+
- **`microformats2`:** Parses Microformats2 `h-entry` markup, extracting `p-name`, `e-content`, `u-url`, `dt-published`, `p-author`, `p-category`, and `u-photo` / `u-featured` media.
35+
- **`json_state`:** Walks in-page JSON (`<script type="application/json">`, `window.__NEXT_DATA__`, `window.__NUXT__`, `window.STATE`) for arrays with `title`/`url` pairs.
36+
- **`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during a Botasaurus **browser** scrape (no extra HTTP). Empty for Faraday and Botasaurus HTTP-request tiers.
37+
- **`wordpress_api`:** Detects `<link rel="https://api.w.org/">` and pulls posts from the REST API. See [WordPress API](/ruby-gem/reference/wordpress-api/).
38+
- **`sitemap`:** Locates XML sitemaps (`<link rel="sitemap">`, `/sitemap.xml`, or `/robots.txt`), filtering by priority and recency, with Google News tags (`<news:news>`).
39+
- **`meta_oembed`:** OpenGraph/Twitter meta tags plus JSON oEmbed (`<link rel="alternate" type="application/json+oembed">`).
40+
- **`semantic_html`:** Semantic HTML5 (`<article>`, `<main>`, `<section>`), with fallback clustering for anchorless lists.
41+
- **`html`:** Repeated item structure (list / cluster / semantic segmenter). Last resort only.
42+
43+
After scrapers, **Cleanup** is the admission gate: merge, dedupe, hard-exclude non-article destinations, drop junk titles, trim to `limit`. Scoring ranks and demotes; it does not refill `limit` after Cleanup.
44+
45+
Drop reasons (counts) land on `Html2rss::Status#admission_drops`. Print them with `html2rss auto URL --explain` (JSON on stderr; the feed stays on stdout). Prefer `Html2rss.auto_feed_result` in Ruby when you need that status object.
46+
47+
`auto_source` also prepends kicker/teaser labels to titles when present, and discovers channel-level `author` and `image` (favicon, touch-icon, or OpenGraph images) from the host page.
48+
49+
Ruby `limit:` and CLI `--limit` cap kept articles (default `25`).
4250

4351
**`json_state` / `xhr_articles` Limitations:** both scrapers require discoverable arrays of hashes containing clear `title` and `url` fields. Minified or
4452
obfuscated payloads, heavily encoded values, or blobs that require executing embedded functions are ignored. `xhr_articles` only sees JSON bodies captured during the current Botasaurus browser scrape (capped by the scrape API).

src/content/docs/ruby-gem/reference/selectors.mdx

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,11 @@ At a minimum, you need an `items` selector to define the list of articles and a
2828

2929
## Automatic Item Enhancement
3030

31-
To simplify configuration, `html2rss` can automatically extract the `title`, `url`, and `image` from each item. This feature is enabled by default.
31+
With `enhance: true` (the default), `html2rss` fills **missing** fields on each matched item from the card HTML: `title`, `url`, `description`, `published_at`, `image`, `categories`, and enclosures. Explicit selectors still win when they produce a value.
32+
33+
Description leftover text drops listing chrome (CTA lines such as “read more”, date-shaped chips, type chips, title echoes, section names). When the `items` selector is a heading or wrapping link with no date or description, extraction can climb to the parent card — it aborts if that parent looks like a listing (multiple headings or article links).
34+
35+
Capture emits `items` + `enhance: true` only. Keep `enhance: true` unless page chrome leaks into items.
3236

3337
<Code
3438
code={`

0 commit comments

Comments
 (0)