You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update timeout chain and botasaurus wait caps (#1246)
* docs: align feed-directory contracts with registry sync
* chore: format monitoring.mdx for Prettier lint
Unblocks lint CI on docs PRs; formatting drift on main.
* docs: update timeout chain and botasaurus wait caps
Copy file name to clipboardExpand all lines: AGENTS.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,8 +14,8 @@ What this repo owns:
14
14
What this repo does not own:
15
15
16
16
- runtime extractor behavior and CLI semantics (`html2rss/`)
17
-
-catalog metadata, YAML configs, or catalog serialization (`html2rss-configs/` → `Html2rss::Configs::Catalog`)
18
-
- catalog HTTP API (`html2rss-web/` → `GET /api/v1/configs`)
17
+
-feed YAML and signed registry bundle publishing (`html2rss-configs/` → `configs/`, `tool/registry-build`)
18
+
- catalog HTTP API and registry sync (`html2rss-web/` → `GET /api/v1/configs`, `Registry::Sync`)
19
19
20
20
When docs describe behavior from other repos, treat those repos as source-of-truth and update docs to match them.
21
21
@@ -30,9 +30,10 @@ Before substantial edits, state cross-repo context in your notes:
30
30
Common contracts:
31
31
32
32
- Feed Directory browse data comes from `{instance}/api/v1/configs` on a running `html2rss-web` instance (see OpenAPI in `html2rss-web`).
33
+
- Catalog wire rows include `id`, `path`, `directory`, `channel`, `parameters`, plus `source` (`registry` | `local`) and `registry` (registry id for bundle entries). The browse UI ignores `source`/`registry` today — wire parsing stays in `adapters/catalog-api.ts`.
33
34
- Instance URL persistence: default public instance, `#!url=` hash deep link from the web app, browser localStorage, and filter state in URL query params (`q`, `topic`, `lang`, `sort`, `page`).
34
35
- Deep link from `html2rss-web`: `https://html2rss.github.io/feed-directory/#!url={encodedInstanceUrl}` must keep working.
35
-
- Catalog metadata in YAML (`directory.title`, `directory.summary`, `directory.topics`) is authored in `html2rss-configs` only.
36
+
- Catalog metadata in YAML (`directory.*`, `registry.id`) is authored in `html2rss-configs` only; instances load verified bundles via registry sync.
36
37
- Ruby gem docs should match `html2rss` behavior and CLI output.
37
38
- Web application docs should match `html2rss-web` behavior and published OpenAPI.
Copy file name to clipboardExpand all lines: src/content/docs/ruby-gem/guides/backward-compatibility.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ Upgrade notes for configs and integrators targeting gem **0.27.0** (current `mas
16
16
3.**Capture output** — `html2rss capture` and MCP `capture_config` emit an items selector plus `enhance: true` only (no per-field title/url/description selector soup). Capture stamps `strategy:` when AutoFallback selects a concrete transport.
17
17
4.**CLI `--explain`** — `html2rss auto --explain` and `html2rss capture --explain` print telemetry JSON on stderr; stdout stays the feed or YAML. Status includes `admission_drops` when Cleanup dropped items.
18
18
5.**`Html2rss.auto_feed_result`** — URL-only auto discovery returns `FeedResult` with the same `status` telemetry as config builds.
19
-
6.**`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..20`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
19
+
6.**`request.botasaurus` OpenAPI 2.0** — Keys must match the scrape-API `ScrapeRequest` (unknown keys rejected). Use `scroll: true` (not `scroll_to_bottom`), `window_size: { width:, height: }` (not a two-item array), and `wait_timeout_seconds` in `1..30`. See [Strategy](/ruby-gem/reference/strategy/#botasaurus).
20
20
7.**Botasaurus wire format** — Client expects OpenAPI 2.0 `ScrapeSuccess` / `ScrapeError` envelopes from the scrape API.
21
21
8.**`--limit` default** — Auto discovery default is `25` (`AutoSource::DEFAULT_LIMIT`).
22
22
@@ -107,7 +107,7 @@ Update any occurrences of `link`, `pubDate`, or `updated` in your `selectors` bl
107
107
When upgrading to **0.27.0** or later:
108
108
109
109
1.**MCP integrators**: Read tool results from the envelope `payload`; configure `BOTASAURUS_SCRAPER_URL` on the MCP process env.
110
-
2.**Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..20` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
110
+
2.**Align `request.botasaurus` with OpenAPI 2.0**: Use `scroll`, `{ width, height }` for `window_size`, and `wait_timeout_seconds` in `1..30` (see [Strategy](/ruby-gem/reference/strategy/#botasaurus)).
111
111
3.**Validate configurations**: Run `html2rss validate config.yml`.
Copy file name to clipboardExpand all lines: src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -63,6 +63,8 @@ Configure browser actions under `request.botasaurus`:
63
63
lang="yaml"
64
64
/>
65
65
66
+
`wait_timeout_seconds` applies to selector wait on the scraper **after** the browser is ready. Slow Chrome or Xvfb boot consumes the scrape **total** wall (`SCRAPE_TIMEOUT_SECONDS`, default **45s**), not this work budget (`SCRAPE_WORK_TIMEOUT_SECONDS`, default **30s**).
Copy file name to clipboardExpand all lines: src/content/docs/ruby-gem/reference/strategy.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,7 +41,7 @@ Requirements:
41
41
-`BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`)
42
42
- pin `strategy: botasaurus` when you want to skip Faraday and force this transport directly
43
43
44
-
html2rss enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries`and `wait_timeout_seconds`are clamped so upstream work fits the remaining budget.
44
+
html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits **total** scrape time (queue, boot, navigate, wait) from **work** time (navigate, selector wait, and scroll after the browser is ready). Feed YAML `wait_timeout_seconds`is validated against the work cap (`1..30` by default); the gem forwards `request.botasaurus` options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Faraday transport timeout for POST `/scrape` is the lesser of the remaining feed-build budget and `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` plus a small buffer (default **45s** scrape total, **47s** transport cap).
45
45
46
46
During the **browser** execution tier, the Botasaurus scrape API always captures JSON XHR/fetch response bodies (`xhr_responses`). html2rss forwards them as `Response#captured_responses` so AutoSource `xhr_articles` can extract articles without extra requests. The HTTP-request tier returns an empty capture list. Caps and filtering live in the scrape API (see its README); article-likeness filtering is a client concern.
-`max_retries` (`0..3`; omit to use the scrape-API default of `2`)
72
72
-`wait_for_selector` (string)
73
-
-`wait_timeout_seconds` (integer `1..20`; omit to use the scrape-API default of `15`)
73
+
-`wait_timeout_seconds` (integer `1..30`; omit to use the scrape-API default of `15`; counts against post-boot **work** budget on the scraper, not browser boot)
|`HTML2RSS_SECRET_KEY`| required in production; development/test gets a temporary default |
11
-
|`HTML2RSS_ACCESS_TOKEN`| token for the main feed-creation flow; paste the same value into the web UI when prompted |
12
-
|`HEALTH_CHECK_TOKEN`| optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
13
-
|`BUILD_TAG`| release metadata used in logs; published Docker images set this to the release version |
14
-
|`GIT_SHA`| deployed commit metadata used in logs; published Docker images set this to the released commit |
15
-
|`SENTRY_DSN`| optional; enables Sentry errors/logs when set |
16
-
|`BOTASAURUS_SCRAPER_URL`| optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass |
17
-
|`AUTO_SOURCE_ENABLED`|`true` by default in development/test, `false` otherwise |
|`HTML2RSS_SECRET_KEY`| required in production; development/test gets a temporary default |
11
+
|`HTML2RSS_ACCESS_TOKEN`| token for the main feed-creation flow; paste the same value into the web UI when prompted |
12
+
|`HEALTH_CHECK_TOKEN`| optional bearer token for authenticated `GET /api/v1/health`; not part of the first-run quickstart; use `/api/v1/health/ready` for standard readiness checks |
13
+
|`BUILD_TAG`| release metadata used in logs; published Docker images set this to the release version |
14
+
|`GIT_SHA`| deployed commit metadata used in logs; published Docker images set this to the released commit |
15
+
|`SENTRY_DSN`| optional; enables Sentry errors/logs when set |
16
+
|`BOTASAURUS_SCRAPER_URL`| optional; Botasaurus scrape API base URL (e.g. `http://botasaurus:4010`) for browser rendering and anti-bot bypass |
17
+
|`AUTO_SOURCE_ENABLED`|`true` by default in development/test, `false` otherwise |
|`RETRY_AFTER_TIMEOUT_SECONDS`| optional integer `>= 1`; default `300`; Retry-After header backoff duration returned on timeouts |
25
+
|`PORT`| app listen port; compose uses `4000`|
26
+
|`RACK_ENV`| Rack environment; quickstart uses `development`, production examples use `production`|
27
+
|`BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`| optional; Faraday client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api |
28
+
|`BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS`| optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api |
29
+
|`SCRAPE_WORK_TIMEOUT_SECONDS`| optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose |
30
+
|`RACK_TIMEOUT_SERVICE_TIMEOUT`| optional; total web-request timeout in seconds; default `55`|
31
+
|`HTML2RSS_TOTAL_TIMEOUT_SECONDS`| optional; internal gem feed-build timeout in seconds; default `50`|
29
32
30
33
## Timeout Chain
31
34
32
-
When fetching feeds from complex websites, multiple components work together. To prevent premature request termination, ensure your timeouts are configured in a "chain" where the outer layers wait longer than the inner layers.
35
+
When fetching feeds from complex websites, multiple components work together. Configure timeouts so outer layers wait longer than inner layers and the scraper exhausts its budget before the web tier aborts the request.
33
36
34
-
For an overall 30-second budget:
37
+
Default compose aligns **scrape total (45) ≤ feed build (50) ≤ Rack (55)**. The **work**budget (**30s**) applies on botasaurus-scrape-api only after the browser is ready; browser boot and queue time count against the **total** scrape wall (**45s**), not against `wait_timeout_seconds`.
35
38
36
-
1.**Botasaurus Scraper (`SCRAPE_TIMEOUT_SECONDS`)**: Set to **20s**. This is the time allowed for the browser to fetch and render the page.
37
-
2.**html2rss Gem (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: Set to **25s**. This allows the gem enough time to wait for Botasaurus and process the results.
38
-
3.**html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: Set to **30s**. This is the total time the web server will wait before killing the request.
Older environment-variable examples from previous docs revisions are obsolete. Use only the supported table above and the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md).
46
+
Set matching `SCRAPE_*` on botasaurus-scrape-api and `BOTASAURUS_SCRAPE_*` on html2rss-web when you override defaults. See the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md) for compose examples.
|`GET /api/v1/health`| bearer | Operator and uptime checks |
15
15
16
16
Set `HEALTH_CHECK_TOKEN`, then call the authenticated endpoint:
17
17
@@ -31,9 +31,9 @@ Set `SENTRY_DSN` to enable Issue capture in html2rss-web (Rack middleware and op
31
31
32
32
Use separate Sentry projects for html2rss-web and botasaurus-scrape-api. Never share a DSN.
33
33
34
-
| Env var | Service |
35
-
| --- | --- |
36
-
|`SENTRY_DSN`| html2rss-web |
34
+
| Env var | Service|
35
+
| -----------------------|--------------------- |
36
+
|`SENTRY_DSN`| html2rss-web|
37
37
|`BOTASAURUS_SENTRY_DSN`| botasaurus-scrape-api |
38
38
39
39
In `docker-compose.yml`, the botasaurus service maps `BOTASAURUS_SENTRY_DSN` into its own `SENTRY_DSN`, requires it at startup, and does not fall back to the web project's DSN.
0 commit comments