Commit ebbf0e0
* feat: add OpenAI Batch API support for SmartScraperMultiGraph (#1036)
Add SmartScraperMultiBatchGraph that uses the OpenAI Batch API for LLM
calls, providing ~50% cost savings when real-time results aren't needed.
Key features:
- SmartScraperMultiBatchGraph: 3-phase pipeline (fetch/parse → batch
submit → merge) that separates HTML fetching from LLM generation
- BatchGenerateAnswerNode: collects prompts from all URLs and submits
them as a single OpenAI Batch API request
- utils/batch_api.py: helpers for creating, polling, and retrieving
batch results with doc_id → URL mapping
- Per-document error handling: partial failures don't break the batch
- Configurable polling interval and max wait time
- OpenAI-only validation (rejects non-OpenAI providers gracefully)
- Results sorted by custom_id for consistent ordering
- 18 unit tests with 100% pass rate
Usage:
graph = SmartScraperMultiBatchGraph(
prompt='Extract key points',
source=['https://url1.com', 'https://url2.com'],
config={'llm': {'model': 'openai/gpt-4o-mini'}}
)
result = graph.run()
Closes #1036
* ci(release): 1.60.0-beta.2 [skip ci]
## [1.60.0-beta.2](v1.60.0-beta.1...v1.60.0-beta.2) (2026-02-24)
### Features
* add OpenAI Batch API support for SmartScraperMultiGraph ([#1036](#1036)) ([9d4eba1](9d4eba1))
* fix: update broken test imports to match current API
- Replace removed ScrapeGraph with SmartScraperGraph in scrape_graph_test.py
- Replace renamed convert_to_csv/convert_to_json with export_to_csv/export_to_json in xml_scraper_openai_test.py
* ci(release): 1.60.0-beta.3 [skip ci]
## [1.60.0-beta.3](v1.60.0-beta.2...v1.60.0-beta.3) (2026-03-15)
### Bug Fixes
* update broken test imports to match current API ([536e5ad](536e5ad))
* ci(release): 1.76.0-beta.1 [skip ci]
## [1.76.0-beta.1](v1.75.1...v1.76.0-beta.1) (2026-04-07)
### Features
* add OpenAI Batch API support for SmartScraperMultiGraph ([#1036](#1036)) ([9d4eba1](9d4eba1))
### Bug Fixes
* update broken test imports to match current API ([536e5ad](536e5ad))
### CI
* **release:** 1.60.0-beta.2 [skip ci] ([54d1473](54d1473)), closes [#1036](#1036)
* **release:** 1.60.0-beta.3 [skip ci] ([637c696](637c696))
* reduce GitHub Actions costs by ~85% on PRs ([403080a](403080a))
* ci(release): 2.1.0-beta.1 [skip ci]
## [2.1.0-beta.1](v2.0.0...v2.1.0-beta.1) (2026-04-19)
### Features
* add OpenAI Batch API support for SmartScraperMultiGraph ([#1036](#1036)) ([9d4eba1](9d4eba1))
### Bug Fixes
* update broken test imports to match current API ([536e5ad](536e5ad))
### CI
* **release:** 1.60.0-beta.2 [skip ci] ([54d1473](54d1473)), closes [#1036](#1036)
* **release:** 1.60.0-beta.3 [skip ci] ([637c696](637c696))
* **release:** 1.76.0-beta.1 [skip ci] ([35ec272](35ec272)), closes [#1036](#1036) [#1036](#1036)
* Add Italian README translation and fix outdated links (#1070)
* fix(batch): use langchain_core.prompts for PromptTemplate import
langchain 1.x removed langchain.prompts; import from langchain_core
to fix ModuleNotFoundError causing all test collection to fail.
* ci(release): 2.2.0-beta.1 [skip ci]
## [2.2.0-beta.1](v2.1.1...v2.2.0-beta.1) (2026-05-16)
### Features
* add OpenAI Batch API support for SmartScraperMultiGraph ([#1036](#1036)) ([9d4eba1](9d4eba1))
### Bug Fixes
* update broken test imports to match current API ([536e5ad](536e5ad))
* **batch:** use langchain_core.prompts for PromptTemplate import ([24127da](24127da))
### CI
* **release:** 1.60.0-beta.2 [skip ci] ([54d1473](54d1473)), closes [#1036](#1036)
* **release:** 1.60.0-beta.3 [skip ci] ([637c696](637c696))
* **release:** 1.76.0-beta.1 [skip ci] ([35ec272](35ec272)), closes [#1036](#1036) [#1036](#1036)
* **release:** 2.1.0-beta.1 [skip ci] ([a2ea9eb](a2ea9eb)), closes [#1036](#1036) [#1036](#1036) [#1036](#1036) [#1036](#1036)
---------
Co-authored-by: MrAliHasan <mrali.hassan997@gmail.com>
Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net>
Co-authored-by: khadyottakale <khadyottakale@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gabriele Maria Bellavia <gabriele.bellavia.m@gmail.com>
1 parent 4dea92e commit ebbf0e0
10 files changed
Lines changed: 1484 additions & 10 deletions
File tree
- docs
- scrapegraphai
- graphs
- nodes
- utils
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
1 | 22 | | |
2 | 23 | | |
3 | 24 | | |
| |||
17 | 38 | | |
18 | 39 | | |
19 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
20 | 60 | | |
21 | 61 | | |
22 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
14 | | - | |
15 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | | - | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
162 | 163 | | |
163 | 164 | | |
164 | 165 | | |
165 | | - | |
| 166 | + | |
166 | 167 | | |
167 | 168 | | |
168 | 169 | | |
| |||
186 | 187 | | |
187 | 188 | | |
188 | 189 | | |
189 | | - | |
| 190 | + | |
190 | 191 | | |
191 | 192 | | |
192 | 193 | | |
| |||
195 | 196 | | |
196 | 197 | | |
197 | 198 | | |
198 | | - | |
| 199 | + | |
199 | 200 | | |
200 | 201 | | |
201 | 202 | | |
| |||
208 | 209 | | |
209 | 210 | | |
210 | 211 | | |
211 | | - | |
| 212 | + | |
212 | 213 | | |
213 | 214 | | |
214 | 215 | | |
| |||
0 commit comments