fix: Adapt to CrawlerMonitor constructor change #1106
Conversation
WalkthroughThe changes simplify the instantiation of the Changes
Poem
Tip ⚡️ Faster reviews with caching
Enjoy the performance boost—your workflow just got faster. ✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
The upstream commit 1630fbd (2025-03-12) introduced a new TerminalUI and modified the CrawlerMonitor. This change updates the codebase to align with the new CrawlerMonitor constructor interface, which no longer requires the `max_visible_rows` and `display_mode` parameters.
devxpain
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (3)
deploy/docker/c4ai-code-context.md (3)
8917-8921: Use default CrawlerMonitor constructor
8932-8936: Use default CrawlerMonitor constructor
8950-8954: Use default CrawlerMonitor constructor
🧹 Nitpick comments (3)
docs/md_v2/advanced/multi-url-crawling.md (3)
122-124: Remove unusedDisplayModeimport and clarify default behavior.
Since you’ve removed the explicit constructor args,DisplayModeis imported but no longer used. It would be clearer to drop it:-from crawl4ai import CrawlerMonitor, DisplayMode +from crawl4ai import CrawlerMonitorAlso consider stating what the default
display_modeandmax_visible_rowsvalues are so readers know the out-of-the-box behavior.
151-152: EnsureCrawlerMonitoris in scope for the MemoryAdaptiveDispatcher example.
You’re passingmonitor=CrawlerMonitor()without importing it in this snippet. To keep examples copy-pasteable, either add:from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher +from crawl4ai import CrawlerMonitoror qualify the class (
crawl4ai.CrawlerMonitor()).
306-307: ImportCrawlerMonitorin the semaphore-based snippet.
Thecrawl_with_semaphoreexample usesmonitor=CrawlerMonitor()but lacks the corresponding import. Add at the top of that block:+from crawl4ai import CrawlerMonitorso readers don’t hit a NameError when following the guide.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
deploy/docker/c4ai-code-context.md(4 hunks)deploy/docker/c4ai-doc-context.md(4 hunks)docs/examples/dispatcher_example.py(4 hunks)docs/examples/proxy_rotation_demo.py(1 hunks)docs/md_v2/advanced/multi-url-crawling.md(4 hunks)tests/async/test_dispatchers.py(1 hunks)tests/memory/test_dispatcher_stress.py(0 hunks)
💤 Files with no reviewable changes (1)
- tests/memory/test_dispatcher_stress.py
🧰 Additional context used
🧬 Code Graph Analysis (2)
docs/examples/dispatcher_example.py (1)
crawl4ai/components/crawler_monitor.py (1)
CrawlerMonitor(350-753)
docs/examples/proxy_rotation_demo.py (1)
crawl4ai/components/crawler_monitor.py (1)
CrawlerMonitor(350-753)
🪛 Ruff (0.8.2)
docs/examples/proxy_rotation_demo.py
106-106: Local variable monitor is assigned to but never used
Remove assignment to unused variable monitor
(F841)
🔇 Additional comments (8)
tests/async/test_dispatchers.py (1)
156-156: LGTM! Constructor parameter update is correct.The
CrawlerMonitorinstantiation has been correctly updated to use the default constructor without parameters, aligning with the upstream constructor interface changes.docs/examples/dispatcher_example.py (4)
26-27: LGTM! Constructor parameter update is correct.The
CrawlerMonitorinstantiation has been correctly updated to use the default constructor without parameters, aligning with the upstream constructor interface changes.
45-46: LGTM! Constructor parameter update is correct.The
CrawlerMonitorinstantiation has been correctly updated to use the default constructor without parameters, aligning with the upstream constructor interface changes.
60-61: LGTM! Constructor parameter update is correct.The
CrawlerMonitorinstantiation has been correctly updated to use the default constructor without parameters, aligning with the upstream constructor interface changes.
78-79: LGTM! Constructor parameter update is correct.The
CrawlerMonitorinstantiation has been correctly updated to use the default constructor without parameters, aligning with the upstream constructor interface changes.deploy/docker/c4ai-code-context.md (1)
8899-8902: Use default CrawlerMonitor constructor
Removed explicit parameters and now instantiateCrawlerMonitor()with no arguments to match the updated constructor signature from upstream.deploy/docker/c4ai-doc-context.md (2)
6656-6656: Documentation update correctly reflects API change.The change properly updates the CrawlerMonitor instantiation by removing parameters that are no longer supported in the constructor, aligning with the upstream changes mentioned in the PR description.
6723-6723: Constructor parameter removal accurately implemented.This change is consistent with the CrawlerMonitor API update, correctly removing the unsupported parameters from the dispatcher configuration example.
| max_visible_rows=10, | ||
| display_mode=DisplayMode.DETAILED | ||
| ) | ||
| monitor = CrawlerMonitor() |
There was a problem hiding this comment.
Unused variable detected.
The monitor variable is created but never used in this function. Either pass it to the MemoryAdaptiveDispatcher by uncommenting line 112 or remove the variable assignment completely.
Either:
- monitor = CrawlerMonitor()Or:
monitor = CrawlerMonitor()
dispatcher = MemoryAdaptiveDispatcher(
memory_threshold_percent=80.0,
check_interval=0.5,
max_session_permit=1, #len(proxies), # Match concurrent sessions to proxy count
- # monitor=monitor
+ monitor=monitor
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| monitor = CrawlerMonitor() | |
| dispatcher = MemoryAdaptiveDispatcher( | |
| memory_threshold_percent=80.0, | |
| check_interval=0.5, | |
| max_session_permit=1, # Match concurrent sessions to proxy count | |
| # monitor=monitor | |
| ) |
| monitor = CrawlerMonitor() | |
| monitor = CrawlerMonitor() | |
| dispatcher = MemoryAdaptiveDispatcher( | |
| memory_threshold_percent=80.0, | |
| check_interval=0.5, | |
| max_session_permit=1, # Match concurrent sessions to proxy count | |
| monitor=monitor | |
| ) |
🧰 Tools
🪛 Ruff (0.8.2)
106-106: Local variable monitor is assigned to but never used
Remove assignment to unused variable monitor
(F841)
The upstream commit 1630fbd (2025-03-12) introduced a new TerminalUI and modified the CrawlerMonitor.
This change updates the codebase to align with the new CrawlerMonitor constructor interface, which no longer requires the
max_visible_rowsanddisplay_modeparameters.Summary
Fix the error below
How Has This Been Tested?
$ uv run docs/examples/dispatcher_example.py
Summary by CodeRabbit
Documentation
Tests