feat(tools): add URLReadTool for reading arbitrary URLs (#6834)

* feat(tools): add URLReadTool for reading arbitrary URLs

FileReadTool is confined to the local filesystem, so there was no way for
an agent to read a document that lives behind an http(s) URL. Rather than
adding a flag to FileReadTool, this adds a separate tool: granting it
grants network egress to addresses an LLM picks at runtime, and that
should be a deliberate choice rather than a toggle on a filesystem tool.

URLReadTool fetches a URL and returns its content as text. PDF and DOCX
bodies have their text extracted, HTML is stripped to visible text, and
text-shaped types (plain text, Markdown, JSON, XML, YAML, CSV) are
decoded using the charset the server declares. Any other content type is
refused rather than returned as base64, keeping the output text-only.

Requests reuse the existing SSRF protections in security/safe_requests:
validate_url resolves every hostname and rejects private, loopback,
link-local and reserved addresses (covering cloud metadata endpoints),
and safe_get never auto-follows redirects, revalidating each hop and
dropping credentials on cross-origin ones. Resolving before validating
also normalizes encoded forms, so http://2130706433/ is rejected as
127.0.0.1 without needing a string blocklist.

Adds safe_get_bounded on top of that, which streams the body and
abandons it once it crosses max_bytes. The cap counts decoded bytes,
which is what a compressed response expands into -- Content-Length
describes the wire size and cannot bound that. It also closes the
redirect hops, which stream=True would otherwise leave holding their
connections.

Two risks are documented rather than closed. Validation resolves the
hostname and requests resolves it again to connect, so DNS rebinding
remains possible; closing it needs the connection pinned to the
validated address, which would change behavior for all existing
safe_get callers. And the returned text is untrusted remote content
entering an agent's context, which input validation cannot address.

Also fixes a temp file leak in PDFLoader, which reached the same
pymupdf-from-URL path. It wrote downloads to NamedTemporaryFile with
delete=False and never unlinked them, so every PDF ingested from a URL
left a file behind. It now opens from memory, the way URLReadTool does,
which removes the leak by construction instead of relying on cleanup on
each error path; its doc.close() also moves into a finally so a failure
mid-extraction still releases the handle. PDFLoader had no test file, so
this adds one covering both paths plus a regression test asserting no
temp file is created.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): address review feedback on URLReadTool

Bound start_line and line_count with ge=1 in the args schema. Both were
unbounded, and _window computes stop as start + line_count, so a negative
line_count reached islice, which rejects a negative stop. The windowing
runs outside the tool's error handling, so that escaped _run as a raw
ValueError instead of an error string. BaseTool.run validates kwargs
against args_schema, so the constraint refuses the value before any
request is made. _window also clamps its own bounds now, so it cannot
raise if called directly.

Bound the PDFLoader download with safe_get_bounded. The body is held in
memory for the whole extraction, so it needed a ceiling; it defaults to
50 MiB and takes a max_bytes kwarg to load() for callers ingesting
larger documents.

Patch the loader's own seam in its tests rather than requests.get. Both
safe_get and safe_get_bounded resolve the hostname before requesting, so
the previous mocks made the tests depend on DNS for example.com and fail
in a network-isolated runner for reasons unrelated to the loader.

Exercise the content-type fallback through run() rather than asserting on
_resolve_kind, so the tests survive a refactor of the classification
internals, and cover the octet-stream PDF, missing-type, query-string and
unknown-extension cases as observable behavior.

Add docstrings to the new tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(tools): close streamed redirect hops and widen type fallback

Four findings from the Copilot and Cursor reviews.

safe_get leaked its accumulated hops on every failure path. It closed the
response it was about to abandon but not the ones already in history, and
a caller handed an exception has no handle on them -- under stream=True
each holds its connection until its body is read or closed. The loop now
closes history before re-raising. Hops are still the caller's on success,
where they arrive via response.history.

safe_get_bounded rejected a non-positive max_bytes only after issuing the
request, and then reported it as an oversized body. It now fails before
the request. Its oversized-body error also named the requested URL rather
than the one that served the body, which differ after a redirect.

The content-type fallback consulted only the final URL for an extension,
so a .pdf link redirecting to an extensionless CDN or presigned path was
refused even though the requested URL identified the type. It now checks
the final URL first, then the requested one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: joaomdmoura

lib/crewai-tools/src/crewai_tools/__init__.py

@@ -209,6 +209,7 @@
 )
 from crewai_tools.tools.tavily_search_tool.tavily_search_tool import TavilySearchTool
 from crewai_tools.tools.txt_search_tool.txt_search_tool import TXTSearchTool
+from crewai_tools.tools.url_read_tool.url_read_tool import URLReadTool
 from crewai_tools.tools.vision_tool.vision_tool import VisionTool
 from crewai_tools.tools.wait_tool.wait_tool import WaitTool
 from crewai_tools.tools.weaviate_tool.vector_search import WeaviateVectorSearchTool
@@ -327,6 +328,7 @@
     "TavilyGetResearchTool",
     "TavilyResearchTool",
     "TavilySearchTool",
+    "URLReadTool",
     "VisionTool",
     "WaitTool",
     "WeaviateVectorSearchTool",

lib/crewai-tools/src/crewai_tools/rag/loaders/pdf_loader.py

@@ -2,13 +2,17 @@
 
 import os
 from pathlib import Path
-import tempfile
 from typing import Any
 from urllib.parse import urlparse
 
 from crewai_tools.rag.base_loader import BaseLoader, LoaderResult
 from crewai_tools.rag.source_content import SourceContent
-from crewai_tools.security.safe_requests import safe_get
+from crewai_tools.security.safe_requests import safe_get_bounded
+
+
+# Remote PDFs are held in memory for the whole extraction, so the download needs
+# a ceiling. Override per call with the ``max_bytes`` kwarg to ``load``.
+DEFAULT_MAX_PDF_BYTES = 50 * 1024 * 1024
 
 
 class PDFLoader(BaseLoader):
@@ -24,18 +28,24 @@ def _is_url(path: str) -> bool:
             return False
 
     @staticmethod
-    def _download_from_url(url: str, kwargs: dict[str, Any]) -> str:
-        """Download PDF from a URL to a temporary file and return its path.
+    def _fetch_from_url(url: str, kwargs: dict[str, Any]) -> bytes:
+        """Download a PDF from a URL and return its bytes.
+
+        The content stays in memory rather than going to a temporary file: the
+        whole body has to be buffered either way, and a temp file would need
+        unlinking on every error path to avoid leaving files behind. Because it
+        is held in memory, the download is capped.
 
         Args:
             url: The URL to download from.
-            kwargs: Optional dict that may contain custom headers.
+            kwargs: Optional dict that may contain custom ``headers`` and a
+                ``max_bytes`` ceiling for the download.
 
         Returns:
-            Path to the temporary file containing the PDF.
+            The raw PDF content.
 
         Raises:
-            ValueError: If the download fails.
+            ValueError: If the download fails or exceeds the size ceiling.
         """
         headers = kwargs.get(
             "headers",
@@ -46,12 +56,13 @@ def _download_from_url(url: str, kwargs: dict[str, Any]) -> str:
         )
 
         try:
-            response = safe_get(url, headers=headers, timeout=30)
-            response.raise_for_status()
-
-            with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as temp_file:
-                temp_file.write(response.content)
-                return temp_file.name
+            body, _content_type, _final_url = safe_get_bounded(
+                url,
+                max_bytes=kwargs.get("max_bytes", DEFAULT_MAX_PDF_BYTES),
+                headers=headers,
+                timeout=30,
+            )
+            return body
         except Exception as e:
             raise ValueError(f"Failed to download PDF from {url}: {e!s}") from e
 
@@ -93,21 +104,25 @@ def load(self, source: SourceContent, **kwargs: Any) -> LoaderResult:  # type: i
 
         try:
             if is_url:
-                local_path = self._download_from_url(file_path, kwargs)
-                doc = pymupdf.open(local_path)
+                doc = pymupdf.open(
+                    stream=self._fetch_from_url(file_path, kwargs), filetype="pdf"
+                )
             else:
                 if not os.path.isfile(file_path):
                     raise FileNotFoundError(f"PDF file not found: {file_path}")
                 doc = pymupdf.open(file_path)
 
-            metadata["num_pages"] = len(doc)
-
-            for page_num, page in enumerate(doc, 1):
-                page_text = page.get_text()
-                if page_text.strip():
-                    text_content.append(f"Page {page_num}:\n{page_text}")
-
-            doc.close()
+            # Closed in a finally so a failure mid-extraction still releases the
+            # document handle.
+            try:
+                metadata["num_pages"] = len(doc)
+
+                for page_num, page in enumerate(doc, 1):
+                    page_text = page.get_text()
+                    if page_text.strip():
+                        text_content.append(f"Page {page_num}:\n{page_text}")
+            finally:
+                doc.close()
         except FileNotFoundError:
             raise
         except Exception as e:

lib/crewai-tools/src/crewai_tools/security/safe_requests.py

@@ -11,6 +11,7 @@
 
 
 _REDIRECT_STATUS_CODES = {301, 302, 303, 307, 308}
+_STREAM_CHUNK_SIZE = 65536
 _SENSITIVE_HEADER_NAMES = {
     "authorization",
     "cookie",
@@ -49,40 +50,124 @@ def _strip_cross_origin_credentials(request_kwargs: dict[str, Any]) -> dict[str,
 
 
 def safe_get(url: str, *, max_redirects: int = 10, **kwargs: Any) -> requests.Response:
-    """GET a URL while validating each redirect target before following it."""
+    """GET a URL while validating each redirect target before following it.
+
+    On success the hops are attached to the returned response's ``history`` and
+    are the caller's to close. On failure they are closed here: a caller given
+    an exception has no handle on them, and a streamed hop holds its connection
+    until its body is read or closed.
+    """
     current_url = validate_url(url)
     request_kwargs = {**kwargs, "allow_redirects": False}
     timeout = request_kwargs.pop("timeout", 30)
     history: list[requests.Response] = []
     redirects_followed = 0
 
-    while True:
-        response = requests.get(current_url, timeout=timeout, **request_kwargs)
-        if (
-            response.status_code not in _REDIRECT_STATUS_CODES
-            or "Location" not in response.headers
-        ):
-            response.history = history
-            return response
-
-        if redirects_followed >= max_redirects:
-            response.close()
-            raise ValueError(f"Too many redirects while fetching URL: {url}")
-
-        location = response.headers.get("Location")
-        if not location:
-            response.history = history
-            return response
-
-        try:
-            redirect_url = validate_url(urljoin(response.url, location))
-        except ValueError:
-            response.close()
-            raise
-
-        if not _same_origin(current_url, redirect_url):
-            request_kwargs = _strip_cross_origin_credentials(request_kwargs)
-
-        history.append(response)
-        current_url = redirect_url
-        redirects_followed += 1
+    try:
+        while True:
+            response = requests.get(current_url, timeout=timeout, **request_kwargs)
+            if (
+                response.status_code not in _REDIRECT_STATUS_CODES
+                or "Location" not in response.headers
+            ):
+                response.history = history
+                return response
+
+            if redirects_followed >= max_redirects:
+                response.close()
+                raise ValueError(f"Too many redirects while fetching URL: {url}")
+
+            location = response.headers.get("Location")
+            if not location:
+                response.history = history
+                return response
+
+            try:
+                redirect_url = validate_url(urljoin(response.url, location))
+            except ValueError:
+                response.close()
+                raise
+
+            if not _same_origin(current_url, redirect_url):
+                request_kwargs = _strip_cross_origin_credentials(request_kwargs)
+
+            history.append(response)
+            current_url = redirect_url
+            redirects_followed += 1
+    except BaseException:
+        for hop in history:
+            hop.close()
+        raise
+
+
+def safe_get_bounded(
+    url: str,
+    *,
+    max_bytes: int,
+    timeout: float | tuple[float, float] = 30,
+    headers: dict[str, str] | None = None,
+    max_redirects: int = 10,
+) -> tuple[bytes, str, str]:
+    """GET a URL through :func:`safe_get`, refusing bodies over *max_bytes*.
+
+    The body is streamed and abandoned as soon as it crosses the limit, so an
+    oversized response costs one chunk of memory instead of all of it. The cap
+    counts decoded bytes, which is what a compressed response expands into --
+    ``Content-Length`` describes the wire size and cannot bound that.
+
+    Args:
+        url: The URL to fetch.
+        max_bytes: Largest body to accept, in decoded bytes.
+        timeout: Request timeout, passed through to requests.
+        headers: Request headers.
+        max_redirects: Hops to follow before giving up.
+
+    Returns:
+        A ``(body, content_type, final_url)`` tuple, where *final_url* is the
+        last validated URL in the redirect chain.
+
+    Raises:
+        ValueError: If *max_bytes* is not positive, URL validation fails, the
+            redirect chain is too long, or the body exceeds *max_bytes*.
+        requests.RequestException: If the request fails or returns an error
+            status.
+    """
+    if max_bytes <= 0:
+        raise ValueError(f"max_bytes must be positive, got {max_bytes}.")
+
+    response = safe_get(
+        url,
+        max_redirects=max_redirects,
+        headers=headers,
+        timeout=timeout,
+        stream=True,
+    )
+    try:
+        response.raise_for_status()
+
+        chunks: list[bytes] = []
+        total = 0
+        for chunk in response.iter_content(chunk_size=_STREAM_CHUNK_SIZE):
+            if not chunk:
+                continue
+            total += len(chunk)
+            if total > max_bytes:
+                # Names the URL that served the body, which after a redirect is
+                # not the one that was requested.
+                raise ValueError(
+                    f"Response body from '{response.url}' exceeds the "
+                    f"{max_bytes} byte limit."
+                )
+            chunks.append(chunk)
+
+        return (
+            b"".join(chunks),
+            response.headers.get("Content-Type", ""),
+            response.url,
+        )
+    finally:
+        # Under stream=True each hop holds its connection until the body is read,
+        # so the redirects need closing too, not just the response we return.
+        for hop in response.history:
+            hop.close()
+        response.close()

lib/crewai-tools/src/crewai_tools/tools/__init__.py

@@ -196,6 +196,7 @@
 )
 from crewai_tools.tools.tavily_search_tool.tavily_search_tool import TavilySearchTool
 from crewai_tools.tools.txt_search_tool.txt_search_tool import TXTSearchTool
+from crewai_tools.tools.url_read_tool.url_read_tool import URLReadTool
 from crewai_tools.tools.vision_tool.vision_tool import VisionTool
 from crewai_tools.tools.wait_tool.wait_tool import WaitTool
 from crewai_tools.tools.weaviate_tool.vector_search import WeaviateVectorSearchTool
@@ -310,6 +311,7 @@
     "TavilyGetResearchTool",
     "TavilyResearchTool",
     "TavilySearchTool",
+    "URLReadTool",
     "VisionTool",
     "WaitTool",
     "WeaviateVectorSearchTool",

lib/crewai-tools/src/crewai_tools/tools/url_read_tool/__init__.py

@@ -0,0 +1,4 @@
+from crewai_tools.tools.url_read_tool.url_read_tool import URLReadTool
+
+
+__all__ = ["URLReadTool"]