Skip to content

Catch agnostic HTTP error types in prefect and swap the pagination raise #24284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mwdd146980 wants to merge 1 commit into
base: mwdd146980/httpx-migration-base
Choose a base branch
from
+37 −5
Open

Catch agnostic HTTP error types in prefect and swap the pagination raise #24284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions prefect/datadog_checks/prefect/check.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@

from datadog_checks.base import AgentCheck
from datadog_checks.base.utils.common import pattern_filter
from datadog_checks.base.utils.http_exceptions import (
HTTPConnectionError,
HTTPInvalidURLError,
HTTPStatusError,
HTTPTimeoutError,
)

from .config_models import ConfigMixin
from .metrics import METRICS_SPEC
Expand Down Expand Up @@ -666,7 +672,17 @@ class PrefectClient:
"""HTTP client wrapping GET/POST requests and pagination for the Prefect API."""

def __init__(self, url: str, http: RequestsWrapper, log: CheckLoggingAdapter):
self.http_exceptions = (HTTPError, InvalidURL, ConnectionError, Timeout, JSONDecodeError)
self.http_exceptions = (

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required changelog entry

The root AGENTS.md requires a changelog entry for shipped Agent changes, including Python sources under datadog_checks/, but this commit changes prefect/datadog_checks/prefect/check.py without adding a prefect/changelog.d/<PR_NUMBER>.fixed entry. That leaves this fix out of release notes/validation, so add the entry after the PR number is known.

Useful? React with 👍 / 👎.

HTTPError,
InvalidURL,
ConnectionError,
Timeout,
JSONDecodeError,
HTTPStatusError,
HTTPInvalidURLError,
HTTPConnectionError,
HTTPTimeoutError,
Comment on lines +681 to +684

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Catch HTTPRequestError for httpx2 request failures

When users opt into use_httpx2, HTTPX2Wrapper maps httpx2.LocalProtocolError and generic RequestError to HTTPRequestError (datadog_checks_base/datadog_checks/base/utils/httpx2.py:129), but this tuple only adds the narrower status/URL/connection/timeout classes. In those request-error cases the get/post exceptions bypass the pagination/check handlers and can abort the check instead of logging incomplete data, so include the library-agnostic base request/error type here.

Useful? React with 👍 / 👎.

)
self.url = url
parsed_url = urlparse(url)
self.base_url_scheme = parsed_url.scheme
Expand Down Expand Up @@ -696,7 +712,7 @@ def post(self, endpoint: str, payload: dict | None = None) -> Any:
def check_pagination_url(self, next_page: str):
parsed_next_page = urlparse(next_page)
if parsed_next_page.scheme != self.base_url_scheme or parsed_next_page.netloc != self.base_url_netloc:
raise InvalidURL(f'Invalid next_page URL with unexpected host: {next_page}')
raise HTTPInvalidURLError(f'Invalid next_page URL with unexpected host: {next_page}')

def paginate_filter(self, endpoint: str, payload: dict | None = None) -> list[dict]:
"""Implements pagination for /filter endpoints using limit/offset loop."""
Expand Down
18 changes: 17 additions & 1 deletion prefect/tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,13 +133,29 @@ def mock_prefect_client(mocker):

from requests.exceptions import ConnectionError, HTTPError, InvalidURL, Timeout

from datadog_checks.base.utils.http_exceptions import (
HTTPConnectionError,
HTTPInvalidURLError,
HTTPStatusError,
HTTPTimeoutError,
)
from datadog_checks.prefect.check import PrefectClient

get_responses = _load_fixture("get_metrics.json")
post_responses = _load_fixture("post_metrics.json")

mock_client = create_autospec(PrefectClient, instance=True)
mock_client.http_exceptions = (HTTPError, InvalidURL, ConnectionError, Timeout, JSONDecodeError)
mock_client.http_exceptions = (
HTTPError,
InvalidURL,
ConnectionError,
Timeout,
JSONDecodeError,
HTTPStatusError,
HTTPInvalidURLError,
HTTPConnectionError,
HTTPTimeoutError,
)

mock_client.get.side_effect = lambda endpoint, **kwargs: get_responses[endpoint]
mock_client.paginate_filter.side_effect = lambda endpoint, payload=None: post_responses[endpoint]
Expand Down
4 changes: 2 additions & 2 deletions prefect/tests/test_unit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
from unittest.mock import Mock

import pytest
from requests.exceptions import InvalidURL

from datadog_checks.base.stubs.aggregator import AggregatorStub
from datadog_checks.base.utils.http_exceptions import HTTPInvalidURLError
from datadog_checks.dev.utils import get_metadata_metrics
from datadog_checks.prefect import PrefectCheck
from datadog_checks.prefect.check import PrefectClient
Expand Down Expand Up @@ -632,7 +632,7 @@ def test_paginate_events_rejects_external_next_page():
log.error.assert_called_once()
args = log.error.call_args[0]
assert args[0] == "Could not collect next page of events: %s, data is incomplete"
assert isinstance(args[1], InvalidURL)
assert isinstance(args[1], HTTPInvalidURLError)
assert str(args[1]) == "Invalid next_page URL with unexpected host: http://attacker.example/evil"


Expand Down
Loading