Skip to content

[AI-6576]: requests to httpx migration#22586

Closed
mwdd146980 wants to merge 27 commits into
masterfrom
mwdd146980/ai-6576
Closed

[AI-6576]: requests to httpx migration#22586
mwdd146980 wants to merge 27 commits into
masterfrom
mwdd146980/ai-6576

Conversation

@mwdd146980

@mwdd146980 mwdd146980 commented Feb 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR adds an optional httpx-backed HTTP client alongside the existing requests-based one and introduces shared types so the codebase can support both backends without changing call sites.

  • New module datadog_checks.base.utils.httpx

    • HTTPXWrapper – Same public API as RequestsWrapper (get/post/head/put/patch/delete/options_method, options, session, etc.).
    • HTTPXResponseAdapter – Wraps httpx.Response to match the current response surface (content, headers, encoding, iter_lines, raise_for_status, etc.).
    • Supports the same config surface (TLS, basic auth, Kerberos via httpx-gssapi, proxy, UDS, timeouts).
    • Connection/SSL errors are re-raised as the shared exception types below.

  • Shared exception and protocol types

    • datadog_checks.base.utils.http_exceptions – HTTPError and SSLError used by both wrappers and by checks so except clauses don’t depend on requests/httpx.
    • datadog_checks.base.utils.http_protocol – HTTPClientProtocol and HTTPResponseProtocol for type hints and future backends (e.g. async).
    • Protocol – HTTPClientProtocol describes the wrapper interface (get/post/…/options_method, options, session, handle_auth_token); HTTPResponseProtocol describes the response (content, headers, iter_lines, raise_for_status, etc.). Both wrappers implement these so type checkers and call sites see a single interface.
    • RequestsWrapper now maps requests connection/SSL errors to these types: connection errors → stdlib ConnectionError; SSL errors → http_exceptions.SSLError. ResponseWrapper.raise_for_status() raises http_exceptions.HTTPError instead of requests.HTTPError. Session handling respects an injected session for tests (e.g. session=make_mock_session()).

  • Base check opt-in

    • use_httpx (default false) in instance or init_config.
    • When true, AgentCheck.http returns HTTPXWrapper; otherwise RequestsWrapper.
    • self.http is typed as HTTPClientProtocol so both implementations are valid.

  • Fixture and mock changes

    • mock_http_response – Check-only: first argument must be a check class with get_http_handler; it patches that class and serves responses from a queue.
    • mock_http_client – New fixture; returns (client, enqueue) for tests without a check; inject client where the HTTP wrapper is created.
    • request_wrapper_mock docstring clarified for custom handler logic when the above two aren’t enough.

  • Dependencies

    • httpx and httpx-gssapi added to agent requirements and base optional deps; LICENSE-3rdparty.csv updated.

Integrations keep using self.http.get(url) etc.; no call-site changes. OpenMetrics/Prometheus wiring (e.g. get_http_handler on the check, scraper using it) and test migrations to the new fixtures are in the base-httpx-migration branch.

Motivation

RFC

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Feb 9, 2026 •
edited by datadog-official Bot
Loading

Copy link
Copy Markdown
Contributor

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

🧪 241 Tests failed

    TestShareLabels::test_metadata from test_compat_scraper.py (Fix with Cursor)

    TestShareLabels::test_share_labels from test_compat_scraper.py (Fix with Cursor)

test from test_counter_gauge.py (Datadog) (Fix with Cursor) 
mock_http_response requires a check class with get_http_handler as first argument (e.g. OpenMetricsBaseCheckV2, OpenMetricsBaseCheck)
View all

ℹ️ Info

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1cf48bc | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@github-actions

github-actions Bot commented Feb 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Major version bump
The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

@codecov

codecov Bot commented Feb 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 133 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.53%. Comparing base (de966db) to head (1cf48bc).
⚠️ Report is 29 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mwdd146980 mwdd146980 changed the title [AI-6576]: Migrate the http layer from requests to httpx phase 1 [AI-6576]: requests to httpx migration Feb 10, 2026
@mwdd146980 mwdd146980 self-assigned this Feb 10, 2026
@mwdd146980
AI-6576: Phase 1 – add HTTPXWrapper and switch AgentCheck.http to httpx
71cdbf6 
- Add httpx dependency and HTTPXResponseAdapter wrapping httpx.Response with
  the same surface as the requests response (content, headers, encoding,
  iter_lines, iter_content, raise_for_status, close).
- Introduce HTTPXWrapper with the same public API as RequestsWrapper (config,
  get/post/head/put/patch/delete/options_method, options dict). Supports
  basic auth, TLS via create_ssl_context, proxy/no_proxy, UDS, and
  auth_token handler. Other auth types (digest, Kerberos, NTLM, AWS) log a
  warning for now.
- Make AgentCheck.http return HTTPXWrapper instead of RequestsWrapper.
- Update test_activate to assert HTTPXWrapper; keep RequestsWrapper in tests
  that exercise it directly.

Prometheus mixin, OpenMetrics mixin, v2 scraper, and direct RequestsWrapper
usage are unchanged (Phases 2–4). Part of the migrate-requests-to-httpx
proposal.
@mwdd146980
Add changelog entry
0326fb3
@mwdd146980
Add httpx as requirement with command ddev dep freeze
884a05e
@mwdd146980
Fix HTTPX wrapper compatibility and add unit tests
3044ad5 
- HTTPXResponseAdapter: add json() for integrations that call response.json()
- HTTPXResponseAdapter: map httpx.HTTPStatusError to requests.HTTPError with response=self
- HTTPXWrapper: add session property (session.close()) for backward compatibility (e.g. cisco_aci)
- _make_httpx_auth: return (user, password) tuple for basic auth so options match tests/expectations
- Add test_httpx.py with unit tests for adapter, session, auth, and request/response paths
@mwdd146980
Revert httpx migration; return to planning phase (separate httpx.py +…
5e7576d 
… use_httpx)
@mwdd146980
Revert change in test_http.py for now as well
1d1d934
@mwdd146980
Add optional httpx-backed HTTP client (Phase 1)
d987fdd 
- Add datadog_checks.base.utils.httpx with HTTPXWrapper and HTTPXResponseAdapter
  (same public API as RequestsWrapper; http.py unchanged).
- Add use_httpx option (default false) to base check; when true, self.http
  returns HTTPXWrapper instead of RequestsWrapper.
- Add httpx dependency to datadog_checks_base.
- Add unit tests in tests/base/utils/http/test_httpx.py.
- Document use_httpx in docs/developer/base/http.md.
- Add RFC proposal docs/proposals/migrate-requests-to-httpx.md.
@mwdd146980
Sync licenses with ddev validate licenses --sync
6a85eb3
@mwdd146980
Fix auth token handling in HTTPXWrapper and expand test coverage
9b71aac 
- UDS path: run auth token retry (handle_auth_token + retry) so token
  refresh works for unix:// URLs like in RequestsWrapper.
- Non-UDS and UDS: pass headers per request via _request_headers() instead
  of at client creation so AuthTokenHeaderWriter updates to
  self.options['headers'] are used on retry after token refresh.
- Add tests for _make_httpx_auth, _parse_uds_url, adapter (iter_content,
  iter_lines delimiter, json, raise_for_status), wrapper (all HTTP methods,
  log_requests, no_proxy, persist=False, extra_headers, init options, UDS,
  session.close with client), and auth token retry (401 then success for
  both TCP and UDS). Coverage for httpx.py raised to ~89%.
@mwdd146980
Merge remote-tracking branch 'origin/master' into mwdd146980/ai-6576
ba0eaaa
@mwdd146980
Reformat with ddev test --fmt
d217518
@mwdd146980 mwdd146980 closed this Feb 11, 2026
@mwdd146980
Fix issue where params is never passed to httpx.Client.request() by i…
dd3df4a 
…ncluding params in the set of options forwarded into request_kwargs
@mwdd146980
Add HTTP client protocol, base check type hint, and test mock helpers
d8f8d3f 
- Introduce HTTPResponseProtocol, HTTPSessionLike, and HTTPClientProtocol
  in datadog_checks.base.utils.http_protocol for type-safe use of
  RequestsWrapper/HTTPXWrapper.
- Change AgentCheck.http return type from RequestsWrapper to
  HTTPClientProtocol (protocol only; behavior unchanged).
- Add HTTPResponseMock and RequestWrapperMock in datadog_checks.base.utils
  .http_mock so tests can mock check.http without depending on requests or
  httpx.
- Add tests for the protocol and mock helpers, and document usage in
  docs/developer/base/http.md.
@mwdd146980 mwdd146980 reopened this Feb 11, 2026
@mwdd146980
Add request_kwargs['timeout'] = timeout_httpx after building request_…
21a58a1 
…kwargs so that every client.request(..., **request_kwargs) receives the right timeout
@mwdd146980 mwdd146980 closed this Feb 11, 2026
@mwdd146980
Move HTTP mock helpers to datadog_checks_dev
4f6c8ed 
Move HTTPResponseMock and RequestWrapperMock from
datadog_checks.base.utils.http_mock to datadog_checks.dev.http so
they are not shipped with the base check. Add http_response_mock
and request_wrapper_mock pytest fixtures in the dev plugin. Update
base tests and docs to use datadog_checks.dev.http.
@mwdd146980 mwdd146980 reopened this Feb 11, 2026
@mwdd146980 mwdd146980 closed this Feb 11, 2026
@mwdd146980 mwdd146980 reopened this Feb 11, 2026
@mwdd146980
Merge protocol mocks into pytest fixtures; drop Session.get in mock_h…
6aef6bd 
…ttp_response

- mock_response: yield HTTPResponseMock (single path)
- mock_http_response: require check class as first arg, patch get_http_handler
  with RequestWrapperMock only; raise ValueError if no class given (no
  requests.Session.get)
- mock_http_response_per_endpoint: use mock_response for 404, add optional
  handler_target to patch get_http_handler by URL
- HTTPResponseMock: support content- and status_code-first calls,
  file_path/content/json_data, .text, .raw, context manager for v2 scraper
@mwdd146980
Add shared HTTP exceptions and use them in wrappers and mocks
91ca7dc 
Introduce datadog_checks.base.utils.http_exceptions (HTTPError, SSLError)
so HTTP client wrappers and tests can use a single exception contract
instead of requests/httpx-specific types.

- ResponseWrapper.raise_for_status(): catch requests.HTTPError, re-raise
  SharedHTTPError(response=self).
- RequestsWrapper._request(): catch requests ConnectionError/SSLError,
  re-raise built-in ConnectionError and SharedSSLError.
- HTTPXResponseAdapter.raise_for_status(): raise http_exceptions.HTTPError.
- HTTPResponseMock.raise_for_status(): raise http_exceptions.HTTPError
  instead of requests.exceptions.HTTPError.

Enables openmetrics/prometheus and other consumers to stop depending on
requests exception types and supports eventually removing the requests
dependency.
@mwdd146980 mwdd146980 closed this Feb 12, 2026
@mwdd146980 mwdd146980 reopened this Feb 12, 2026
@mwdd146980
Map httpx connection/SSL errors to shared exception types in HTTPXWra…
42fb0dc 
…pper

Convert httpx.ConnectError and httpx.ConnectTimeout from _request to
built-in ConnectionError and SharedSSLError so callers (e.g. OpenMetrics
v2 base_scraper) that catch ConnectionError or SharedSSLError behave
correctly when use_httpx=True. Matches exception handling in RequestsWrapper.
@mwdd146980
Reformat with ddev test --fmt
940c62d
@mwdd146980
Merge remote-tracking branch 'origin/master' into mwdd146980/ai-6576
90ec853
@mwdd146980
Remove openmetrics refernces from proposal doc for now
16ecdbb
@mwdd146980
Use injected session when provided in RequestsWrapper
41cdefd
@mwdd146980
Support mock_http_response for OpenMetrics v2 and legacy without chec…
076b520 
…k class

- mock_http_response: when no check class is given, patch both
  base_scraper.RequestsWrapper and mixins.RequestsWrapper so v2 and legacy
  OpenMetrics tests work; set options['headers'] on the mock for scraper init
- RequestWrapperMock: add .text for legacy tests that use
  mock_http_response(...).return_value.text
@mwdd146980
Require check class in mock_http_response; drop legacy RequestsWrappe…
c831c10 
…r patching

mock_http_response now only supports the get_http_handler path: the first
argument must be a check class with get_http_handler (e.g.
OpenMetricsBaseCheckV2, OpenMetricsBaseCheck). The fixture patches that
class's get_http_handler and uses a per-test response queue so multiple
calls enqueue responses in order. Remove the legacy branch that patched
base_scraper.RequestsWrapper and mixins.RequestsWrapper when no class was
passed.
@mwdd146980
Add Kerberos support to HTTPXWrapper
8f9c66e 
- Add httpx-gssapi dependency (http extra and agent_requirements) and
  LICENSE-3rdparty entry.
- Implement _create_kerberos_auth_httpx() in httpx.py, mapping the same
  config keys as RequestsWrapper (kerberos_auth, delegate, force_initiate,
  hostname, principal) to HTTPSPNEGOAuth.
- Use it from _make_httpx_auth when auth_type is kerberos; on ImportError
  log a warning and send requests without auth.
- Add unit tests for Kerberos auth (mocked httpx_gssapi) and for
  _make_httpx_auth delegating to the Kerberos helper.
@mwdd146980
Add general-mode mock_http_response and update migration docs
9cb6961 
- mock_http_response: support general mode when called without a check
  class; return (client, enqueue) so tests without a check can inject
  a RequestWrapperMock and feed responses via enqueue().
- Keep check mode unchanged: mock_http_response(CheckClass, ...) still
  patches get_http_handler with a queue-backed RequestWrapperMock.
- Add tests for general mode in test_http_mock.py (client, enqueue,
  empty-queue default).
- docs/developer/base/http.md: document check mode, general mode, and
  direct HTTPResponseMock/RequestWrapperMock usage.
- docs/proposals/migrate-requests-to-httpx.md: note implementation
  branches (ai-6576, base-httpx-migration) and testing/mock design.
@mwdd146980
Split mock HTTP fixtures and trim redundant ones
1cf48bc 
- mock_http_response: check-only; require check class as first arg, patch
  get_http_handler with queue-backed RequestWrapperMock.
- mock_http_client: new fixture; returns (client, enqueue) for tests without
  a check; inject client where the HTTP wrapper is created.
- Remove http_response_mock (redundant with mock_response); document in
  mock_response docstring.
- request_wrapper_mock: clarify use for custom handler logic when
  mock_http_response/mock_http_client are not enough; example uses
  mock_response.
- test_http_mock: add TestMockHttpClient for mock_http_client; use
  mock_http_client fixture in those tests.
- docs/developer/base/http.md, docs/proposals/migrate-requests-to-httpx.md:
  describe mock_http_response and mock_http_client; note request_wrapper_mock
  for custom logic only.
@mwdd146980 mwdd146980 closed this Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@mwdd146980 mwdd146980

Labels

agent/review-requested base_package dependencies dev_package docs/review-requested documentation ecosystems/review-requested product/review-requested

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mwdd146980