Add HTTP response headers access to OtpHttpClient #6971

vpaturet · 2025-10-14T13:34:57Z

Summary

Enable access to HTTP response headers through the ResponseMapper interface by introducing OtpHttpResponse wrapper class that encapsulates the body content, headers and status code.
This is required for example to implement support for e-tag.

Note: The integration tests use com.sun.net.httpserver.HttpServer for emulating a simple HTTP server.

Key Changes:

Created OtpHttpResponse class with case-insensitive header access
Updated ResponseMapper interface signature from InputStream to OtpHttpResponse
Modified all ResponseMapper implementations to use response.body()
Added comprehensive unit tests for OtpHttpResponse functionality
Enhanced integration tests to verify header access works correctly

Benefits:

Enables HTTP caching with ETag and Last-Modified headers
Supports rate limit monitoring via response headers
Allows content type inspection and validation
Facilitates debugging with header access

Implementation Details:

OtpHttpResponse provides immutable, case-insensitive header access
Supports multi-value headers (e.g., Set-Cookie)
InputStream lifecycle remains managed by OtpHttpClient
Breaking change: All ResponseMapper implementations require update

Issue

No

Unit tests

Added unit tests

Documentation

No

Enable access to HTTP response headers through the ResponseMapper interface by introducing OtpHttpResponse wrapper class that encapsulates both headers and body content. Key Changes: - Created OtpHttpResponse class with case-insensitive header access - Updated ResponseMapper interface signature from InputStream to OtpHttpResponse - Modified all ResponseMapper implementations to use response.body() - Added comprehensive unit tests for OtpHttpResponse functionality - Enhanced integration tests to verify header access works correctly Benefits: - Enables HTTP caching with ETag and Last-Modified headers - Supports rate limit monitoring via response headers - Allows content type inspection and validation - Facilitates debugging with header access Implementation Details: - OtpHttpResponse provides immutable, case-insensitive header access - Supports multi-value headers (e.g., Set-Cookie) - InputStream lifecycle remains managed by OtpHttpClient - Breaking change: All ResponseMapper implementations require update Files Modified: - 13 source files updated to use response.body() - 2 test files updated - 1 integration test enhanced with header access tests All existing tests pass. No functional changes to HTTP request behavior. Only adds capability to access response headers when needed.

codecov · 2025-10-14T13:46:55Z

Codecov Report

❌ Patch coverage is 61.33333% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.17%. Comparing base (e895874) to head (fc4c68a).
⚠️ Report is 142 commits behind head on dev-2.x.

Files with missing lines	Patch %	Lines
...rg/opentripplanner/framework/io/OtpHttpClient.java	26.66%	22 Missing ⚠️
.../opentripplanner/framework/io/OtpHttpResponse.java	92.30%	1 Missing and 1 partial ⚠️
...anner/ext/siri/updater/azure/SiriAzureUpdater.java	0.00%	1 Missing ⚠️
...anner/ext/vehicleparking/bikely/BikelyUpdater.java	0.00%	1 Missing ⚠️
.../updater/alert/gtfs/GtfsRealtimeAlertsUpdater.java	0.00%	1 Missing ⚠️
...ipplanner/updater/support/siri/SiriHttpLoader.java	0.00%	1 Missing ⚠️
...r/google/GooglePubsubEstimatedTimetableSource.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #6971      +/-   ##
=============================================
+ Coverage      72.13%   72.17%   +0.03%     
- Complexity     19673    19853     +180     
=============================================
  Files           2127     2156      +29     
  Lines          79562    80090     +528     
  Branches        8041     8089      +48     
=============================================
+ Hits           57396    57804     +408     
- Misses         19332    19438     +106     
- Partials        2834     2848      +14

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

application/src/main/java/org/opentripplanner/framework/io/OtpHttpClient.java

leonardehrenfried · 2025-10-16T17:26:45Z

If you add the label "Integration Test", then they are run for this PR.

habrahamsson-skanetrafiken

I think we need a more general strategy for handling response codes. We now have special cases for 204 and 304. I don't think there is a one-size-fits-all solution here and we will need to have different handling of response codes for different situations.

This would be my suggestion:

By default we fail on anything other than 200
We expose another api executeAndMapWithoutValidation() (or some better name) where you don't get response code validation and then your ResponseMapper can do whatever it wants depending on the response code. That way we don't need special logic in the OtpHttpClient for 204/304/whatever. This should be feasible now that the mapper gets the whole response.

habrahamsson-skanetrafiken · 2025-10-17T08:37:13Z

application/src/main/java/org/opentripplanner/framework/io/OtpHttpClient.java

    }
+
+    // Handle NOT_MODIFIED response.
+    if (response.getEntity() == null) {


Suggestion: you could change this to if (response.getCode() == SC_NOT_MODIFIED) and move it to before the previous if statement. I think that would make the logic more straightforward.

habrahamsson-skanetrafiken · 2025-10-17T08:46:24Z

application/src/main/java/org/opentripplanner/framework/io/OtpHttpClient.java

+  private static boolean isFailedRequest(org.apache.hc.core5.http.HttpResponse response) {
+    return (
+      (response.getCode() < SC_OK || response.getCode() >= SC_REDIRECTION) &&
+      response.getCode() != SC_NOT_MODIFIED


It's a bit inconsistent to add an exception for 304. I think the correct thing in that case would be to treat all 3XX as non-failed?

It seems to me that OTP cannot process correctly 3XX responses, apart from 304.
I don't think we have support for redirection for example.

habrahamsson-skanetrafiken · 2025-10-17T09:43:46Z

application/src/main/java/org/opentripplanner/framework/io/OtpHttpClient.java

+        response.getHeaders(),
+        response.getCode()
+      );
+      return contentMapper.apply(httpResponse);


I think contentMapper should be renamed responseMapper now. For all places in this file.

vpaturet · 2025-10-20T13:45:40Z

I think we need a more general strategy for handling response codes. We now have special cases for 204 and 304. I don't think there is a one-size-fits-all solution here and we will need to have different handling of response codes for different situations.

This would be my suggestion:
* By default we fail on anything other than 200

* We expose another api `executeAndMapWithoutValidation()` (or some better name) where you don't get response code validation and then your `ResponseMapper` can do whatever it wants depending on the response code. That way we don't need special logic in the OtpHttpClient for 204/304/whatever. This should be feasible now that the mapper gets the whole response.

We can discuss this in the developer meeting.

vpaturet force-pushed the feature/http-response-headers branch from 1b1af02 to ebe3585 Compare October 14, 2025 13:39

vpaturet self-assigned this Oct 14, 2025

vpaturet added !Improvement A functional improvement or micro feature +Skip Changelog This is not a relevant change for a product owner since last release. labels Oct 14, 2025

vpaturet and others added 2 commits October 14, 2025 16:09

Use the root locale

5e967ab

improve http client handling non 2xx responses and empty body

7e26a18

vpaturet force-pushed the feature/http-response-headers branch 2 times, most recently from a984c7b to 51a14da Compare October 16, 2025 09:24

testower reviewed Oct 16, 2025

View reviewed changes

application/src/main/java/org/opentripplanner/framework/io/OtpHttpClient.java Outdated Show resolved Hide resolved

vpaturet force-pushed the feature/http-response-headers branch 3 times, most recently from 21114c2 to e9d6910 Compare October 16, 2025 13:21

vpaturet marked this pull request as ready for review October 16, 2025 13:31

vpaturet requested a review from a team as a code owner October 16, 2025 13:31

Accept empty body for NOT_MODIFIED response

38f4063

vpaturet force-pushed the feature/http-response-headers branch from e9d6910 to 38f4063 Compare October 16, 2025 13:34

habrahamsson-skanetrafiken self-requested a review October 16, 2025 14:10

testower previously approved these changes Oct 16, 2025

View reviewed changes

habrahamsson-skanetrafiken requested changes Oct 17, 2025

View reviewed changes

vpaturet added the +Integration Test Activate the integration tests on PR label Oct 17, 2025

Apply review suggestions

fc4c68a

vpaturet dismissed testower’s stale review via fc4c68a October 20, 2025 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add HTTP response headers access to OtpHttpClient #6971

Add HTTP response headers access to OtpHttpClient #6971

Uh oh!

vpaturet commented Oct 14, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

leonardehrenfried commented Oct 16, 2025 •

edited

Loading

Uh oh!

habrahamsson-skanetrafiken left a comment

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Uh oh!

vpaturet Oct 20, 2025

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Uh oh!

vpaturet Oct 17, 2025

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Uh oh!

vpaturet Oct 20, 2025

Uh oh!

vpaturet commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add HTTP response headers access to OtpHttpClient #6971

Are you sure you want to change the base?

Add HTTP response headers access to OtpHttpClient #6971

Uh oh!

Conversation

vpaturet commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Issue

Unit tests

Documentation

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

leonardehrenfried commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

habrahamsson-skanetrafiken left a comment

Choose a reason for hiding this comment

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

vpaturet Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

vpaturet Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

habrahamsson-skanetrafiken Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

vpaturet Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

vpaturet commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vpaturet commented Oct 14, 2025 •

edited

Loading

codecov bot commented Oct 14, 2025 •

edited

Loading

leonardehrenfried commented Oct 16, 2025 •

edited

Loading