-
Notifications
You must be signed in to change notification settings - Fork 15
Description
Some Guardian endpoints like https://support.theguardian.com/uk/contribute now require an Okta cookie, and will redirect endlessly if the cookie can't be dropped (see eg guardian/support-frontend#5152 (comment)):
$ curl -I https://support.theguardian.com/uk/contribute
HTTP/2 303
location: /oauth/authorize
Giving us an error like this:
Could not read from this url, got java.net.ProtocolException: Too many follow-up requests: 21
We wouldn't want Prout to maintain cookies across separate checkpoint snapshots (each snapshot should be isolated), but when taking a single snapshot, which might redirect (eg in order to acquire cookies), it could be that we do want to maintain cookies.
Prout currently uses OkHttp as the HTTP client used to take checkpoint snapshots:
prout/app/lib/CheckpointSnapshot.scala
Line 40 in 705f02a
| val client = new OkHttpClient() |
Switch HTTP client to java.net.http.HttpClient ?
Documentation on how to get OkHttp to store cookies between requests (with a CookieJar) is limited, but Java 9 introduced an alternative HTTP client, java.net.http.HttpClient:
...which seems to supply all of Prout's requirements in a HTTP client, including this new one for cookies:
- Cookies :
java.net.CookieManagerseems to offer a suitable level of cookie control - Redirects :
HttpClient.Redirectlets us enable redirects - and by default, there is a redirect limit of 5, which should be fine. - Disabling security : Tolerate insecure hostnames when a checkpoint is 'insecure' #19 can be supported with https://stackoverflow.com/q/52988677/438886
- Async : See
google-search-indexing-observatoryfor an example ofjava.net.http.HttpClientbeing used asynchronously with Scala code &Futures.
Of course, switching to the JDK HTTP client would also mean we'd get to drop a dependency, which would be nice!