SOLR-10998: Obey 'Accept' header in v2 APIs

[gerlowskija]

'wt' takes precedence if specified for now, if not provided Solr (through our use of Jersey) will pick a response format in keeping with the specified 'Accept' header.

JSON responses remains the "default" if neither 'wt' nor an "Accept" header are specified.

https://issues.apache.org/jira/browse/SOLR-10998

Description

The HTTP spec defines an "Accept" header that users can provide to specify the response format (or formats, plural) that they're willing to "accept" in a response. This is part of the HTTP specs support for "content negotiation" more generally.

Solr today doesn't support this, and opts to have users specify their response format in a "wt" (i.e. "writer type") query-parameter. We should obey "Accept" headers, when they are provided.

Solution

This PR modifies Solr to obey "Accept" headers on v2 API requests, unless an overriding "wt" parameter is also provided. Jersey (which is used for v2 requests) actually obeys the "Accept" header out of the box, but a bug in our MediaTypeOverridingFilter (which implements the 'wt' override) was causing the value of "Accept" to be ignored altogether.

With this change, the order of precedence for v2 APIs is: wt parameter, then Accept header, and lastly "JSON" is used as a fallback/default.

Tests

See test cases in V2ApiIntegrationTest

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[epugh]

This is some nice alignment with normal web expectations..

[dsmiley]

I love what you wrote -- what this PR accomplishes. It's a mystery to me how this PR accomplishes that goal. A test would be nice.

[gerlowskija]

a test would be nice

Tests added in V2ApiIntegrationTest

It's a mystery to me how this PR accomplishes that goal.

Really it doesn't. Jersey will parse the "Accept" header out of the box, and choose a "MessageBodyWriter" (akin to our ResponseWriter") based on that. But I'd hidden this support in the past by creating a "response filter" (MediaTypeOverridingFilter) to key off of only the 'wt' parameter.

So what this PR is really doing is tweaking MediaTypeOverridingFilter to be slightly less aggressive. It still obeys "wt" when provided, and makes sure that JSON is the default otherwise. But now it only leans on that "JSON" default when there's no "wt" AND there's no "Accept" header. Jersey is still doing the "Accept" parsing and support and all that - we're just getting out of its way in the cases where we don't want our own custom behavior (e.g. "wt")

[dsmiley]

A forward-looking question: In SolrJ, assuming "V2", how might we compose an Accept header for multiple formats, and then on the receiving end, choose the right ResponseParser for the final format chosen by the server?

Even assuming Jackson, I see us supporting application/json, application/cbor, and could add "smile" to the list, as all 3 are supported with small changes thanks to Jackson's well-designed API.

Our current abstractions are a bit rigid, with the ResponseParser that must supply a "wt" and that which consumes an inputStream without knowledge of the actual mime type in the response header, as it's not passed to the ResponseParser. Maybe we basically just need to change ResponseParser, and make the SolrClients get content types from the RP (to put in "Accept") instead of the "wt" param? Sounds pretty straight-forward actually.

[dsmiley]

We're trying to get away from Apache HttpClient. Can you please try Jetty HttpClient instead?

[gerlowskija]

What's the best way to do that? Is there one?

Per the Slack discussion here, there's not yet a general utility for creating a Jetty client. And adding a well-thought-out version of that is conceptually unrelated and beyond the scope of this PR.

I can throw together a client-creation snippet just for this test, but that feels a little hacky and the sort of thing that'd be likely to run afoul of our test randomization and cause flaky failures down the line whenever SSL or whatever variable is randomly toggled.

[gerlowskija]

In SolrJ, assuming "V2", how might we compose an Accept header for multiple formats, and then on the receiving end, choose the right ResponseParser for the final format chosen by the server?

Hmm... SolrRequest already has a ResponseParser getter/setter pair that is used by the clients to override the default parser on a per-request basis. Ditto for request-headers - SolrRequest has methods to set/get those as well. So it seems like we've got a lot of the puzzle pieces and plumbing already.

The one thing that's missing I think is some sort of "CompoundResponseParser" - something that supports a number of content types via delegation and then chooses one at response-parsing time based on the "Content-type" header in the response.