Cache responses on the "other side" of redirects with status code 303

[zooba]

Description

See #8489 (cannot comment there because the bot locked it)

Expected behavior

See #8489

pip version

Latest

Python version

any

OS

any

How to Reproduce

See #8489

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[zooba]

Okay, now this is effectively "reopen" (feel free to transfer this comment to #8489 and just reopen it if you like):

I have a private conversation going on with the Azure DevOps team about this issue, and they're using the 303 response to hide the fact that the target URL is not cacheable (because of query parameters). However, it is intentional that subsequent requests should come to the original URL and treat it as cached ¹.

The intent on their side is that the content should be cached, but not the redirect. That is, it'll always be the same file at the end, it'll just come from another source/mirror. There don't appear to be any 30x codes that specifically represent this, but maybe someone is aware of some other precedent?

(FWIW, the Azure DevOps implementation here probably uses the same headers for their other package manager implementations, and may actually be the best precedent there is. But if we can at least agree on a code/header to indicate that pip can cache the resulting data against that URL, even though the final URL may change over time, we can make changes upstream to work with that.)

Footnotes

1. Apart from it having a bad Expires header right now... so it can cache once that is fixed ;) ↩

[zooba]

Also somewhat related is #10075, which it turns out only mattered because the 303 was being followed and the final URL used for caching instead of the original one.

If I hack enable caching on 303's (and bypass the currently not-okay headers) then that issue is redundant.

[Ivoz]

I think dstuff's comment that it would be useful to see the headers for both/all responses from the server is still pertinent

[pradyunsg]

This needs the following information: #8489 (comment)

[zooba]

I can get the headers, but they don't really matter. The response being checked is the redirected URL right now, which expires within minutes and is not going to be returned by subsequent 303s.

Basically, pip ought to cache packages against the URL returned by the index, not by where that URL may eventually direct to (e.g. a local CDN endpoint). That way an index can consistently return the same URL when it knows the package is the same, rather than having that control taken away.

If there's a particular header that could be used to signal that this is a canonical URL, we can add it. However, the existing Cache-Control options seem to sufficiently cover the case where an index wants to return a stable URL that should not be cached, because they want to return different files at different times from the same URL. There don't seem to be any good options for indicating that the redirecting URL is canonical for caching content (as opposed to caching the redirect location), so we're a little bit stuck.

[uranusjr]

IIRC the cache behaviour is inherited from cachecontrol so it may make sense to loop in its maintainers. (First someone needs to check and make sure it is indeed the case, of course.)

[pradyunsg]

FWIW, it would also be useful for whomever maintains these bits of software, to have a public reproducer for this.

[zooba]

It definitely comes from cachecontrol, but the option to cache redirects is part of their public API. And I'm proposing that pip use its additional knowledge about the semantics of the requests it is making to change its behaviour, not proposing that the behaviour should change generally for all 303 redirects (though I'm not opposed to a global status code that works this way).

it would also be useful for whomever maintains these bits of software, to have a public reproducer for this.

You mean like a test index that 303 redirects file links? This one will do it (it only has a pip package on there right now): https://pkgs.dev.azure.com/Python/cpython/_packaging/TestFeed%40Release/pypi/simple/

Append pip to the URL and you get this response (newlines added):

<html>
<head><title>Links for pip</title></head>
<body>
<h1>Links for pip</h1>
<a href="https://pkgs.dev.azure.com/Python/8e426817-76c0-4b99-ba9e-a48a1e4bd5db/_packaging/ad28b313-ee18-4ca2-912f-58714a0d2a78@c804ff44-3b35-4e5d-b661-69d5809c7788/pypi/download/pip/21.3.1/pip-21.3.1-py3-none-any.whl#sha256=deaf32dcd9ab821e359cd8330786bcd077604b5c5730c0b096eda46f95c24a2d" data-requires-python="&gt;=3.6">pip-21.3.1-py3-none-any.whl</a>
<br/>
</body>
</html>

The file URL here is perfectly static, but it returns 303 to point to a temporary, authenticated download link. The 303 response itself should not be cached, because it only contains a temporary URL, but the content at that URL is cacheable, it just won't be available from the redirected URL in the future (it remains accessible at the URL provided in the index data). There's no existing status code or cache control directive that adequately captures these semantics, but I also see no reason why pip can't cache the final response at the initial URL (as well as the final one too, if you want).