Allow users to override cache control headers

[charliermarsh]

Some registries don't provide any cache-control headers, which can make for a suboptimal experience (see: #10359). (PyPI returns 10-minute caching headers.) I wonder if it makes sense to allow users to override the cache-control headers?

[zanieb]

See also #10380

[charliermarsh]

#10380 seems different -- that appears to be a request that we add a mode that attempts to hit the index, but falls back to --offline behavior if it's unavailable?

[zanieb]

I agree it's different. I think the similarity is that we may need to override cache control headers (as I mentioned there) to read from a stale cache? I'm not actually familiar with how that interacts with the offline mode.

[charliermarsh]

I believe we allow reading from a stale cache in offline mode.

[chanderson0]

+1 to this, just contributing a use case for our organization:

We are working on installing NVIDIA IsaacLab via uv - a huge step up from their bespoke conda envs. This requires downloading a few gigabytes of packages that do not cache because (I assume) NVIDIA's package index is set to return cache-control: max-age=0, no-cache, no-store. This makes fresh Docker build times using uv quite slow compared to the previous approach, which was to install the package at the system level.

An example wheel and HTTP headers

curl -v https://pypi.nvidia.com/isaacsim-benchmark/isaacsim_benchmark-4.5.0.0-cp310-none-manylinux_2_34_x86_64.whl
*   Trying 23.209.72.14:443...
* Connected to pypi.nvidia.com (23.209.72.14) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /usr/lib/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Santa Clara; O=NVIDIA Corporation; CN=it.nvidia.com
*  start date: Dec  2 00:00:00 2024 GMT
*  expire date: Dec  2 23:59:59 2025 GMT
*  subjectAltName: host "pypi.nvidia.com" matched cert's "*.nvidia.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x63b43ddf0eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /isaacsim-benchmark/isaacsim_benchmark-4.5.0.0-cp310-none-manylinux_2_34_x86_64.whl HTTP/2
> Host: pypi.nvidia.com
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< content-type: application/x-gzip
< content-length: 3318653
< last-modified: Wed, 29 Jan 2025 23:45:30 GMT
< x-amz-server-side-encryption: AES256
< accept-ranges: bytes
< server: AmazonS3
< etag: "d92eac51baff4bf6ba14a6d4f0bb8bba"
< x-amz-cf-pop: EWR53-C1
< x-amz-cf-id: E6Wtn-2__Xl3Zj_upc2AkLe2TfEo7vVta1wTO6Pk9j3JXPLiFKwlBg==
< expires: Wed, 12 Feb 2025 23:14:25 GMT
< cache-control: max-age=0, no-cache, no-store
< pragma: no-cache
< date: Wed, 12 Feb 2025 23:14:25 GMT
< x-cdn-version: v1
< x-cdn: akam
< akamai-grn: 0.8e04d217.1739402065.1f05112d
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* stopped the pause stream!
* Connection #0 to host pypi.nvidia.com left intact