Add retry config for urllib3 requests

[jhamon]

Problem

We want to automatically retry when errors occur

Solution

Implement urllib3 retry configuration. We implemented the backup calculation with jitter ourselves because this is not available for all versions of urllib3 that the SDK uses.

Type of Change

• New feature (non-breaking change which adds functionality)

Test Plan

I created a mock server in scripts/text-server.py that simulates a high rate of failures (80% failure, only 1 in 5 requests succeed).

poetry run python3 scripts/test-server.py

Then I made some requests and observed logging to see what was going on.

>>>  from pinecone import Pinecone
>>>  # Testing control plane with retries
>>> pc = Pinecone(host='http://localhost:8000')
>>> pc.list_indexes()
>>> 
>>> # Data plane
>>> idx = pc.Index(host='http://localhost:8000')
>>> # enable debug logging
>>> idx._vector_api.api_client.configuration.debug = True
>>> 
>>> idx.upsert(vectors=[('1', [0.1, 0.2])])
DEBUG    | pinecone.openapi_support.rest_urllib3:126 | Calling urllib3 request()
DEBUG    | urllib3.connectionpool:546 | http://localhost:8000 "POST /vectors/upsert HTTP/10" 500 None
DEBUG    | urllib3.util.retry:521 | Incremented Retry for (url='/vectors/upsert'): JitterRetry(total=4, connect=None, read=None, redirect=None, status=None)
DEBUG    | pinecone.openapi_support.retries:20 | Calculating retry backoff: 0.15197003184454544 (jitter: 0.15197003184454544)
DEBUG    | urllib3.connectionpool:943 | Retry: /vectors/upsert
DEBUG    | urllib3.connectionpool:546 | http://localhost:8000 "POST /vectors/upsert HTTP/10" 500 None
DEBUG    | urllib3.util.retry:521 | Incremented Retry for (url='/vectors/upsert'): JitterRetry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG    | pinecone.openapi_support.retries:20 | Calculating retry backoff: 0.7352149950424516 (jitter: 0.2352149950424516)
DEBUG    | urllib3.connectionpool:943 | Retry: /vectors/upsert
DEBUG    | urllib3.connectionpool:546 | http://localhost:8000 "POST /vectors/upsert HTTP/10" 500 None
DEBUG    | urllib3.util.retry:521 | Incremented Retry for (url='/vectors/upsert'): JitterRetry(total=2, connect=None, read=None, redirect=None, status=None)
DEBUG    | pinecone.openapi_support.retries:20 | Calculating retry backoff: 1.1307109027442626 (jitter: 0.13071090274426245)
DEBUG    | urllib3.connectionpool:943 | Retry: /vectors/upsert
DEBUG    | urllib3.connectionpool:546 | http://localhost:8000 "POST /vectors/upsert HTTP/10" 500 None
DEBUG    | urllib3.util.retry:521 | Incremented Retry for (url='/vectors/upsert'): JitterRetry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG    | pinecone.openapi_support.retries:20 | Calculating retry backoff: 2.142226695165083 (jitter: 0.14222669516508277)
DEBUG    | urllib3.connectionpool:943 | Retry: /vectors/upsert
DEBUG    | urllib3.connectionpool:546 | http://localhost:8000 "POST /vectors/upsert HTTP/10" 200 None
DEBUG    | pinecone.openapi_support.rest_urllib3:266 | response body: b'{"upsertedCount": 10}'
DEBUG    | pinecone.openapi_support.rest_utils:34 | response status: 200
{'upserted_count': 10}