Replace deprecated XML-RPC Pypi APIs

[pombredanne]

See https://mail.python.org/archives/list/pypi-announce@python.org/thread/5VOX33ARFQUYKIMKM5NS7PM7Z6ZNCSJY/ and:

• Deprecation of PyPI XMLRPC Methods pypi/warehouse#16642

The following PyPI XMLRPC methods are being permanently deprecated:

list_packages
package_releases
release_urls
release_data

Technically https://warehouse.pypa.io/api-reference/xml-rpc.html#mirroring-support did not deprecated the list_packages_with_serial RPC, but in practice it times out and should likely be replaced with the new simple JSON API that provides a list of packages with serials
See also https://docs.pypi.org/api/index-api/

@ewdurbin I assume that list_packages_with_serial is effectively deprecated even if not documented yet, correct?

[ewdurbin]

No, that endpoint is still supported and the documentation is up to date.

The intention is to support the three methods changelog_last_serial, changelog_since_serial, and list_packages_with_serial until a better solution for mirroring API is developed, deployed, and adopted.

In this instance the JSON simple API is a great candidate for adoption to replace list_packages_with_serial, except for the caching aspect. XMLRPC list_packages_with_serial will give you an honest un-cached answer every time (which is why it takes 10-15 seconds). The JSON simple API endpoint will remain cached for 24 hours, which is why it is often nearly immediate.

Our metrics for that endpoint show that has been generally stable, with responses ranging from 20-30 seconds from the backends, and generally well below the 60 second internal timeout. Is there a timeout being set by bandersnatch's client libraries?

[ewdurbin]

Looking into this, I did find a bit of a glaring performance issue in the list_packages_with_serial method (and others!).

Once deployed, that endpoint should get a bit faster. Turns out internally we were executing the DB query and dictionary build for response... twice 🙃

[cooperlees]

O how I'd love to remove all XMLRPC from bandersnatch. I've tied. Long long ago. PR's welcome using other "approved" and preferable PEP'd APIs to get the same data if its:

• At last as efficient (client side - but up to add a little complexity/compute client side)
• Decently tested

Thanks @ewdurbin for finding some inefficiencies in the server tho! I feel mirrors don't really need "realtime" responses and that we could probably cache these responses for 1 minute on to the CDN if we wanted, but don't know how friendly the service is to that. And also there probably isn't enough mirrors to really save a huge amount of load ...

[ewdurbin]

Thanks for flagging regardless. We have shipped a few optimizations that should at least make the projects with serial endpoint live to fight another day:

[pombredanne]

I pushed a PR dropping support for XMLRPC but the API works again with @ewdurbin patches.

• Drop XMLRPC API to use PyPI Index API #1898
  @cooperlees because of this I have mixed feelings about this.

We could:

• keep XML-RPC alive but mark it somehow deprecated and available only behind an extra and an option, and not used by default?
• or just drop XML-RPC entirely as proposed?

The only concern is that the once-a-day update of the JSON simple index may be an issue for some use cases. One day of wait feels like a long time for new releases.