429: Too Many Requests, TLS fingerprinting issue

[monik3r]

Hi,

I've been using the crate for one month, about 10 requests a day, but after May 1 I started getting 429s. I did try to change the user agent, but get the same error. As a quick test I tried using the python yfinance package, which works. After some digging it seems like there may have been a fix for this on the yfinance package:

ranaroussi/yfinance#2422 (comment)
ranaroussi/yfinance#2434

[xemwebe]

Could this be related to #76 ? The refrences you provided seem also point into the direction that is has something to to with TLS settings.

[monik3r]

For #76 I believe I got that sometimes when manually modifying the user agent. I believe it's another failure mode.

The linked yfinance issues seem to switch the python package over to use https://github.com/lwthiker/curl-impersonate so that SSL/TLS negotiations look more like a real web browser. Yahoo might be using TLS & HTTP/2 fingerprinting to block non-browser-like requests.

[monik3r]

Seeing that http2 might be a working feature while tls being disabled leads me to believe that this would be the cipher suites presented by the client. See https://scrapfly.io/web-scraping-tools/ja3-fingerprint as one website that shows you your ja3 fingerprint in your browser. I wonder if there is a different approach we can use with reqwest here, or something along the lines there.

[LCTL]

Same issue here. This problem was fixed in the yfinance library, and here is the solution for reference: ranaroussi/yfinance#2422 (comment)

[monik3r]

Closing this in favor of #76 as these are the same underlying issue. Using the rustls backend in the reqwest client builder in #76 (comment) fixed the issue for me. There might be an issue in the future again if the TLS fingerprint doesn't match common browsers/clients that yahoo wants, but hopefully this will work for some time.