getAnimeStatistics times out frequently

[YoshiWalsh]

Over the span of 9 minutes, I made 83 requests to GET /anime/:id/statistics. Of these, 7 failed with "408 Request Timeout" errors.

Within the same time span I made 83 requests to GET /anime/:id and they all succeeded. So this issue seems to affect statistics specifically.

An example error response body is

{
    "status": 408,
    "type": "TimeoutException",
    "message": "Request to MyAnimeList.net timed out (10 seconds)",
    "error": "Idle timeout reached for \"https:\/\/myanimelist.net\/anime\/579\/jikan\/stats\"."
}

A request a few minutes later for this same anime took 2 seconds to complete successfully.

A few extra notes:

• 408 Request Timeout is a 4xx error, which means it indicates the fault lies with the client. Because the issue here is happening on Jikan's end, Jikan should return a 5xx error. Typically 504 Gateway Timeout is returned when an upstream server fails to respond in a timely manner.
• The error response mentions a 10 second timeout, but the error seems to consistently happen after 60-63 seconds or 120-123 seconds. (Of my 7 failed requests, 2 took between 120-123 seconds, the other 5 took between 60-63 seconds)

[irfan-dahir]

Over the span of 9 minutes, I made 83 requests to GET /anime/:id/statistics. Of these, 7 failed with "408 Request Timeout" errors.

GET /anime/:id/statistics unique IDs were supplied through here for the 408 to occur, right?

---

Statistics is a sub-page that has a different serving strategy. It's cached and served rather than as the main anime page indexed in a database, then served.

The 408 is a response handled from MAL - you can get a brief on why it happens and what are some best practises to resolve it client-side here: #177 (comment)

---

408 Request Timeout is a 4xx error, which means it indicates the fault lies with the client. Because the issue here is happening on Jikan's end, Jikan should return a 5xx error. Typically 504 Gateway Timeout is returned when an upstream server fails to respond in a timely manner.

Good point 👍

[YoshiWalsh]

GET /anime/:id/statistics unique IDs were supplied through here for the 408 to occur, right?

Yes, I requested stats for 83 different shows.

The 408 is a response handled from MAL - you can get a brief on why it happens and what are some best practises to resolve it client-side here: #177 (comment)

Yeah that's pretty much what I'm doing. I'm caching all the Jikan responses on my end, with a TTL of at least 1 week. If Jikan returns an error, I retry the request up to 3 times.

I might also set a 15 second timeout on my end, since if my program idles for 9 minutes my compute costs will increase. If I re-request after only 15 seconds, is there still a high likelihood that the second request succeeds?

Statistics is a sub-page that has a different serving strategy. It's cached and served rather than as the main anime page indexed in a database, then served.

Do you plan to change stats to use the database approach in future?

[irfan-dahir]

I might also set a 15 second timeout on my end, since if my program idles for 9 minutes my compute costs will increase. If I re-request after only 15 seconds, is there still a high likelihood that the second request succeeds?

It really just depends on MAL. Would suggest experimenting with the retry strategy.
We are reworking the resource/cache updater so hopefully the user should see no 408s as that'd then be something that would just be handled at the backend.

Do you plan to change stats to use the database approach in future?

Yes, though, probably not anytime soon. But we are open to discussing/working out how that could be added and any PRs meanwhile.