Handle rate limits (API 429 and 406)

[dux119-cmd]

Currently the script often fails with error 429 due to exceeding OpenSubtitles API rate limit. Even if these rate limits are honored, the script will then fail with error 406 if the user's account has exceeded its daily download limit.

This PR adds handling for both:

• https://apidog.com/blog/opensubtitles-api/ (429)
• https://forum.opensubtitles.org/viewtopic.php?t=16072 (406)

[emericg]

I'm OK with the second commit, but I don't really understand why the first one is necessary, can you tell me more about how we can hit the limits when using this script?

[dux119-cmd]

I was using the original script to add subtitles to my ripped movie collection, in bulk, and it was often failing - so I would just keep re-running it. When I printed the errors in detail, I was getting lots of 429 responses.

I checked the website and OpenSubtitles says that applications should respect the rate-limit and retry after the provided cool down period:

5. Rate Limiting:

Like most public APIs, OpenSubtitles enforces rate limits to ensure fair usage and server stability. Exceeding these limits will result in error responses (often 429 Too Many Requests). Pay close attention to the following response headers to manage your request rate:

X-RateLimit-Limit: The maximum number of requests allowed in the current window. X-RateLimit-Remaining: The number of requests remaining in the current window. X-RateLimit-Reset: The timestamp (often Unix epoch time) when the rate limit window resets. Retry-After: If you receive a 429 error, this header indicates how many seconds you should wait before retrying.

Implementing logic in your application to respect these headers is essential for reliable operation.

https://apidog.com/blog/opensubtitles-api/

So that's what the first commit does: look for 429 responses and retry after the given Retry-After cool down.
Since then, my script runs reliably and I can do bulk scrapes until I consume my daily quota (at which point OpenSubtitles sends a 406 code, which is handled in the second commit).

[emericg]

I was using the original script to add subtitles to my ripped movie collection, in bulk

like run ./OpenSubtitlesDownload.py once on a folder with deep hierarchy?

[dux119-cmd]

I was using the original script to add subtitles to my ripped movie collection, in bulk

like run ./OpenSubtitlesDownload.py once on a folder with deep hierarchy?

Yes, I run it in CLI mode either in bulk or against specific shows, managed by a simple script that calls it on a file-by-file basis:

getsubs(){
  if [[ $# -eq 0 ]]; then
    shopt -s nullglob
    set -- *.mkv
    shopt -u nullglob
  fi
  for f; do
    local srt="${f%.mkv}.srt"
    if [[ -f "$srt" ]]; then
      printf 'Skipping %s, already has subtitles\n' "$srt"
    elif ! mkvinfo "$f" 2>/dev/null | grep -qi "Track type: subtitles"; then
      OpenSubtitlesDownload.py -g cli -s filename --noai --nohi -l en -t auto -a -i "$f"
    else
      printf 'Skipping %s, already has subtitles\n' "$f"
    fi
  done
}