PlexSyncList Container Crashing Due to Spotify API Rate Limit and Error Handling Issues

[tyzingit]

Hello,

I am experiencing consistent crashes with the PlexSyncList Docker container (rnagabhyrava/plexplaylistsync:latest). The primary issue seems to be related to the handling of Spotify API requests, leading to rate limit errors (HTTP 429) and occasional 502 errors.

Environment
Docker version: [Container Manager 20.10.23-1437]
Host OS: [Synology DSM]
PlexSyncList version: Latest (rnagabhyrava/plexplaylistsync:latest)
Configuration Details
Here's the relevant part of my docker-compose.yml:

services:
  playlistSync:
    image: rnagabhyrava/plexplaylistsync:latest
    container_name: playlistSync
    volumes:
      - /volume1/data/media/missing:/data
    environment:
      - PLEX_URL=[REDACTED]
      - PLEX_TOKEN=[REDACTED]
      - WRITE_MISSING_AS_CSV=1 
      - APPEND_SERVICE_SUFFIX=1 
      - ADD_PLAYLIST_POSTER=1 
      - ADD_PLAYLIST_DESCRIPTION=1 
      - APPEND_INSTEAD_OF_SYNC=0 
      - SECONDS_TO_WAIT=84000
      - SPOTIFY_CLIENT_ID=[REDACTED]
      - SPOTIFY_CLIENT_SECRET=[REDACTED]
      - SPOTIFY_USER_ID=[REDACTED]
    restart: unless-stopped
    network_mode: synobridge 

Observed Behavior
The container consistently crashes or encounters issues during its operation.
The logs indicate errors related to Spotify's API, specifically HTTP status code 429, suggesting rate limit exceeding.
There are also mentions of "too many 502 error responses" and "max retries exceeded" errors.
Troubleshooting Steps Taken
Verified SECONDS_TO_WAIT is set to 84000 seconds (23.3 hours), which seems adequate to avoid rate limiting issues under normal circumstances.
Reviewed Docker configuration for any misconfigurations or missing settings.

Possible Causes
The application might be making multiple requests in a short period that are not governed by the SECONDS_TO_WAIT setting.
The error handling for Spotify API requests (especially for HTTP 429 and 502 errors) may not be robust, leading to crashes.

Request for Assistance
Could you please help in identifying why these errors are occurring and suggest any potential fixes or workarounds? Additionally, if there are any specific logs or further information you need, I am happy to provide them.

Thank you for your time and assistance with this issue.

[rnagabhyrava]

The application might be making multiple requests in a short period that are not governed by the SECONDS_TO_WAIT setting.

I don't think this is not an issue as program literally shutsdown for defined seconds unless you restart the program.

How many playlists do you have in Spotify? How many songs? Can you paste logs?

[rnagabhyrava]

As for

The error handling for Spotify API requests (especially for HTTP 429 and 502 errors) may not be robust, leading to crashes

This program uses spotipy package for accessing spotipfy. Although the version is now out by a minor version I don't see any recent changes to spotify API.

I would love to take a look at the logs and errors messages if you can get them added.

[tyzingit]

Thank you for the quick response! Also, forgive me, I'm a n00b and asked AI to help me write a bug report that is likely to help the developer because there's no way I could have given you what you needed on my own! :)

FWIW: I've been using this for years with no issues, it's been a gamechanger for me (THANK YOU BTW!). It just started happening about a week ago where it's crashing every few minutes on Docker. I can't think of any changes that have been made on my end either.

Playlists: 46

Is there a quick way to see how many songs total?

Relevant Log (I can provide more if needed)

2023/10/31 13:23:27 INFO:root:Starting Spotify playlist sync
2023/10/31 13:23:27 INFO:root:Starting playlist sync
2023/10/31 13:23:24 ERROR: Max Retries, reason: too many 502 error responses
2023/10/31 13:23:24 spotipy.exceptions.SpotifyException: http status: 429, code:-1 - /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track

2023/10/31 13:23:24 During handling of the above exception, another exception occurred:
2023/10/31 13:23:24 requests.exceptions.RetryError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses'))

2023/10/31 13:23:24 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses'))

2023/10/31 13:23:24 ERROR:spotipy.client:Max Retries reached
2023/10/31 13:21:22 INFO:root:Starting Spotify playlist sync
2023/10/31 13:21:22 INFO:root:Starting playlist sync
2023/10/31 13:21:20 ERROR: Max Retries, reason: too many 502 error responses
2023/10/31 13:21:20 spotipy.exceptions.SpotifyException: http status: 429, code:-1 - /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track

2023/10/31 13:21:20 During handling of the above exception, another exception occurred:
2023/10/31 13:21:20 requests.exceptions.RetryError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses'))

[rnagabhyrava]

A quick google search got me to this answer. Please read the first answer and the reply to the answer on stack overflow. The other thing you could do is to stop the program for a day or two and let the spotify API reset the limit. Also, if you use the same Spotify client ids for any other programs turn them off for a bit as well. https://stackoverflow.com/questions/63888923/maxretryerror-with-spotify-api

…

On Tue, Oct 31, 2023, 13:04 tyzingit ***@***.***> wrote: Thank you for the quick response! Also, forgive me, I'm a n00b and asked AI to help me write a bug report that is likely to help the developer because there's no way I could have given you what you needed on my own! :) FWIW: I've been using this for years with no issues, it's been a gamechanger for me (THANK YOU BTW!). It just started happening about a week ago where it's crashing every few minutes on Docker. I can't think of any changes that have been made on my end either. Playlists: 46 Is there a quick way to see how many songs total? Relevant Log (I can provide more if needed) 2023/10/31 13:23:27 INFO:root:Starting Spotify playlist sync 2023/10/31 13:23:27 INFO:root:Starting playlist sync 2023/10/31 13:23:24 ERROR: Max Retries, reason: too many 502 error responses 2023/10/31 13:23:24 spotipy.exceptions.SpotifyException: http status: 429, code:-1 - /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track 2023/10/31 13:23:24 During handling of the above exception, another exception occurred: 2023/10/31 13:23:24 requests.exceptions.RetryError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses')) 2023/10/31 13:23:24 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses')) 2023/10/31 13:23:24 ERROR:spotipy.client:Max Retries reached 2023/10/31 13:21:22 INFO:root:Starting Spotify playlist sync 2023/10/31 13:21:22 INFO:root:Starting playlist sync 2023/10/31 13:21:20 ERROR: Max Retries, reason: too many 502 error responses 2023/10/31 13:21:20 spotipy.exceptions.SpotifyException: http status: 429, code:-1 - /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track 2023/10/31 13:21:20 During handling of the above exception, another exception occurred: 2023/10/31 13:21:20 requests.exceptions.RetryError: HTTPSConnectionPool(host='api.spotify.com', port=443): Max retries exceeded with url: /v1/playlists/37i9dQZF1EQqZgBURAEzWH/tracks?limit=100&offset=0&additional_types=track (Caused by ResponseError('too many 502 error responses')) — Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJEHWXDULF4XLHCPG7SSUDYCE4THAVCNFSM6AAAAAA6YDKL6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBXG4ZDIMZUGM> . You are receiving this because you commented.Message ID: ***@***.***>