Error: 429 Too Many Requests

[peteolbrys]

Started getting 429 errors this morning while running the bot locally in a Docker container, after pulling down latest main:

vaccine-bot_1  | E, [2021-04-05T08:43:45.596599 #1] ERROR -- : [Southcoast] Failed to get appointments: 429 Too Many Requests
vaccine-bot_1  | E, [2021-04-05T08:43:45.596883 #1] ERROR -- : [Main] Error: 429 Too Many Requests
vaccine-bot_1  | /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 429 Too Many Requests (RestClient::TooManyRequests)
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
vaccine-bot_1  | 	from /usr/local/lib/ruby/3.0.0/net/http.rb:960:in `start'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
vaccine-bot_1  | 	from /usr/local/bundle/gems/rest-client-2.1.0/lib/restclient.rb:70:in `post'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:67:in `json_data'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:44:in `clinics'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:31:in `block (2 levels) in all_clinics'
vaccine-bot_1  | 	from /usr/src/app/lib/sentry_helper.rb:5:in `catch_errors'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:29:in `block in all_clinics'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:27:in `each'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:27:in `flat_map'
vaccine-bot_1  | 	from /usr/src/app/lib/sites/southcoast.rb:27:in `all_clinics'
vaccine-bot_1  | 	from run.rb:47:in `block in all_clinics'
vaccine-bot_1  | 	from run.rb:46:in `each'
vaccine-bot_1  | 	from run.rb:46:in `all_clinics'
vaccine-bot_1  | 	from run.rb:97:in `block (2 levels) in main'
vaccine-bot_1  | 	from run.rb:61:in `sleep_for'
vaccine-bot_1  | 	from run.rb:95:in `block in main'
vaccine-bot_1  | 	from run.rb:94:in `loop'
vaccine-bot_1  | 	from run.rb:94:in `main'
vaccine-bot_1  | 	from run.rb:122:in `<main>'

Running on MacOS 11.1, Docker Desktop 3.2.2 (61853)

[sgarfinkel]

Looks like Southcoast is throttling. Probably need to reduce the frequency of checks against them. Is this a persistent issue for you or were you maybe restarting the container a bunch of times?

[dpca]

If you run the container with the environment variable ENVIRONMENT=production it will bypass these errors without crashing (it will still log them though). Check out lib/sentry_helper.rb for how this works, but it essentially lets each scraper be isolated from crashing the others. I'll put some additional info in the readme so this behavior is more clear.

I've seen Southcoast return 429s occasionally in the middle of the night, but it seems to go away after a few minutes so I haven't put much effort into changing things there.