Stitchdata Integration 502 Errors

[katered8]

Hey! I'm using the stitchdata yotpo integration, and I've been getting these, or similar, 502 errors once or twice a day since we created the connection. It looks like all/most of the data is coming through properly, but I'm not sure why this error is occurring so frequently or if there's something I should be trying fix on my end?

502 Server Error: Bad Gateway for url: https://api.yotpo.com/v1/widget/......................................../products/<productid>/reviews.json?per_page=100&page=3&direction=asc&sort=date&sort=time

502 Server Error: Bad Gateway for url: https://api.yotpo.com/v1/widget/......................................../products/<productid>/reviews.json?direction=asc&sort=date&sort=time&per_page=100&page=7

[florenciacomuzzi]

@katered8 The error Bad Request for URL is being caused by the fact that the URL being called is incorrect (it is missing the APP KEY and instead contains a series of dots)

When you test it directly it will return the same 404 error all the time: https://api.yotpo.com/v1/widget/......................................../products/2677202976853/reviews.json?direction=asc&per_page=100&sort=date&sort=time&page=83

The correct URL must be https://api.yotpo.com/v1/widget/{app_key}/products/{utoken}/reviews.json + any additional parameters that are in the API documentation.

[wesseljt]

I also have this problem (using stichdata platform) - I tried putting in my API info again - and this didn't seem to fix it. Any other recommendations?

[florenciacomuzzi]

@wesseljt After testing the tap locally, I am quite convinced that the ellipses are actually a result of Stitchdata obfuscating secrets in the error logs!

During local testing of the tap, I found that requests to the API often returned 502 Bad Gateway and 400 Client Error only to successfully return data when I manually re-sent the requests. The tap retries requests, using exponential backoff pattern, when 429, 503, and 504 HTTP error codes are received. This pattern is best practice for handling API rate limiting. No such handling occurs for other error codes; an exception is immediately raised for other error codes.

Long running processes increase the possibility of failure and the Yotpo tap is long running (several hours) even when the data date lookback range is relatively small, i.e. 30 days. The tap paginates through all records of each object and filters on the client side. After adding retry handling for 502 and 400 errors, the tap eventually did fail to create a new connection because authentication only occurs once at the beginning of a tap run. Yotpo docs state clearly that tokens expire when a connection is closed.

To make this tap more robust, the following changes are needed:

• Tap retries 502 and 400 errors several times. Right now, only requests that receive 429, 503, and 504 errors are retried.
• Tap requests a new token whenever a new connection is made. Right now, a token is requested only once at the beginning of a tap run. Yotpo docs state clearly that tokens expire when a connection is closed.
• Tap only requests data for dates within lookback range. Right now, the tap paginates through all records of each object and filters on the client side.

[florenciacomuzzi]

@drewbanin @dmosorast @jacobrobertbaca @KAllan357 @cmerrick can I put in a PR with the changes above?

[dave-connors-3]

@flopotok I saw you opened and closed a PR - I am currently experiencing this issue - were those changes rejected for some reason?