Shopify: Exceeded 2 calls per second for api client

[dnlmzw]

Hi there,

I have a large catalogue of products +600 and are trying to change them all from Draft > Active through Shopify. However, due to an API limit HULL and Shopify are giving some issues.

Maybe there needs to be added some sort of queue to prevent this from happening, as this could easily also happen if two or more people are updating Shopify at the same time.

[POST] /api/shopify/product-update — 10:24:47:73
url: 'https://[shop-name].myshopify.com/admin/products/[pid]/metafields.json',
method: 'get',
...
data: {
  errors: 'Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service.'
}

I found this link: https://help.apidrop.com/en/articles/5220902-shopify-platform-comunication-error-exceeded-2-calls-per-second-for-api-client-reduce-request-rates-to-resume-uninterrupted-service

[ndimatteo]

Hey @dnlmzw!

Great catch here. You're right, that some sort of Queue would be one way to help spread updates out so you don't hit the limit. The other is (as suggested in the link you shared) is to up your limit by moving to Shopify Plus.

I don't have plans at the moment for a built-in Queue to handle this, but if you (or anyone else) has ideas/solutions, feel free to submit a PR!

[j4ysen]

Hey @dnlmzw,
I ran into a similar problem with a client and after a bunch of research found this tool called hookdeck which allows you to queue/throttle webhooks.

All I had to do was change out the webhook urls from my own api endpoint to one they give. From there I set a rate limit of 2 per second to the /api/shopify/product-update endpoint, and it seems like it's working fine for now! There's a free plan of 10,000 events per month.

[dnlmzw]

@j4ysen What a lovely solution. Would love to have this built in at some point, but it also feels like quite some work to build something that works reliably (which I don't have time for with this client), so will definitely check out hookdeck. 10K events also seems reasonable.

[dnlmzw]

@j4ysen I've now had time to test it out and it works like a charm! Had to set the limit to 1 sec though, since 2 seconds was apparently still close enough for it to trigger the rate limiting — Also, having the interface to retry failed calls, and having that general overview is such a great feature.

@ndimatteo I'll leave it up to you to decide whether this is solved or not. Maybe adding it to the documentation could be useful :)

[dnlmzw]

Here's my final working setup in case anyone wonders in the future:

Remember to turn on "rate limiting"

[Jore]

Here's my final working setup in case anyone wonders in the future:

Remember to turn on "rate limiting"

Thanks for the explanation, this is a great solution.

[djclarkson]

Thanks for sharing this. I tried this but got error code 500 when trying to connect to my vercel domain (the CNAME is successfully validated). When do those /api/shopify/ API endpoints get created? I wonder if I am missing a step.

[j4ysen]

@djclarkson
Is the site already deployed? If you're able to get the url to work in shopify notifications webhooks (as per the hull readme instructions), you should be able to add it to hookdeck

[djclarkson]

Thanks @j4ysen yes site was deployed. I got error 500s for product-delete and product-update and 404 for product-create. but haven't had a chance to look further.

[dnlmzw]

I have previously used ngrok to set up a local server that I pointed my Shopify hooks to. You'll then be able to see errors in your CLI, and can therefore debug it a bit faster.

[alexbouchardd]

Hi folks, I'm one of the creators of hookdeck. It was rewarding to read about your experience. I wanted to jump in to contribute a bit more and offer help!

First of all, you can check out our cli, a free alternative to ngrok with a permanent URL (amongst other things). It almost mimics the behaviour of Hookdeck for what you are ready to go to production, you'll just need to add a destination URL.

@djclarkson The /api/shopify/* endpoints seem to be defined here. When you add the Hookdeck URL to Shopify, you don't need to append your server path to the path. That could be leading to 404. Your URL would be https://events.hookdeck.com/e/src_YOUR_ID and not https://events.hookdeck.com/e/src_YOUR_ID/api/shopify/.... Happy to help debug if you reach out in the dashboard live chat!

One more thing, all though this might not be obvious, people tend to prefer using a single Hookdeck URL. You can create a unique shopify source and send all your webhooks to it. Using the filters, you can then forward to the right webhooks to the right URL. Optional but some prefer this approach. It would look like this:

@ndimatteo All of this configuration can also be automated, would there be interest in baking methods to configure the webhooks within the repo? I could find some time to work on it and open a PR if there is interest :)

[multiplehats]

Hi folks, I'm one of the creators of hookdeck. It was rewarding to read about your experience. I wanted to jump in to contribute a bit more and offer help!

First of all, you can check out our cli, a free alternative to ngrok with a permanent URL (amongst other things). It almost mimics the behaviour of Hookdeck for what you are ready to go to production, you'll just need to add a destination URL.

@djclarkson The /api/shopify/* endpoints seem to be defined here. When you add the Hookdeck URL to Shopify, you don't need to append your server path to the path. That could be leading to 404. Your URL would be https://events.hookdeck.com/e/src_YOUR_ID and not https://events.hookdeck.com/e/src_YOUR_ID/api/shopify/.... Happy to help debug if you reach out in the dashboard live chat!

One more thing, all though this might not be obvious, people tend to prefer using a single Hookdeck URL. You can create a unique shopify source and send all your webhooks to it. Using the filters, you can then forward to the right webhooks to the right URL. Optional but some prefer this approach. It would look like this:

@ndimatteo All of this configuration can also be automated, would there be interest in baking methods to configure the webhooks within the repo? I could find some time to work on it and open a PR if there is interest :)

Very cool product! Going to keep a close eye on my logs. Will def use this if I hit rate limits.

Edit: Scratch that, this looks awesome. Going to move some of my webhooks to Hookdeck. I like that it keeps a record of webhooks. ❤️

[ndimatteo]

@alexbouchardd so sorry I didn't see your follow-up on this! Great job with hookdeck and for explaining alternative solutions for this to HULL users!

I'd love to see what an automated approach looks like, feel free to submit a PR for something like that whenever you'd like 😃