Method for throttling

[roippi]

Hi, love the tool.

I have a list of somewhere around ~5k URLs to fetch from the CrUX API per day. When I run AWP on full blast, I blow through the 150 req/minute quota for the CrUX API and get Quota exceeded errors for the majority of requests. Is there any way to throttle the request rate down?

[roippi]

I can kind of do this via an extension, but as the extension system is currently written it needs to be a busy loop:

class ThrottleExtension extends Extension {

  afterRun(context, options) {
    const then = Date.now();
    while ((Date.now() - then) < 420) {
      // busy loop eew :(
    }
  }
}

More plumbing would let you e.g. return a Promise from extension hooks and clean that up. But again, maybe I'm going off the rails here, so want to get your feedback.

[gilbertococchi]

Hi Ben, nice to e-meet you!

That's a great idea, we didn't yet thought about adding a throttle on request rate just yet, we could limit the number of concurrent APIs being retrieved by the NPM CLI command if exceeding the 150q/min quota but it's up to the cronjob you have on your end to ask AWP again after 1min to trigger new APIs request on your end.

Do you think a similar setup would work on your case?
We can think about extending the CrUX API to work similarly to the WPT one that is considering the test Submit and Retrieval as different steps.

[roippi]

Howdy. I use jenkins for my scheduling/running, so pretty much any of the above works for me. I am running the job on a once per day cadence so it taking a long time is not a problem 😁

Here is my current solution for throttling via an extension. A production-ready version would be configurable and maybe have some tests 😁 stable...roippi:throttle_extension