429 (too many requests) after running 3 unit tests?

[axolotl-logic]

Hello,

A coworker and I are both getting 429s trying to run unit tests from the Evals/ directory. They only invoked the command three times (1 file each invocation) before being blocked and I made the mistake of running all of Evals/ in one invocation (still blocked 10 hours later). In both our cases, the first 3 succeeded and subsequent were 429s.

Is it possible to support more than a couple invocations in general?

Ideally we'd have this as part of our CI/CD, but that sounds like a recipe for instant-throttling.

Thank you,
Autumn.

TLDR: we get blocked after a few requests and remain blocked seemingly indefinitely.