Return 429 and 503 occasionally

[jcjones]

The Let's Encrypt API (overall, through a combination of Boulder and the load balancers) returns code 503 when the servers are overloaded, and code 429 when a given client is retrying too often, both with a Retry-After header.

Pebble should return those codes occasionally, too.

[webprofusion-chrisc]

Adding to this we've recently seen a couple of CAs report more rate limiting and server busy type messages which not all clients are handling all that well. It would be great if there was a Chaos mode which could (with configurable frequency/bias) return:

• 429 rate limits, optionally with Retry After
• 503
• 500
• 200 OK, with empty response when a response was expected
• very slow finalization or very slow auth (e.g. several minutes)
• perhaps make it configurable which endpoints would participate in the above.

[jvanasco]

I was surprised to see there is no way to trigger 429 (or other) rate limits.

Different CAs have different limits, so I understand it's hard to model these

These are the current ratelimits for LetsEncrypt (2025/06/12) https://letsencrypt.org/docs/rate-limits/

• New Registrations
  • per IP Address
  • per IPv6 Range
• Certificate Issuance Limits
  • New Orders per Account
  • New Certificates per Registered Domain
  • New Certificates per Exact Set of Hostnames
• Authorization Failures per Hostname per Account
• Overall Requests Limit

BuyPass (2018/08/14) https://community.buypass.com/t/m2r5cj/rate-limits

• Certificate Issuance Limits
  • Certificates per Registered Domain [not new?]
  • New Certificates per Exact Set of Hostnames
• Authorization Failures per Hostname per Account
• Pending Authz
• Overall Requests Limit

While I like the ability in pebble to select a percentage of bad responses for testing, it would be quite difficult to mock tests for ratelimits that require some setup. For example, I might want to successfully obtain a cert, then trip rate limits 100% of the time.

Two approaches that could facilitate easier testing:

1- Actually implement ratelimits, though they need not be timebased (reset by restarting). e.g. There could just be a ratelimit set to "1 cert per domain" or "1 exact hostname match"

2- Allow a POST command to the management port that would change the ratelimits. e.g. after getting a cert, one could set the ratelimit to error on 100% of requests