[CRCR] Raise default callback rate limit to 60/min/repo (#8203)

## Summary

The cross-repo CI relay **callback** Lambda (`crcr-callback-prod` /
`cross_repo_ci_callback`) enforces a **per-repo** sliding-window rate
limit in `utils/redis_helper.check_rate_limit`, defaulting to **20
requests/min** when `RATE_LIMIT_PER_MIN` is unset:

```python
# utils/config.py
rate_limit_per_min = int(os.getenv("RATE_LIMIT_PER_MIN", "20"))
```

No env override is deployed (the deploy Makefile only runs
`update-function-code`, and no IaC sets the variable), so the Lambda
runs at the in-code default of 20/min. That 429s legitimate job-fan-out
bursts — e.g. a downstream reporting `in_progress` for many jobs near
workflow start — which is what trips the L2 edge-case tests for
`TorchedHat/pytorch-redhat-ci`:

```
##[error]Callback server returned HTTP 429.
{"detail": "rate limit exceeded for TorchedHat/pytorch-redhat-ci"}
```

## Change

- `utils/config.py`: raise the default `RATE_LIMIT_PER_MIN` from `20` →
`60` (1 callback/sec/repo). An env var still overrides it for
per-deployment tuning.
- `utils/hud.py`: guard the user-facing "retry after N seconds" message
with `max(1, 60 // rate_limit_per_min)` so it stays sensible if the
limit is ever set above 60 (otherwise `60 // 61 == 0`).

## Why 60 is safe

- The limit is **per-repo** (key `oot:rate:{repo}`), so aggregate
downstream load = `60 × (#L2+ repos)`. There are currently 2 L2 repos
(`pytorch/crcr-test`, `TorchedHat/pytorch-redhat-ci`), so worst case ≈ 2
callbacks/sec — trivial for the HUD API / DynamoDB, even accounting for
the up-to-4× amplification from `HUD_MAX_RETRIES` during a HUD slowdown.
- Still a real abuse guard at 1 callback/sec/repo.
- The Redis ops per callback are negligible; the only meaningful
downstream cost is the single HUD POST, which this comfortably bounds.

## Deploy note

Only the **callback** Lambda evaluates the limit (`webhook` never calls
`check_rate_limit`), so only `cross_repo_ci_callback` needs redeploying:

```
make -C aws/lambda/cross_repo_ci_relay deploy-callback CALLBACK_FUNCTION_NAME=cross_repo_ci_callback
```

## Test plan

- Existing unit tests unaffected: `test_callback_handler.py` /
`test_redis_helper.py` set `cfg.rate_limit_per_min` explicitly per-test
(independent of the default); `test_config.py` does not assert the
default value.
- AI assistance (Claude) was used to prepare this change.

---------

Signed-off-by: Andrey Talman <atalman@fb.com>

Author: atalman

aws/lambda/cross_repo_ci_relay/tests/test_hud.py

@@ -7,11 +7,17 @@
 from utils.misc import HTTPException
 
 
-def _cfg(url="http://hud/api/oot-ci-events", key="bot-key", max_retries=3):
+def _cfg(
+    url="http://hud/api/oot-ci-events",
+    key="bot-key",
+    max_retries=3,
+    rate_limit_per_min=60,
+):
     cfg = MagicMock()
     cfg.hud_api_url = url
     cfg.hud_bot_key = key
     cfg.hud_max_retries = max_retries
+    cfg.rate_limit_per_min = rate_limit_per_min
     return cfg
 
 

aws/lambda/cross_repo_ci_relay/utils/config.py

@@ -140,7 +140,7 @@ def from_env(cls) -> "RelayConfig":
             raise RuntimeError("HUD_MAX_RETRIES must be a non-negative integer")
 
         try:
-            rate_limit_per_min = int(os.getenv("RATE_LIMIT_PER_MIN", "20"))
+            rate_limit_per_min = int(os.getenv("RATE_LIMIT_PER_MIN", "60"))
             if rate_limit_per_min <= 0:
                 raise ValueError("must be positive")
         except ValueError:

aws/lambda/cross_repo_ci_relay/utils/hud.py

@@ -97,7 +97,7 @@ def forward_to_hud(config: RelayConfig, trusted: dict, untrusted: dict) -> None:
             "An internal failure occurred. "
             "Your update was not saved, but the CI run is still valid. "
             "You can attempt progressive retries after "
-            f"{60 // config.rate_limit_per_min} seconds or ignore this failure.",
+            f"{max(1, 60 // config.rate_limit_per_min)} seconds or ignore this failure.",
         ) from last_exception
     else:
         logger.error(