(UserError) Received too many requests in a short amount of time when calling create_agent_evaluation()

[shinji-Yama77]

I'm testing an Agent Evaluation workflow locally using the Azure AI Foundry SDK (azure-ai-projects), and evaluation triggers that previously worked are now consistently failing with a UserError: Received too many requests in a short amount of time.

The issue persists even after multiple retries and across restarts, suggesting a potential rate-limit or quota enforcement on the evaluation endpoint.

I’m using the Agent Framework with the following evaluation configuration and retry logic:

@dataclass
class EvaluationConfig:
    sampling_percent: int = 10
    max_request_rate: int = 50

    def to_sampling_config(self, agent_id: str):
        return AgentEvaluationSamplingConfiguration(
            name=agent_id,
            sampling_percent=self.sampling_percent,
            max_request_rate=self.max_request_rate
        )

Evaluation trigger:

from agent_framework.azure import AzureAIAgentClient

self.client = AzureAIAgentClient()
await self.client.project_client.evaluations.create_agent_evaluation(
    evaluation=AgentEvaluationRequest(
        thread_id=thread_id,
        run_id=run_id,
        evaluators=self.evaluators,
        sampling_configuration=self._get_sampling_config(),
        app_insights_connection_string=app_insights_conn
    )
)

Evaluators

EVALUATORS = {
    "Fluency": {"Id": EvaluatorIds.FLUENCY.value},
    "Coherence": {"Id": EvaluatorIds.COHERENCE.value},
}

Even with exponential backoff (up to 5 retries, 2–32 s delay), I still get:

Full Stack trace:

azure.core.exceptions.HttpResponseError: (UserError) Received too many requests in a short amount of time
Code: UserError
Message: Received too many requests in a short amount of time

Current Region: WestUS2
Current Model: GPT-4.1 (Standard, 1 million TPM quota)

1. What are the current evaluation request limits (per project, per user, or per region)?

2. Does max_request_rate in the sampling configuration influence throttling, or is it informational only?

3. Is there a recommended delay or concurrency policy for evaluations triggered through the Agent Framework?

[kseager]

1. What are the current evaluation request limits (per project, per user, or per region)?
   I would have to get back to you on exact capacity limitations, but we are seeing higher usage in WestUS2 at this time should I would suggest a different region if possible. It is possible to BYO deployment if that would help your situation.

2. Does max_request_rate in the sampling configuration influence throttling, or is it informational only?
   Yes max_request_rate does influence throttling.

3. Is there a recommended delay or concurrency policy for evaluations triggered through the Agent Framework?
   What I would recommend is to firstly increase your sampling percentage. Your sampling config is set to 10 so 90% of the reqs will be 429s.
   Secondary/additional steps if that doesn't solve it is move off WestUS2 and increase request rate

[shinji-Yama77]

@kseager Thank you for your response! I was able to increase my sampling request rate to 100% to test it locally and it was triggering my evaluations successfully.