5000 Character Limit for Model Grader Payload

[Decadz]

Technical Feedback

Hi,

It looks like the Model Grader has a 5000-character limit for the entire payload. I suspect this might be unintentional, since @leestott mentioned that I could split the grader into multiple messages under 5000 characters each and it should still work. However, when I tried splitting the long prompt into several smaller messages (each validated programmatically to be under 5000 characters), the system still returned the 5000-character limit error. When I truncated the full payload to exactly 5000 characters, the training session launched successfully.

EDIT: The 5000-character limit is not just for model graders, but also python graders.

Desired Outcome

Let the model grader use longer prompts, or let user split multiple messages into <5000 character chunks.

Current Workaround

No current work around.

[leestott]

Hi @Decadz
Your understanding is correct:
Model Grader and Python Grader both enforce a 5000-character limit on the entire payload, not per message.
Splitting into multiple messages doesn’t help because the backend aggregates them into one payload before processing. Truncating to exactly 5000 characters allows the session to launch successfully.

It looks like the limit is applied at the request level, not per message.
Even if you chunk prompts, the system concatenates them internally, so the combined size still exceeds 5000 characters.
This constraint is currently hardcoded for graders (both model and Python) in Azure AI Foundry.

Grader Documentation
Azure OpenAI Graders for Generative AI - https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-evaluators/azure-openai-graders – explains grader types and usage (Label, String Checker, Python Grader).

Python Grader Class Reference
https://learn.microsoft.com/python/api/azure-ai-evaluation/azure.ai.evaluation.azureopenaipythongrader?view=azure-python – details custom evaluation logic and constraints.

Here are practical steps:

Optimise Prompt Size

• Remove redundant text, comments, and whitespace.
• Use token-efficient phrasing (shorter instructions, avoid verbose examples).
• Externalise large context into files or URLs and reference them in the prompt.

Modularise Logic

• Instead of one massive grader prompt, break evaluation into multiple runs:
• Each run handles a subset of tasks or examples.
• Aggregate results programmatically after execution.