Skip to content

fix(aime): ensure non-empty prompt for API backends#3555

Open
jayvenn21 wants to merge 1 commit intoEleutherAI:mainfrom
jayvenn21:fix/aime-api-empty-prompt
Open

fix(aime): ensure non-empty prompt for API backends#3555
jayvenn21 wants to merge 1 commit intoEleutherAI:mainfrom
jayvenn21:fix/aime-api-empty-prompt

Conversation

@jayvenn21
Copy link

Fix AIME'24 empty prompt for API backends

This PR fixes an issue where AIME'24 tasks generate an empty prompt when evaluated using OpenAI-style / local-completions backends, causing API errors like:

Prompt cannot be empty

HF local models tolerate empty prompts, but API backends do not.

Changes

  • Ensure doc_to_text always returns a non-empty prompt (instructional prefix: "Solve the following AIME problem.")
  • Applied to aime24, aime, and aime25 with correct dataset fields (Problem / Question / problem)
  • Add regression test tests/test_aime_prompt.py to prevent future empty prompts

Related issue

Fixes #3497

@jayvenn21 jayvenn21 requested a review from baberabb as a code owner February 4, 2026 03:00
@jayvenn21
Copy link
Author

Note: this does not force apply_chat_template; API users should still pass it when evaluating chat/instruct models. This PR only guarantees a non-empty base prompt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AIME'24 is broken w/ Openai-API backend

1 participant