Skip to content

fix: preserve text task context in OpenAI chat adapter#1320

Closed
Genmin wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
Genmin:fix/openai-chat-preserve-text-ctx
Closed

fix: preserve text task context in OpenAI chat adapter#1320
Genmin wants to merge 1 commit into
EvolvingLMMs-Lab:mainfrom
Genmin:fix/openai-chat-preserve-text-ctx

Conversation

@Genmin
Copy link
Copy Markdown
Contributor

@Genmin Genmin commented May 2, 2026

Summary

  • preserve the constructed ctx for text-only OpenAI chat requests instead of sending only the raw doc_to_messages question text
  • keep explicit multi-message chats and multimodal messages on the existing doc_to_messages path
  • add regression coverage for text-only replacement plus multimodal/chat-native preservation

Fixes #1319.

Validation

  • uv run --with pytest --with loguru --with python-dotenv --with openai --with pydantic --with tqdm --with numpy --with pillow --with accelerate python -m pytest test/models/test_openai_chat.py -q
  • uv run --with loguru --with python-dotenv --with openai --with pydantic --with tqdm --with numpy --with pillow --with accelerate python -m py_compile lmms_eval/models/chat/openai.py test/models/test_openai_chat.py
  • git diff --check

Copy link
Copy Markdown
Collaborator

@kcz358 kcz358 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I think this would hurt most of the tasks that use doc to messages to construct the messages. If there are tasks with wrong auto wrapped doc to messages function, the tasks should be revise instead of changing the doc to messages logic in the oai model.

@Genmin
Copy link
Copy Markdown
Contributor Author

Genmin commented May 12, 2026

Thanks for taking a look. I agree with the concern: this is too broad for the model adapter layer, and task-specific message construction should be fixed at the task level instead. Closing this to avoid pushing behavior that could regress normal doc_to_messages tasks.

@Genmin Genmin closed this May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI chat adapter drops text-task ctx, causing lm-eval-harness task mismatch

2 participants