Feature/inference throughput logging#747
Merged
Merged
Conversation
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
kcz358
reviewed
Jul 11, 2025
Collaborator
kcz358
left a comment
kcz358
left a comment
There was a problem hiding this comment.
I think the idea and most of the implementation looks okay. But the format seems to be inconsistent for each model. I would see if I can create a context manager for this or fix the algorithm format. Also, are TTFT and TPOT necessary in our current settings? I think currently are all estimation so I feel this is a bit inaccurate and maybe tokens per second would be enough to log everything
Implements TPOT (Time Per Output Token) and inference speed metrics: - TPOT = (e2e_latency - TTFT) / (num_output_tokens - 1) - Inference Speed = 1 / TPOT tokens/second Modified chat models: - openai_compatible.py: API call timing with token counting - vllm.py: Batch-level timing with per-request metrics - sglang.py: Timing with meta_info extraction - huggingface.py: Batch processing with token calculation - llava_hf.py: Single-request timing with error handling - qwen2_5_vl.py: Batch timing implementation Features: - Precise timing around model.generate() calls - TTFT estimation when not available from model - Comprehensive logging with formatted metrics - Batch processing support - Error handling for robustness 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
kcz358
force-pushed
the
feature/inference-throughput-logging
branch
from
f75f812 to
740cbc0
Compare
Contributor
Author
Let me check, I think we should only implement this in chat models as standard guidance to community developers. So dont necessarily introduce more abstraction. I think TTFT/TPOT is from vllm/sglang's metrics so I keep it, let me dig deeper or you could start commit to revise it. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Bug: Conditional Variable Usage Causes Error
The avg_speed variable is conditionally defined only when len(outputs) > 1. If len(outputs) <= 1, avg_speed remains undefined, causing a NameError when it is subsequently used in the metric_dict.
lmms_eval/models/chat/sglang.py#L183-L195
lmms-eval/lmms_eval/models/chat/sglang.py
Lines 183 to 195 in 17217f2
Bug: Model Generation Exception Causes Decoding Error
When an exception occurs during model generation, cont is set to an empty string. This leads to a TypeError when self.tokenizer.batch_decode() is subsequently called, as it expects a tensor or list of token IDs instead of a string. The conditional check if cont != "" else "" is insufficient to prevent this issue, and the comparison itself can be fragile when cont is a tensor.
lmms_eval/models/chat/llava_hf.py#L107-L114
lmms-eval/lmms_eval/models/chat/llava_hf.py
Lines 107 to 114 in 17217f2
BugBot free trial expires on July 22, 2025
Learn more in the Cursor dashboard.
Was this report helpful? Give feedback by reacting with 👍 or 👎
kcz358
added a commit
that referenced
this pull request
Jul 30, 2025
* Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Refactor evaluation logic to ensure distributed execution only occurs when multiple processes are active. Update metrics handling in OpenAI Math task to correctly track exact matches and coverage based on results. * Fix text auto messages * Update docs * Add vllm chat models * Add openai compatible * Add sglang runtime * Fix errors * Fix sglang error * Add Claude Code Action workflow configuration * Refactor VLLM model initialization and update generation parameters across tasks. Change model version to a more generic name and adjust sampling settings to enable sampling and increase max new tokens for better performance. * Update max_new_tokens in Huggingface model and enhance metrics handling in OpenAI math task. Remove breakpoint in VLLM model initialization. * Allow logging task input * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * fix repr and group * Add call tools for async openai with mcp client * Add examples * Support multi-node eval * Fix grouping func * Feature/inference throughput logging (#747) * Add inference throughput logging to chat models Implements TPOT (Time Per Output Token) and inference speed metrics: - TPOT = (e2e_latency - TTFT) / (num_output_tokens - 1) - Inference Speed = 1 / TPOT tokens/second Modified chat models: - openai_compatible.py: API call timing with token counting - vllm.py: Batch-level timing with per-request metrics - sglang.py: Timing with meta_info extraction - huggingface.py: Batch processing with token calculation - llava_hf.py: Single-request timing with error handling - qwen2_5_vl.py: Batch timing implementation Features: - Precise timing around model.generate() calls - TTFT estimation when not available from model - Comprehensive logging with formatted metrics - Batch processing support - Error handling for robustness 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add throughput metrics documentation and update logging in chat models * Add gen metric utils * Revise qwen logging * Revise llava_hf logging * Revise hf model loggging * Revise sglang logging * Support vllm logging * Add open logging --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: kcz358 <kaichenzhang358@outlook.com> * Refactor evaluation process to utilize llm_judge API - Updated internal evaluation scripts for D170, DC100, and DC200 tasks to replace GPT evaluation with llm_judge evaluation. - Introduced custom prompts for binary evaluation based on model responses and ground truth. - Modified YAML configuration files to reflect changes in the evaluation metrics and aggregation methods. - Enhanced error handling and logging for evaluation failures. This change aims to improve the accuracy and reliability of model evaluations across different tasks. * Dev/olympiad bench (#762) * Refactor vLLM model files and add OlympiadBench evaluation utilities - Cleaned up imports and removed unused variables in `vllm.py`. - Updated threading configuration in `simple/vllm.py` to use environment variables. - Introduced new utility functions for processing OlympiadBench documents and results in `utils.py`, `zh_utils.py`, and `en_utils.py`. - Added evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py`. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Implemented aggregation functions for results processing in the OlympiadBench context. * Implement OlympiadBench evaluation utilities and refactor math verification - Introduced new utility functions for processing OlympiadBench documents and results in `en_utils.py` and `zh_utils.py`. - Added a custom timeout decorator in `math_verify_utils.py` to replace the previous signal-based timeout. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Removed outdated files from the `olympiadbench_official` directory to streamline the codebase. - Enhanced evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py` and added aggregation functions for results processing. * Update mathvision utility imports and modify YAML configurations for OlympiadBench - Added error handling for importing evaluation utilities in `utils.py` to improve robustness. - Changed `doc_to_target` from "answer" to "final_answer" in both `olympiadbench_all_boxed.yaml` and `olympiadbench_boxed.yaml` to ensure consistency in output naming. * Update YAML configurations for AIME tasks - Changed `do_sample` parameter to `true` in `aime24_figures_agg64.yaml` to enable sampling during generation. - Added new configuration file `aime25_nofigures_agg64.yaml` for a new task, including detailed metrics and filtering options for evaluation. These updates enhance the flexibility and functionality of the AIME evaluation tasks. * Refactor internal evaluation scripts for consistency and readability - Removed unnecessary blank lines in `d170_cn_utils.py`, `d170_en_utils.py`, `dc100_en_utils.py`, and `dc200_cn_utils.py` to improve code clarity. - Streamlined the `evaluate_binary` API call formatting for better readability. These changes enhance the maintainability of the evaluation scripts across different tasks. * Update documentation for `lmms_eval` to enhance clarity and usability - Revised command-line interface section in `commands.md` for improved readability and updated links to the main README. - Enhanced `current_tasks.md` with clearer instructions for listing supported tasks and their question counts. - Added comprehensive model examples in `model_guide.md` for image, video, and audio models, including implementation details and key notes. - Expanded `README.md` to provide an overview of the framework's capabilities and updated the table of contents for better navigation. - Included new audio model examples in `run_examples.md` to demonstrate usage. - Introduced an audio task example in `task_guide.md` to guide users in configuring audio tasks effectively. These updates aim to improve the overall documentation experience for users and developers working with `lmms_eval`. * Introduce LMMS-Eval v0.4: Major update with unified message interface, multi-node distributed evaluation, and enhanced judge API * Add agg8 task and fix data path * Fix warning * Remove bug report documentation from the codebase, consolidating information on identified bugs and fixes for improved clarity and maintainability. * Add announcement for the release of `lmms-eval` v0.4.0 in README.md * Enhance documentation for LMMS-Eval v0.4 with detailed installation instructions, system requirements, and troubleshooting tips. * Remove outdated system requirements and installation instructions from LMMS-Eval v0.4 documentation to streamline content and improve clarity. * Fix datetime format string in olympiadbench submission file naming Co-authored-by: drluodian <drluodian@gmail.com> * Fix video frame handling in protocol with range() for consistent iteration Co-authored-by: drluodian <drluodian@gmail.com> * Convert vLLM environment variables to integers for proper type handling Co-authored-by: drluodian <drluodian@gmail.com> * Fix force_simple model selection to check model availability Co-authored-by: drluodian <drluodian@gmail.com> * Fix format issue and add avg@8 for aime * Allow vllm for tp * fix parsing logic * Fix OpenAI payload max tokens parameter to use max_new_tokens Co-authored-by: drluodian <drluodian@gmail.com> * Update OpenAI payload handling to include support for model version o4 and remove max_tokens parameter * Refactor model version handling across evaluation tasks by removing hardcoded GPT model names and replacing them with environment variable support for dynamic model versioning. Update server configuration to utilize the unified judge API for improved response handling. * batch update misused calls for eval model * Update evaluation tasks to use environment variables for GPT model versioning, replacing hardcoded values with dynamic configuration. Remove unused YAML loading logic in multilingual LLAVA benchmark utilities. * Enhance VLLM configuration to support distributed execution for multiple processes. Update multilingual LLAVA benchmark YAML files to include dataset names and remove deprecated config entries. * Remove reviewer guideline and co-authored-by mention from contribution instructions in claude.md * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * Refactor score parsing logic in multiple utility files to include stripping whitespace from the score string before processing. * Update .gitignore to include new workspace directory and modify utility files to enhance response handling by replacing Request object usage with direct server method calls for text generation across multiple evaluation tasks. * Refactor evaluation tasks to utilize the unified judge API by replacing direct server method calls with Request object usage. Update server configuration in multiple utility files to enhance response handling and streamline evaluation processes. * Refactor generation parameter handling in Llava_OneVision model to streamline configuration. Remove redundant default settings and ensure proper handling of sampling parameters based on the do_sample flag. Update multiple YAML task files to increase max_new_tokens and comment out temperature settings for clarity. Introduce new YAML configuration for MMMU validation reasoning task. * Enhance score processing logic in utility functions to improve error handling and validation. Implement robust regex patterns for score extraction, ensuring all components are accounted for and scores are clamped within valid ranges. Add logging for better traceability of errors and fallback mechanisms for invalid inputs in the mia_bench evaluation process. * Fix launch error when num proc = 1 * Refactor VLLM model parameter handling to simplify distributed execution logic. Remove redundant checks for tensor parallelism and streamline generation parameter settings by eliminating unused temperature and top_p configurations. * Refactor VLLM message handling to prioritize image URLs before text content. Remove unused distributed executor backend parameter for cleaner execution logic. * feat(vllm): Set default max_new_tokens to 4096, temperature to 0, and top_p to 0.95 * docs: Update lmms-eval-0.4 documentation with images and installation instructions * docs: Update lmms-eval-0.4 documentation to include backward compatibility check * refactor: Simplify server config instantiation in utils files * docs: Update supported tasks count in README * Update docs * Fix mathverse bugs * docs: Update images in lmms-eval-0.4.md * docs: Remove API Benefits and Upcoming Benchmarks sections * docs: Update image URL for Unified Message Interface * docs: Fix typo in LMMS-Eval v0.4 documentation Corrected "27.8/16.40" to "27.8/26.40" in the performance comparison table. Also corrected "16.78/13.82" to "16.78/15.82" in the performance comparison table. * fix(docs): Correct typo in LMMS-Eval v0.4 performance comparison table * refactor(docs): Refactor LMMS-Eval v0.4 performance table for clarity * Update docs --------- Co-authored-by: kcz358 <kaichenzhang358@outlook.com> Co-authored-by: Cong <101887866+pbcong@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Luodian
added a commit
that referenced
this pull request
Feb 28, 2026
* Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Refactor evaluation logic to ensure distributed execution only occurs when multiple processes are active. Update metrics handling in OpenAI Math task to correctly track exact matches and coverage based on results. * Fix text auto messages * Update docs * Add vllm chat models * Add openai compatible * Add sglang runtime * Fix errors * Fix sglang error * Add Claude Code Action workflow configuration * Refactor VLLM model initialization and update generation parameters across tasks. Change model version to a more generic name and adjust sampling settings to enable sampling and increase max new tokens for better performance. * Update max_new_tokens in Huggingface model and enhance metrics handling in OpenAI math task. Remove breakpoint in VLLM model initialization. * Allow logging task input * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * fix repr and group * Add call tools for async openai with mcp client * Add examples * Support multi-node eval * Fix grouping func * Feature/inference throughput logging (#747) * Add inference throughput logging to chat models Implements TPOT (Time Per Output Token) and inference speed metrics: - TPOT = (e2e_latency - TTFT) / (num_output_tokens - 1) - Inference Speed = 1 / TPOT tokens/second Modified chat models: - openai_compatible.py: API call timing with token counting - vllm.py: Batch-level timing with per-request metrics - sglang.py: Timing with meta_info extraction - huggingface.py: Batch processing with token calculation - llava_hf.py: Single-request timing with error handling - qwen2_5_vl.py: Batch timing implementation Features: - Precise timing around model.generate() calls - TTFT estimation when not available from model - Comprehensive logging with formatted metrics - Batch processing support - Error handling for robustness 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Add throughput metrics documentation and update logging in chat models * Add gen metric utils * Revise qwen logging * Revise llava_hf logging * Revise hf model loggging * Revise sglang logging * Support vllm logging * Add open logging --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: kcz358 <kaichenzhang358@outlook.com> * Refactor evaluation process to utilize llm_judge API - Updated internal evaluation scripts for D170, DC100, and DC200 tasks to replace GPT evaluation with llm_judge evaluation. - Introduced custom prompts for binary evaluation based on model responses and ground truth. - Modified YAML configuration files to reflect changes in the evaluation metrics and aggregation methods. - Enhanced error handling and logging for evaluation failures. This change aims to improve the accuracy and reliability of model evaluations across different tasks. * Dev/olympiad bench (#762) * Refactor vLLM model files and add OlympiadBench evaluation utilities - Cleaned up imports and removed unused variables in `vllm.py`. - Updated threading configuration in `simple/vllm.py` to use environment variables. - Introduced new utility functions for processing OlympiadBench documents and results in `utils.py`, `zh_utils.py`, and `en_utils.py`. - Added evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py`. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Implemented aggregation functions for results processing in the OlympiadBench context. * Implement OlympiadBench evaluation utilities and refactor math verification - Introduced new utility functions for processing OlympiadBench documents and results in `en_utils.py` and `zh_utils.py`. - Added a custom timeout decorator in `math_verify_utils.py` to replace the previous signal-based timeout. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Removed outdated files from the `olympiadbench_official` directory to streamline the codebase. - Enhanced evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py` and added aggregation functions for results processing. * Update mathvision utility imports and modify YAML configurations for OlympiadBench - Added error handling for importing evaluation utilities in `utils.py` to improve robustness. - Changed `doc_to_target` from "answer" to "final_answer" in both `olympiadbench_all_boxed.yaml` and `olympiadbench_boxed.yaml` to ensure consistency in output naming. * Update YAML configurations for AIME tasks - Changed `do_sample` parameter to `true` in `aime24_figures_agg64.yaml` to enable sampling during generation. - Added new configuration file `aime25_nofigures_agg64.yaml` for a new task, including detailed metrics and filtering options for evaluation. These updates enhance the flexibility and functionality of the AIME evaluation tasks. * Refactor internal evaluation scripts for consistency and readability - Removed unnecessary blank lines in `d170_cn_utils.py`, `d170_en_utils.py`, `dc100_en_utils.py`, and `dc200_cn_utils.py` to improve code clarity. - Streamlined the `evaluate_binary` API call formatting for better readability. These changes enhance the maintainability of the evaluation scripts across different tasks. * Update documentation for `lmms_eval` to enhance clarity and usability - Revised command-line interface section in `commands.md` for improved readability and updated links to the main README. - Enhanced `current_tasks.md` with clearer instructions for listing supported tasks and their question counts. - Added comprehensive model examples in `model_guide.md` for image, video, and audio models, including implementation details and key notes. - Expanded `README.md` to provide an overview of the framework's capabilities and updated the table of contents for better navigation. - Included new audio model examples in `run_examples.md` to demonstrate usage. - Introduced an audio task example in `task_guide.md` to guide users in configuring audio tasks effectively. These updates aim to improve the overall documentation experience for users and developers working with `lmms_eval`. * Introduce LMMS-Eval v0.4: Major update with unified message interface, multi-node distributed evaluation, and enhanced judge API * Add agg8 task and fix data path * Fix warning * Remove bug report documentation from the codebase, consolidating information on identified bugs and fixes for improved clarity and maintainability. * Add announcement for the release of `lmms-eval` v0.4.0 in README.md * Enhance documentation for LMMS-Eval v0.4 with detailed installation instructions, system requirements, and troubleshooting tips. * Remove outdated system requirements and installation instructions from LMMS-Eval v0.4 documentation to streamline content and improve clarity. * Fix datetime format string in olympiadbench submission file naming Co-authored-by: drluodian <drluodian@gmail.com> * Fix video frame handling in protocol with range() for consistent iteration Co-authored-by: drluodian <drluodian@gmail.com> * Convert vLLM environment variables to integers for proper type handling Co-authored-by: drluodian <drluodian@gmail.com> * Fix force_simple model selection to check model availability Co-authored-by: drluodian <drluodian@gmail.com> * Fix format issue and add avg@8 for aime * Allow vllm for tp * fix parsing logic * Fix OpenAI payload max tokens parameter to use max_new_tokens Co-authored-by: drluodian <drluodian@gmail.com> * Update OpenAI payload handling to include support for model version o4 and remove max_tokens parameter * Refactor model version handling across evaluation tasks by removing hardcoded GPT model names and replacing them with environment variable support for dynamic model versioning. Update server configuration to utilize the unified judge API for improved response handling. * batch update misused calls for eval model * Update evaluation tasks to use environment variables for GPT model versioning, replacing hardcoded values with dynamic configuration. Remove unused YAML loading logic in multilingual LLAVA benchmark utilities. * Enhance VLLM configuration to support distributed execution for multiple processes. Update multilingual LLAVA benchmark YAML files to include dataset names and remove deprecated config entries. * Remove reviewer guideline and co-authored-by mention from contribution instructions in claude.md * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * Refactor score parsing logic in multiple utility files to include stripping whitespace from the score string before processing. * Update .gitignore to include new workspace directory and modify utility files to enhance response handling by replacing Request object usage with direct server method calls for text generation across multiple evaluation tasks. * Refactor evaluation tasks to utilize the unified judge API by replacing direct server method calls with Request object usage. Update server configuration in multiple utility files to enhance response handling and streamline evaluation processes. * Refactor generation parameter handling in Llava_OneVision model to streamline configuration. Remove redundant default settings and ensure proper handling of sampling parameters based on the do_sample flag. Update multiple YAML task files to increase max_new_tokens and comment out temperature settings for clarity. Introduce new YAML configuration for MMMU validation reasoning task. * Enhance score processing logic in utility functions to improve error handling and validation. Implement robust regex patterns for score extraction, ensuring all components are accounted for and scores are clamped within valid ranges. Add logging for better traceability of errors and fallback mechanisms for invalid inputs in the mia_bench evaluation process. * Fix launch error when num proc = 1 * Refactor VLLM model parameter handling to simplify distributed execution logic. Remove redundant checks for tensor parallelism and streamline generation parameter settings by eliminating unused temperature and top_p configurations. * Refactor VLLM message handling to prioritize image URLs before text content. Remove unused distributed executor backend parameter for cleaner execution logic. * feat(vllm): Set default max_new_tokens to 4096, temperature to 0, and top_p to 0.95 * docs: Update lmms-eval-0.4 documentation with images and installation instructions * docs: Update lmms-eval-0.4 documentation to include backward compatibility check * refactor: Simplify server config instantiation in utils files * docs: Update supported tasks count in README * Update docs * Fix mathverse bugs * docs: Update images in lmms-eval-0.4.md * docs: Remove API Benefits and Upcoming Benchmarks sections * docs: Update image URL for Unified Message Interface * docs: Fix typo in LMMS-Eval v0.4 documentation Corrected "27.8/16.40" to "27.8/26.40" in the performance comparison table. Also corrected "16.78/13.82" to "16.78/15.82" in the performance comparison table. * fix(docs): Correct typo in LMMS-Eval v0.4 performance comparison table * refactor(docs): Refactor LMMS-Eval v0.4 performance table for clarity * Update docs --------- Co-authored-by: kcz358 <kaichenzhang358@outlook.com> Co-authored-by: Cong <101887866+pbcong@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
stisiTT
pushed a commit
to bgoelTT/lmms-eval
that referenced
this pull request
Mar 6, 2026
* Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (EvolvingLMMs-Lab#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (EvolvingLMMs-Lab#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Update task utils and logger * [Main Update] Doc to messages feature support and Split simple and chat mode (EvolvingLMMs-Lab#692) * Update deps * Restructured * Delete models * Remove deprecated models * Set up auto doc to messages and chat models * Lint * Allow force simple mode * Add auto doc to messages for audio and video Fix lint Init server structure Restructure to server folder Clean base and providers Add clean method for models Fix loggers save result Fix dummy server error Suppress llava warnings Sample evaluator on llava in the wild Update mmmu doc to messages Update version * Add judge server implementation with various providers and evaluation protocols Add AsyncAzureOpenAIProvider implementation and update provider factory Refactor sample saving in EvaluationTracker to use cleaned data and improve logging Add llm_as_judge_eval metric to multiple tasks and integrate llm_judge API for evaluation * Refactor MathVerseEvaluator to utilize llm_judge server for response generation and evaluation, enhancing API integration and error handling. Update MMBench_Evaluator to streamline API key handling based on environment variables. * Refactor EvaluationTracker to directly modify sample data for improved clarity and efficiency. Update MathVerseEvaluator to streamline answer scoring by eliminating unnecessary extraction steps and enhance evaluation prompts. Remove deprecated metrics from configuration files. * Refactor MathVistaEvaluator to integrate llm_judge server for enhanced response generation and evaluation. Streamline API configuration and error handling by removing direct API key management and utilizing a custom server configuration for requests. * Update MathVista task configurations to replace 'gpt_eval_score' with 'llm_as_judge_eval' across multiple YAML files and adjust the result processing function accordingly. This change aligns with the integration of the llm_judge server for enhanced evaluation metrics. * Add new OlympiadBench task configurations for mathematics and physics evaluation. Introduce 'olympiadbench_OE_MM_maths_en_COMP.yaml' and 'olympiadbench_OE_MM_physics_en_COMP.yaml' files, while removing outdated English and Chinese test configurations. Update evaluation metrics to utilize 'llm_as_judge_eval' for consistency across tasks. * Add reasoning model utility functions and integrate into Qwen2_5_VL model. Introduced `parse_reasoning_model_answer` to clean model responses and updated answer processing in the Qwen2_5_VL class to utilize this new function, enhancing response clarity and logging. * Update OlympiadBench task configuration to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Refactor olympiadbench_process_results to enhance response clarity. Updated the return format to include question, response, and ground truth for improved evaluation context. Simplified judge result determination logic. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for improved clarity in response generation. * Update olympiadbench_OE_MM_physics_en_COMP.yaml to change 'doc_to_target' from 'answer' to 'final_answer' for consistency with recent configuration updates and improved clarity in response generation. * Add launcher and sglang launcher for local llm as judge * Lint * add new tasks MMVU and Visual Web Bench (EvolvingLMMs-Lab#727) * add mmvu task * fix linting videomathqa * fix mmvu to use llm judge * add visualwebbench task * Add Qwen2_5 chat to support doc_to_messages * Refactor documentation and codebase to standardize naming conventions from 'lm_eval' to 'lmms_eval'. Update task configurations and evaluation metrics accordingly for consistency across the project. * Update model guide and task configurations to replace 'max_gen_toks' with 'max_new_tokens' for consistency across YAML files and documentation. This change aligns with recent updates in the generation parameters for improved clarity in model behavior. * Refactor evaluation logic to ensure distributed execution only occurs when multiple processes are active. Update metrics handling in OpenAI Math task to correctly track exact matches and coverage based on results. * Fix text auto messages * Update docs * Add vllm chat models * Add openai compatible * Add sglang runtime * Fix errors * Fix sglang error * Add model Code Action workflow configuration * Refactor VLLM model initialization and update generation parameters across tasks. Change model version to a more generic name and adjust sampling settings to enable sampling and increase max new tokens for better performance. * Update max_new_tokens in Huggingface model and enhance metrics handling in OpenAI math task. Remove breakpoint in VLLM model initialization. * Allow logging task input * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * fix repr and group * Add call tools for async openai with mcp client * Add examples * Support multi-node eval * Fix grouping func * Feature/inference throughput logging (EvolvingLMMs-Lab#747) * Add inference throughput logging to chat models Implements TPOT (Time Per Output Token) and inference speed metrics: - TPOT = (e2e_latency - TTFT) / (num_output_tokens - 1) - Inference Speed = 1 / TPOT tokens/second Modified chat models: - openai_compatible.py: API call timing with token counting - vllm.py: Batch-level timing with per-request metrics - sglang.py: Timing with meta_info extraction - huggingface.py: Batch processing with token calculation - llava_hf.py: Single-request timing with error handling - qwen2_5_vl.py: Batch timing implementation Features: - Precise timing around model.generate() calls - TTFT estimation when not available from model - Comprehensive logging with formatted metrics - Batch processing support - Error handling for robustness 🤖 Generated with [model Code](https://model.ai/code) Co-Authored-By: model <noreply@API.com> * Add throughput metrics documentation and update logging in chat models * Add gen metric utils * Revise qwen logging * Revise llava_hf logging * Revise hf model loggging * Revise sglang logging * Support vllm logging * Add open logging --------- Co-authored-by: model <noreply@API.com> Co-authored-by: kcz358 <kaichenzhang358@outlook.com> * Refactor evaluation process to utilize llm_judge API - Updated internal evaluation scripts for D170, DC100, and DC200 tasks to replace GPT evaluation with llm_judge evaluation. - Introduced custom prompts for binary evaluation based on model responses and ground truth. - Modified YAML configuration files to reflect changes in the evaluation metrics and aggregation methods. - Enhanced error handling and logging for evaluation failures. This change aims to improve the accuracy and reliability of model evaluations across different tasks. * Dev/olympiad bench (EvolvingLMMs-Lab#762) * Refactor vLLM model files and add OlympiadBench evaluation utilities - Cleaned up imports and removed unused variables in `vllm.py`. - Updated threading configuration in `simple/vllm.py` to use environment variables. - Introduced new utility functions for processing OlympiadBench documents and results in `utils.py`, `zh_utils.py`, and `en_utils.py`. - Added evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py`. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Implemented aggregation functions for results processing in the OlympiadBench context. * Implement OlympiadBench evaluation utilities and refactor math verification - Introduced new utility functions for processing OlympiadBench documents and results in `en_utils.py` and `zh_utils.py`. - Added a custom timeout decorator in `math_verify_utils.py` to replace the previous signal-based timeout. - Created multiple YAML configuration files for various OlympiadBench tasks, including math and physics in both English and Chinese. - Removed outdated files from the `olympiadbench_official` directory to streamline the codebase. - Enhanced evaluation logic for OlympiadBench tasks in `olympiadbench_evals.py` and added aggregation functions for results processing. * Update mathvision utility imports and modify YAML configurations for OlympiadBench - Added error handling for importing evaluation utilities in `utils.py` to improve robustness. - Changed `doc_to_target` from "answer" to "final_answer" in both `olympiadbench_all_boxed.yaml` and `olympiadbench_boxed.yaml` to ensure consistency in output naming. * Update YAML configurations for AIME tasks - Changed `do_sample` parameter to `true` in `aime24_figures_agg64.yaml` to enable sampling during generation. - Added new configuration file `aime25_nofigures_agg64.yaml` for a new task, including detailed metrics and filtering options for evaluation. These updates enhance the flexibility and functionality of the AIME evaluation tasks. * Refactor internal evaluation scripts for consistency and readability - Removed unnecessary blank lines in `d170_cn_utils.py`, `d170_en_utils.py`, `dc100_en_utils.py`, and `dc200_cn_utils.py` to improve code clarity. - Streamlined the `evaluate_binary` API call formatting for better readability. These changes enhance the maintainability of the evaluation scripts across different tasks. * Update documentation for `lmms_eval` to enhance clarity and usability - Revised command-line interface section in `commands.md` for improved readability and updated links to the main README. - Enhanced `current_tasks.md` with clearer instructions for listing supported tasks and their question counts. - Added comprehensive model examples in `model_guide.md` for image, video, and audio models, including implementation details and key notes. - Expanded `README.md` to provide an overview of the framework's capabilities and updated the table of contents for better navigation. - Included new audio model examples in `run_examples.md` to demonstrate usage. - Introduced an audio task example in `task_guide.md` to guide users in configuring audio tasks effectively. These updates aim to improve the overall documentation experience for users and developers working with `lmms_eval`. * Introduce LMMS-Eval v0.4: Major update with unified message interface, multi-node distributed evaluation, and enhanced judge API * Add agg8 task and fix data path * Fix warning * Remove bug report documentation from the codebase, consolidating information on identified bugs and fixes for improved clarity and maintainability. * Add announcement for the release of `lmms-eval` v0.4.0 in README.md * Enhance documentation for LMMS-Eval v0.4 with detailed installation instructions, system requirements, and troubleshooting tips. * Remove outdated system requirements and installation instructions from LMMS-Eval v0.4 documentation to streamline content and improve clarity. * Fix datetime format string in olympiadbench submission file naming Co-authored-by: drluodian <drluodian@gmail.com> * Fix video frame handling in protocol with range() for consistent iteration Co-authored-by: drluodian <drluodian@gmail.com> * Convert vLLM environment variables to integers for proper type handling Co-authored-by: drluodian <drluodian@gmail.com> * Fix force_simple model selection to check model availability Co-authored-by: drluodian <drluodian@gmail.com> * Fix format issue and add avg@8 for aime * Allow vllm for tp * fix parsing logic * Fix OpenAI payload max tokens parameter to use max_new_tokens Co-authored-by: drluodian <drluodian@gmail.com> * Update OpenAI payload handling to include support for model version o4 and remove max_tokens parameter * Refactor model version handling across evaluation tasks by removing hardcoded GPT model names and replacing them with environment variable support for dynamic model versioning. Update server configuration to utilize the unified judge API for improved response handling. * batch update misused calls for eval model * Update evaluation tasks to use environment variables for GPT model versioning, replacing hardcoded values with dynamic configuration. Remove unused YAML loading logic in multilingual LLAVA benchmark utilities. * Enhance VLLM configuration to support distributed execution for multiple processes. Update multilingual LLAVA benchmark YAML files to include dataset names and remove deprecated config entries. * Remove reviewer guideline and co-authored-by mention from contribution instructions in model.md * Add development guidelines document outlining core rules, coding best practices, and error resolution strategies for the codebase. * Refactor score parsing logic in multiple utility files to include stripping whitespace from the score string before processing. * Update .gitignore to include new workspace directory and modify utility files to enhance response handling by replacing Request object usage with direct server method calls for text generation across multiple evaluation tasks. * Refactor evaluation tasks to utilize the unified judge API by replacing direct server method calls with Request object usage. Update server configuration in multiple utility files to enhance response handling and streamline evaluation processes. * Refactor generation parameter handling in Llava_OneVision model to streamline configuration. Remove redundant default settings and ensure proper handling of sampling parameters based on the do_sample flag. Update multiple YAML task files to increase max_new_tokens and comment out temperature settings for clarity. Introduce new YAML configuration for MMMU validation reasoning task. * Enhance score processing logic in utility functions to improve error handling and validation. Implement robust regex patterns for score extraction, ensuring all components are accounted for and scores are clamped within valid ranges. Add logging for better traceability of errors and fallback mechanisms for invalid inputs in the mia_bench evaluation process. * Fix launch error when num proc = 1 * Refactor VLLM model parameter handling to simplify distributed execution logic. Remove redundant checks for tensor parallelism and streamline generation parameter settings by eliminating unused temperature and top_p configurations. * Refactor VLLM message handling to prioritize image URLs before text content. Remove unused distributed executor backend parameter for cleaner execution logic. * feat(vllm): Set default max_new_tokens to 4096, temperature to 0, and top_p to 0.95 * docs: Update lmms-eval-0.4 documentation with images and installation instructions * docs: Update lmms-eval-0.4 documentation to include backward compatibility check * refactor: Simplify server config instantiation in utils files * docs: Update supported tasks count in README * Update docs * Fix mathverse bugs * docs: Update images in lmms-eval-0.4.md * docs: Remove API Benefits and Upcoming Benchmarks sections * docs: Update image URL for Unified Message Interface * docs: Fix typo in LMMS-Eval v0.4 documentation Corrected "27.8/16.40" to "27.8/26.40" in the performance comparison table. Also corrected "16.78/13.82" to "16.78/15.82" in the performance comparison table. * fix(docs): Correct typo in LMMS-Eval v0.4 performance comparison table * refactor(docs): Refactor LMMS-Eval v0.4 performance table for clarity * Update docs --------- Co-authored-by: kcz358 <kaichenzhang358@outlook.com> Co-authored-by: Cong <101887866+pbcong@users.noreply.github.com> Co-authored-by: model <noreply@API.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before you open a pull-request, please check if a similar issue already exists or has been closed before.
When you open a pull-request, please be sure to include the following
If you meet the lint warnings, you can use following scripts to reformat code.
Thank you for your contributions!