Evaluation Parameters for models on Office-Bench

[ArkaZhuo]

Thanks for the great work on Simia-OB! I'm trying to reproduce the results for Simia-OB (Qwen3-8B) on Office-Bench but haven't had much luck with the accuracy. Could you share the evaluation parameters or scripts used in the original experiments? Thanks again!

[1GaryLi]

Thank you for your interest in our work!

We have shared the evaluation codebase we used for OfficeBench. Our internal implementation is slightly different from the official codebase. Specifically:

1. We added a think step before each tool call.

2. We changed the Docker-based execution into local execution for faster runtime (all files are generated locally in local_workdir/).

3. We removed OCR-related tasks from the benchmark.

Because our SFT data was generated under the format used in this modified codebase , the trained model Simia-Agent/Simia-OfficeBench-SFT-Qwen3-8B is somewhat overfitted to local execution patterns. As a result, its generation format differs from the requirements in the official OfficeBench implementation.

We also recommend running multiple evaluations for 3-apps tasks to avoid high variances.

Please let me know if you have any other questions!