Improve documentation to help reproduce the great work

[kailigo]

Great work, thank you for the contribution.

When possible, could you please improve documentation about how to reproduce the results. The current version can hardly achieve this. Things include (not limited to)

• The third-party dependencies. The current version seems highly customized for a specific infra system
• The online evaluation script, not all the listed "TASKS" in script are supported
• The training script. There are data or model files dependencies in the training scripts. However, not all of them are easy to infer and locate. If it would be super helpful if more documentation on these dependency can be included.

Thanks a again for the contribution.

[LanXingXuan]

Thank you very much for your timely and valuable feedback!

We’ll be updating and improving the documentation soon to make it easier to reproduce our results. In the meantime, here are a few temporary solutions that might help:

• Environment setup
  We apologize for missing the requirements.txt file in the initial release, which caused some environment configuration issues. (The original version inherited dependencies from the base code, and we later upgraded and added several packages.)
  We’ve now updated the requirements.txt and also provided an exported environment.yml. You can try reinstalling your environment using these updated files.

• Training scripts
  We’ve updated the LLM_VERSION field with clearer comments. You can start the 4th and 5th stages of online fine-tuning directly using the checkpoint released on Hugging Face.
  Please also download the YAML file corresponding to DATA_VERSION from MCG-NJU/StreamForest-Annodata and place it under the ./anno directory.

• Testing issues
  If testing scripts fail to start properly, it might be due to incorrect video data paths. Please check the utils.py file under each test task in ./lmms_eval/tasks. Some tasks include a DATA_LIST entry, which means you’ll need to manually specify the directory where your local test videos are stored.

Again, thank you for your helpful suggestions and for taking the time to try out our work!
If these fixes don’t fully resolve the issue or you have further questions, please don’t hesitate to reach out — we’d be happy to help.

[kailigo]

Thanks for the clarification.

However, I think there are still some gaps from this repo can be easily utilized. It is hardly possible to get the expected results from the current version of the README. To name a few issues:

• StreamForest/lmms_eval/api/metrics.py

  Line 149 in a64f24e

  exact_match = evaluate.load("/mnt/petrelfs/zengxiangyu/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/9d3b67e0c429cd7460b2b05aab53419b48eea369b73e1d9f185a56ca90c373d4/exact_match.py")

• 2025-10-14 20:23:15.588 | ERROR | main:cli_evaluate_single:277 - Tasks were not found: streamingbench. Try lmms-eval --tasks list for list of available tasks

• raise ValueError(f"Attempted to load model '{model_name}', but no model for this name found! Supported model names: {', '.join(MODEL_REGISTRY.keys())}")

While these issues might be addressed if delving deep into the code, it costs much extra effort to do this. I guess one good practice, if possible, is to test the code in an isolated env different from the one that was used to develop, which helps rule out potential issues from users.

Thanks a again for the great effort.

[kailigo]

@LanXingXuan , could you help improve the code release and make it easier to replicate the results? Thanks a lot.

[LanXingXuan]

Apologies for the delayed response due to our recent busy schedule. We are now preparing to verify our code in a completely new environment and will continue to fix potential bugs that may arise in heterogeneous settings. Please stay tuned for our updates!