No video paths for the `ovbench_tao.yaml` on the OVBench task

[kailigo]

There are 4 subtask for the ovbench.

• ovbench_tao
• ovbench_coin
• ovbench_hirest
• ovbench_ava
  Unlike the other 3 tasks which can find the corresponding videos on MCG-NJU/OVBench, there is no specific folder corresponding to ovbench_tao task.

Indeed, one can find the corresponding frames (not the video) across multiple folders, as an examples shown below

{
        "fps": 30,
        "middle_frame_timestamp": 12.0,
        "question": "What direction do you think the person [0.64, 0.00, 1.00, 1.00] may move towards in the next second?",
        "answer": "stationary",
        "sub_answer_type": "FuturePrediction-1",
        "image_file": "val/HACS/Spinning_v_8rhkybFaad4_scene_0_0-3734/frame0361.jpg",
        "answer_type": "FuturePrediction",
        "video": "Spinning_v_8rhkybFaad4_scene_0_0-3734",
        "candidates": [
            "stationary",
            "below",
            "right",
            "left-top"
        ],
        "start": 0.0,
        "end": 12.0
    },

but the code, as least it is right now, does not support loading video frames.

There is no place introducing how to run evaluation on this task and how the numbers from different subtasks are aggregated to get the results reported in the paper.

Could the authors help resolve this? Thx

[LanXingXuan]

During the test-time processing, the field "image_file": "val/HACS/Spinning_v_8rhkybFaad4_scene_0_0-3734/frame0361.jpg" does not play an active role in data loading; it is included only to indicate the annotation source (HACS) and to assist debugging (e.g., to verify that the frame corresponding to the end timestamp — frame0361.jpg — is read correctly).

In practice, our code actually reads the entry "video": "Spinning_v_8rhkybFaad4_scene_0_0-3734". That entry denotes a directory containing the video’s discrete, sampled frames (the example shown uses 30 FPS; other videos may have different FPS values — see each annotation entry for details). Concretely, see the load_video() function in lmms_eval/models/streamforest.py. When load_video() encounters a frame-folder (i.e., a path not ending with .mp4), it dispatches to llava/video_utils.py’s read_frames_img() function. read_frames_img() loads and sorts all discrete frame images in the folder and uses the "fps" field from the annotation to determine which frame indices correspond to the annotated start/end time interval.

Our frame storage is organized under paths of the form pnorm2:s3://tao/frames/val/Spinning_v_8rhkybFaad4_scene_0_0-3734. The prefix pnorm2:s3://tao/frames/val/ points to a collection of discrete-frame folders from multiple sources (for example, the annotation "image_file": "val/AVA/lWXhqIAvarw_scene_9_57855-58695/frame0051.jpg" corresponds to the actual read path pnorm2:s3://tao/frames/val/lWXhqIAvarw_scene_9_57855-58695).

If you need the original video files, you may download the corresponding tao video zip packages from the OVBench source-video releases (they include videos from HACS, AVA, etc., as specified in the annotation files) and extract them to the same root path so that the frame-folder layout above is available to the evaluation code.

[LanXingXuan]

Regarding task type correspondence, OVBench consists of seven major task categories — FP, THV, PM, SP, STP, and TP. These categories correspond to the prefix of the "answer_type" field in the annotations (for example, "answer_type": "PastMemory" corresponds to task PM, and "answer_type": "FuturePrediction" corresponds to task FP).

Each major task category further includes two or three subtasks. Specifically, AVA corresponds to action-related subtasks, CION+HIREST correspond to event-related subtasks, and TAO corresponds to object-related subtasks.
A simple way to verify this mapping is to check the performance tables in the original OVBench paper: within each major task category, the three subtasks are arranged from left to right in the order — AVA (left), CION+HIREST (middle), and TAO (right).
For task categories with only two subtasks, one column is naturally missing, but the relative left-to-right ordering remains consistent.

We plan to release an additional utility soon that will automatically compute the accuracy for each OVBench subtask directly from evaluation logs, making subtask-level performance analysis more convenient.

[yingift]

Hi. THKS for the reply. I am still confused by the TAO subset.

1. In the Tab. 11 of paper, it seems like a public dataset. But according to your reply, it consists of videos from several datasets?
2. In you reply, 'tao 视频 zip 包（它们包括来自 HACS、AVA 等的视频，如注释文件中所述）. Which annotation files have these info?
   Look for your reply and update.

[LanXingXuan]

@yingift Thanks for reaching out and for your question about the TAO subset.

TAO is an integrated dataset, composed of data from other datasets such as HACS and AVA. You can find its official repository here: TAO-Dataset/tao. You can also download the raw TAO videos directly from MCG-NJU/OVBench.

To avoid confusion, we have modified the data format of the TAO annotation. You can now determine the video source for each entry by checking the path included in the "video" field. The link to the JSON is: MCG-NJU/StreamForest-Annodata/blob/main/eval/OVBench/json/tao.json.

I hope this clears up the confusion. Please let us know if you have any other questions!