NavQA eval issue

[Yixun-Hu]

Thank you so much for your wonderful work. Other than #19 , I also find some other issues with this dataset.

Problem Overview of NavQA dataset

ALL position-related questions in the NavQA dataset have identical ground truth answers, which is a data annotation error that affects evaluation accuracy.

Cause Analysis

1. Time Parsing Issue
   Human annotations: Time stamps like 7:57:53 (morning 7:57 AM)
   Actual caption time range: 10:49:45 to 11:03:17 (morning 10:49-11:03 AM)
   Parsed timestamps: 1673873873.0 (corresponding to 7:57:53 AM)
   Problem: Parsed timestamps are ~3 hours earlier than the actual caption time range
2. Caption Index Lookup Failure
   Due to incorrect time parsing, np.argmax(diff > 0) - 1 returns -1
   All position questions fail to find corresponding captions
   context_captions becomes an empty list
3. Fallback Mechanism Issue
   When context_captions is empty, the code uses a fallback mechanism
   All position questions end up using the same caption (index 162)
   This caption has position: [-0.39364902000000007, 0.0023255999999999897, -0.011701379999999999]
   Data Flow Analysis

Human CSV annotation → Time parsing → Caption lookup → Position extraction
     ↓                    ↓              ↓                ↓
  "7:57:53"        → 1673873873.0  →  No match    →  Fallback to same position

Code Location

File: remembr/scripts/question_scripts/form_question_jsons.py
Function: parse_answer() (lines 60-64):

elif q_type == 'position':
    if len(context) == 1:
        out_dict = {
            'position': context[0]['position']  # Always same position
        }

Impact

Evaluation accuracy: All position questions have identical ground truth

Current Status

Script runs without errors but produces the same caption (index 162), with position error ~ 200m
This is a data quality issue that needs to be addressed at the annotation level rather than just the code level.

[wzy1849155537]

非常感谢作者出色的工作和您细心的思考，我在评估也是发现了很多的问题，尤其是在数据集质量的问题方面，使用13b模型生成captions之后，通过form_questions_json.py脚本生成qa问答之后，我发现这些答案存在大量的不准确性，时间问题存负数，位置问题很多都是重复的，二元问题在评估过程中出现模型幻觉问题，评估脚本的指标与论文存在差异，还想请问您有没有遇到类似的问题，是什么解决的呢

[Yixun-Hu]

Qwen2.5VL-3B has hallucidation too. I think for that, you can try API as well. I think the difference from original paper is common. Dataset was highly possibly to be not set up correctly.

[Yixun-Hu]

Actually I solve this issue. the utc time will convert to the local time, which is different with regard to the region. A more robust way to do so is to use: dt_utc=datatime.fromtimestamp(ts, tz=timezone.utc) as the standard timestamp. Then no time shift needs to do further.

[Yixun-Hu]

Hi @bksuh !

I think the fastest way to solve this may be redownloading the navqa dataset following the instructions. I got the data.csv right after I finish all preprocessing steps in eval.md.

If you cannot find it, I highly suspect that there are also other problems in your dataset along with this problem.

[bksuh]

@Yixun-Hu

Thanks for the reply! :>
I actually realized it was already included in the original repo and re-downloaded it right after asking you.
I deleted the post once I found it, but it seems you already saw my question 👍

Appreciate your time and help!

[bksuh]

Hi @Yixun-Hu,

I finally managed to run eval.py and encountered the same issue you mentioned with form_questions_jsons.py.
I saw your note about fixing it by converting timestamps to UTC:

“Actually I solved this issue. The UTC time will convert to the local time, which is different with regard to the region.
A more robust way to do so is to use dt_utc = datetime.fromtimestamp(ts, tz=timezone.utc) as the standard timestamp.”

Could you please clarify which file and which part of the code you modified to apply that fix?
It would really help to know where exactly datetime.fromtimestamp was used so I can make the same change.

Thanks in advance for your help!

[Yixun-Hu]

This thing is a little bit complicated. we can chat through a zoom or something? I am happy to help. Please contact my email.

[wang-qiuchen]

https://huggingface.co/datasets/Qiuchen-Wang/NavQA_Revised may be helpful.
Revised to the time zone of UT capmus and aligned the time annotations as much as possible