[Question] How to eval textqa

[fanminshi]

Question

I looked at the eval script

CONV="conv_template"
CKPT_NAME="your_ckpt_name"
CKPT="checkpoints/${CKPT_NAME}"
EVAL="eval"
deepspeed moellava/eval/model_vqa_loader.py \
    --model-path ${CKPT} \
    --question-file ${EVAL}/textvqa/llava_textvqa_val_v051_ocr.jsonl \
    --image-folder ${EVAL}/textvqa/train_images \
    --answers-file ${EVAL}/textvqa/answers/${CKPT_NAME}.jsonl \
    --temperature 0 \
    --conv-mode ${CONV}

python3 -m moellava.eval.eval_textvqa \
    --annotation-file ${EVAL}/textvqa/TextVQA_0.5.1_val.json \
    --result-file ${EVAL}/textvqa/answers/${CKPT_NAME}.jsonl

what's ${CKPT_NAME}.jsonl in ${EVAL}/textvqa/answers/${CKPT_NAME}.jsonl ?

My ./textvqa/answers/ has llava-v1.5-13b.jsonl but my CKPT_NAME is llavaphi-2.7b-finetune-moe. Should I use llava-v1.5-13b.jsonl for eval instead?

[fanminshi]

ALso what's CONV="conv_template"?

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/fshi/.conda/envs/moellava/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/fshi/.conda/envs/moellava/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/fshi/.conda/envs/moellava/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/gpfs/home5/fshi/MoE-LLaVA/moellava/eval/model_vqa_loader.py", line 48, in __getitem__
    conv = conv_templates[args.conv_mode].copy()
KeyError: 'conv_template'