epic: Evaluating Search Models

[Yip-Jia-Qi]

Project

1. Search Eval without tool calling (@new5558)
2. Visualising Tool Calling Traces (@DESU-CLUB)
3. MixEval for Search (KIV)

MixEval for Search

The current MixEval is based on the LMArena human preference leaderboard
Its main selling point is that it aligns well to the human preference on LMArena
However, a limitation of MixEval is that it does not account for new tool-calling web-search models such as Jan-v1
LMArena has another benchmark called SearchArena
This could be good source material for a web-search based MixEval, which spans across all the major search-based evals
Jan Exam v1, SimpleQA, FreshQA, SealQA

Approach

• Replicate the embedding strategy of MixEval
• Mine a dataset of human search queries from SearchArena
• Prepare a plot across all the datasets just like in MixEval

[new5558]

Search Eval without Tool Calling

Sanity check of "fast" search eval

1. Correlation Analysis

[new5558]

Embedding Analysis of search arena vs general arena

[Yip-Jia-Qi]

https://colab.research.google.com/drive/1Euim1Tz-jqT1zuBTkH_RfwpEENp_29R9?usp=sharing#scrollTo=fAcuImw5EP5T

Likely issue with the approach:
For false premise questions in FreshQA, the step-by-step evaluation approach isn't able to come with with a good grade

from discussion with @new5558

[new5558]

Problem with FreshQA with different judges

FreshQA

• Problem with different judges

Model	Judge Model	Score
deepseek3.1	deepseek3.1	0.504
deepseek3.1	kimi_k2_0905	0.771
deepseek-chat-v3.1	gpt-5	0.568

SimpleQA

• Do not have a problem with different judges

Model	Judge Model	Score
deepseek-chat-v3.1	deepseek3.1	0.943
deepseek-chat-v3.1	kimi_k2_0905	0.943
deepseek-chat-v3.1	gpt-5	0.945

Example of when model has problem with FreshQA

Q:

When did Meta move its headquarters to Austin?

A:

Based on my search results, I need to clarify an important distinction. Meta (formerly Facebook) has not moved its headquarters to Austin, Texas. The company's headquarters remains in Menlo Park, California.\n\nHowever, there have been several recent developments regarding Meta's operations in Texas:\n\n1. Meta has significant office operations in Austin and has been investing in Texas facilities\n2. Recent news (January 2025) indicates Meta is considering reincorporating in Texas, which would be a legal/corporate structure change, not a physical headquarters relocation\n3. Meta has been moving some teams, particularly content moderation operations, to Texas\n\nThe search results show that Meta has offices and operations in Austin, but there's no evidence that the company has actually moved its corporate headquarters from California to Austin. The headquarters remains in Menlo Park, California, where Meta (Facebook) has been headquartered since 2011

GPT-5 judge: false premise but misleading -> incorrect
Qwen 235B 2507: false premise -> correct

[new5558]

SimpleQA vs Stepwise accuracies

model	simpleqa_acc	stepwise_acc
gpt-5-mini	0.9616	0.9422
gpt-4.1	0.93	0.9689
qwen-253-think-2507	0.9162	0.9533
qwen-30-instruct-2507	0.9133	0.9089
gpt-4.1-mini	0.8533	0.9378
kimi_k2_0905	0.805	0.7578
qwen-next_80b	0.7967	0.92
gpt-4.1-nano	0.7067	0.7578
llama_3.3_70b	0.5866	0.7022
llama_3.1_70b	0.4933	0.7489

generated trajectory by: gpt-5-mini
judge: gpt-5-mini

Pearson correlation (per-model accuracy): 0.8496

[Yip-Jia-Qi]

🌳 Issue Transfer Tree Structure

This issue and all its sub-issues are being transferred. Below is the complete hierarchy:

 menloresearch/jan-incub-agents#3 - epic: Evaluating Search Models [open]
   menloresearch/jan-incub-agents#2 - feat: visualising search traces [open]

Transfer Details:

• Source Repository: menloresearch/jan-incub-hub
• Destination Repository: menloresearch/jan-incub-agents
• Total Issues: 2
• Transfer Date: Wed 17 Sep 2025 14:00:38 +08