can not reproduce the result

[TheNewBeeKing]

Thanks for your fantastic work! When I tried to reproduce the evaluation result following the quick-start, I found the output EM and F1 were very small:

F1: 0.08271649461649468
EM: 0.074
MBE: 1.0
Compression Ratio: 0.6203602461794725
Peak Token Compression: 634.2591792656588
Dependency Compression: 318.0281863573691
Peak Token Non-Compression: 1111.0507559395248
Dependency Non-Compression: 637.0007221557524

I have no idea that whether I wrongly followed the quick-start instruction or there were some problems in the model parameters in Huggingface.

[BobbyZhouZijian]

Hi. Are you evaluating using your own trained model or using our released model?

[TheNewBeeKing]

I downloaded the parameter weights from https://huggingface.co/Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release/tree/main.
And then I edited the MODEL_PATH in start_vllm.sh and started the script, but I found there were relatively few "Exact_match" in the output json

[BobbyZhouZijian]

My educated guess is that the RAG server has not started properly. So all searches fail. Could you double check this?

[TheNewBeeKing]

I have cheked it again. It seems that the rag server works fine.

and the vllm server could also work normally.

[chenzhuofu]

I encountered the same issue. Do you have any suggestion? @BobbyZhouZijian

[chenzhuofu]

data:
data_source                                                 popqa
prompt          [{'content': 'You will answer multiple complex...
ability                                            fact-reasoning
reward_model    {'ground_truth': {'target': [['Alice Rohrwache...
extra_info             {'indices': [6718, 6719], 'split': 'test'}
hash            345278a73262e534c4cbdf01f240a411d553f4b168a1ad...
Name: 10820, dtype: object

==================iteration_cnt: 0==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant


response:
{'id': 'cmpl-b63d95d4fe364dd5b311dae56f6fd220', 'object': 'text_completion', 'created': 1761526370, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': '<think>\nThe first question asks about the director of "The Wonders," and the second question asks about the director of "Fear." To find the answers, I will search for information on the directors of these films.\n\n<search> Director of "The Wonders" ', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</search>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 298, 'total_tokens': 356, 'completion_tokens': 58}}

==================iteration_cnt: 1==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant
<think>
The first question asks about the director of "The Wonders," and the second question asks about the director of "Fear." To find the answers, I will search for information on the directors of these films.

<search> Director of "The Wonders"</search><information>
[HINT]You have 5 turns left.[/HINT]
Doc 1(Title: "The Wonders (film)") Wonders"" premiered at the 2014 Cannes Film Festival to positive reviews. On review aggregator Rotten Tomatoes, the film holds an approval rating of 95% based on 63 reviews, with an average rating of 7.3/10. The website's critical consensus reads, """"The Wonders"" offers a charming coming-of-age tale that doubles as a quietly effective tribute to a vanishing way of life."" On Metacritic, the film has a score of 76 out of 100, based on 17 critics, indicating ""generally positive reviews"". The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to
Doc 2(Title: "The Wonders (film)") The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to compete for the Palme d'Or at the 2014 Cannes Film Festival; it won the Grand Prix. Gelsomina and her three younger sisters live with their parents on a farm in the Italian countryside. As the eldest child, she takes on a heavier burden, taking care of her younger sisters and assisting her father with bee keeping. One day as the family is playing on the beach after a long day, they are told to quit making so much noise
Doc 3(Title: "Wonder (film)") had departed from the project. In October 2014, John Krokidas was reported to be directing the film, though in April 2015, Paul King was hired to direct instead. Steven Conrad was writing the script at that time. On April 14, 2016, Jacob Tremblay was cast to play the lead role, Auggie Pullman, while Julia Roberts was in talks to play Auggie's mother. On May 5, 2016, Roberts' casting was confirmed, and Stephen Chbosky was set as the film's director. On June 27, 2016, Owen Wilson joined to play Auggie's father. On July 11, 2016, newcomer Noah Jupe joined the cast

</information>

response:
{'id': 'cmpl-8f276b0645a041aea7665dde06957422', 'object': 'text_completion', 'created': 1761526370, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': ' <search> Director of "Fear. Enola Holmes" ', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</search>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 872, 'total_tokens': 887, 'completion_tokens': 15}}

==================iteration_cnt: 2==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant
<search> Director of "Fear. Enola Holmes"</search><information>
[HINT]You have 4 turns left.[/HINT]
Doc 1(Title: "Adaptations of Sherlock Holmes") films were Otto Lagoni, Einar Zangenberg, Lauritz Olsen, and Alwin Neuss. In 1911 the American Biograph company produced a series of 11 short comedies based on the Holmes character with Mack Sennett (later of Keystone Kops fame) in the title role. By 1916, Harry Arthur Saintsbury, who had played Holmes on stage hundreds of times in Gillette’s play, reprised the role in the 1916 film ""The Valley of Fear"". The next significant cycle of Holmes films were produced by the Stoll Pictures company in Britain. Between 1921 and 1923 they produced a total of 47 two-reelers, all featuring noted West
Doc 2(Title: "Sherlock Holmes (2010 film)") Sherlock Holmes (2010 film) Sir Arthur Conan Doyle's Sherlock Holmes, also known simply as Sherlock Holmes, is a British-American 2010 steampunk mystery film directed by Rachel Lee Goldenberg and produced by independent American film studio The Asylum. It features the Sherlock Holmes characters created by Sir Arthur Conan Doyle, though it follows an original plot. The film details an unrecorded case in which eccentric detective Holmes and his companion Dr. Watson investigate a series of unusual monster attacks and a plot to destroy London. Gareth David-Lloyd plays Dr. John Watson and Ben Syder, making his film debut, plays Sherlock Holmes.
Doc 3(Title: "Young Sherlock Holmes") Young Sherlock Holmes Young Sherlock Holmes (also known with the title card name of ""Pyramid of Fear"") is a 1985 American mystery adventure film directed by Barry Levinson and written by Chris Columbus, based on the characters created by Sir Arthur Conan Doyle. The film depicts a young Sherlock Holmes and John Watson meeting and solving a mystery together at a boarding school. Teenagers Sherlock Holmes and John Watson meet and become good friends as students at Brompton Academy, a school in London. Watson is introduced to Elizabeth Hardy, who is Holmes' love interest. He is also introduced to Rupert

</information>

response:
{'id': 'cmpl-d88910b4b4384076b9ae684c3d820788', 'object': 'text_completion', 'created': 1761526371, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': ' <search> Director of "The Wonders" ', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</search>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 787, 'total_tokens': 800, 'completion_tokens': 13}}

==================iteration_cnt: 3==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant
<search> Director of "The Wonders"</search><information>
[HINT]You have 3 turns left.[/HINT]
Doc 1(Title: "The Wonders (film)") Wonders"" premiered at the 2014 Cannes Film Festival to positive reviews. On review aggregator Rotten Tomatoes, the film holds an approval rating of 95% based on 63 reviews, with an average rating of 7.3/10. The website's critical consensus reads, """"The Wonders"" offers a charming coming-of-age tale that doubles as a quietly effective tribute to a vanishing way of life."" On Metacritic, the film has a score of 76 out of 100, based on 17 critics, indicating ""generally positive reviews"". The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to
Doc 2(Title: "The Wonders (film)") The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to compete for the Palme d'Or at the 2014 Cannes Film Festival; it won the Grand Prix. Gelsomina and her three younger sisters live with their parents on a farm in the Italian countryside. As the eldest child, she takes on a heavier burden, taking care of her younger sisters and assisting her father with bee keeping. One day as the family is playing on the beach after a long day, they are told to quit making so much noise
Doc 3(Title: "Wonder (film)") had departed from the project. In October 2014, John Krokidas was reported to be directing the film, though in April 2015, Paul King was hired to direct instead. Steven Conrad was writing the script at that time. On April 14, 2016, Jacob Tremblay was cast to play the lead role, Auggie Pullman, while Julia Roberts was in talks to play Auggie's mother. On May 5, 2016, Roberts' casting was confirmed, and Stephen Chbosky was set as the film's director. On June 27, 2016, Owen Wilson joined to play Auggie's father. On July 11, 2016, newcomer Noah Jupe joined the cast

</information>

response:
{'id': 'cmpl-0b51a8ef81f7466a80ed53190ac7acd3', 'object': 'text_completion', 'created': 1761526371, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': ' <search> Director of "Fear, Inc." ', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</search>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 827, 'total_tokens': 840, 'completion_tokens': 13}}

==================iteration_cnt: 4==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant
<search> Director of "Fear, Inc."</search><information>
[HINT]You have 2 turns left.[/HINT]
Doc 1(Title: "Fear, Inc.") Fear, Inc. Fear, Inc. is a 2016 American comedy horror film directed by Vincent Masciale and written by Luke Barnett. The film stars Lucas Neff, Caitlin Stasey, Chris Marquette, and Stephanie Drake. A young woman, Jennifer Adams (Abigail Breslin) is seemingly chased by an attacker in a parking garage. She grabs her phone and attempts to end the attack by insisting that ""it"" wasn't part of the plan. The person on the other end of the line puts her on hold. Jennifer then spots the man robed and masked approaching her with a nail studded baseball bat. As she rushes
Doc 2(Title: "Fear, Inc.") Masciale. Principal photography on the film began in early August 2015 in Los Angeles. Fear, Inc. Fear, Inc. is a 2016 American comedy horror film directed by Vincent Masciale and written by Luke Barnett. The film stars Lucas Neff, Caitlin Stasey, Chris Marquette, and Stephanie Drake. A young woman, Jennifer Adams (Abigail Breslin) is seemingly chased by an attacker in a parking garage. She grabs her phone and attempts to end the attack by insisting that ""it"" wasn't part of the plan. The person on the other end of the line puts her on hold. Jennifer then spots the man
Doc 3(Title: "Fear, Inc.") slashed Lindsey's throat. Before Abe kills Joe, he explains he can't let him leave without experiencing his all-time favorite death scenes from films. He holds a knife to Joe's throat and says ""Cut to black bitch!"" The film ends with a phone ringing and Judson (Leslie Jordan), ""Fear, Inc.'s"" phone operator answers the phone by saying their service is sold out. He then tells the rest of ""Fear, Inc."" that they've got another customer. The film is directed by Vincent Masciale based on the script by Luke Barnett, both are also producing the film along with Heather Kasprzak and Natalie

</information>

response:
{'id': 'cmpl-4332773a158e45f486f20b340e883b20', 'object': 'text_completion', 'created': 1761526371, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': ' <search> Director of "The Wonders" ', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</search>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 767, 'total_tokens': 780, 'completion_tokens': 13}}

==================iteration_cnt: 5==================
prompt_message:
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
You will answer multiple complex questions using iterative reasoning, summarization, and web search.

At each step, you will see the questions, a cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions are provided). Your task is to:

1. Perform reasoning and update a cumulative, concise summary within <think> ... </think>. This acts as persistent memory and must include all essential information from previous <think> and <information> tags.

2. Then choose one of the following actions:
   - If any question remains unanswered, issue a single query for one question inside <search> ... </search>. The query should consist of keywords or a short phrase. Only search one question at a time.
   - If all questions are answered, provide the final answers—separated by semicolons—within <answer> answer1; answer2; ... </answer>. The answers must be concise, contain only essential words, and avoid any explanations.

Important:
- Always follow this structure after <information> or the initial questions: <think> ... </think><search> ... </search> or <think> ... </think><answer> ... </answer>.
- Do not search multiple queries or questions simultaneously.

Answer the following questions: Who was the director of The Wonders?; Who was the director of Fear?
<|im_end|>
<|im_start|>assistant
<search> Director of "The Wonders"</search><information>
[HINT]You have 1 turn left. You must answer the question now.[/HINT]
Doc 1(Title: "The Wonders (film)") Wonders"" premiered at the 2014 Cannes Film Festival to positive reviews. On review aggregator Rotten Tomatoes, the film holds an approval rating of 95% based on 63 reviews, with an average rating of 7.3/10. The website's critical consensus reads, """"The Wonders"" offers a charming coming-of-age tale that doubles as a quietly effective tribute to a vanishing way of life."" On Metacritic, the film has a score of 76 out of 100, based on 17 critics, indicating ""generally positive reviews"". The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to
Doc 2(Title: "The Wonders (film)") The Wonders (film) The Wonders () is a 2014 internationally co-produced drama film directed by Alice Rohrwacher. It was selected to compete for the Palme d'Or at the 2014 Cannes Film Festival; it won the Grand Prix. Gelsomina and her three younger sisters live with their parents on a farm in the Italian countryside. As the eldest child, she takes on a heavier burden, taking care of her younger sisters and assisting her father with bee keeping. One day as the family is playing on the beach after a long day, they are told to quit making so much noise
Doc 3(Title: "Wonder (film)") had departed from the project. In October 2014, John Krokidas was reported to be directing the film, though in April 2015, Paul King was hired to direct instead. Steven Conrad was writing the script at that time. On April 14, 2016, Jacob Tremblay was cast to play the lead role, Auggie Pullman, while Julia Roberts was in talks to play Auggie's mother. On May 5, 2016, Roberts' casting was confirmed, and Stephen Chbosky was set as the film's director. On June 27, 2016, Owen Wilson joined to play Auggie's father. On July 11, 2016, newcomer Noah Jupe joined the cast

</information>

response:
{'id': 'cmpl-588dec7a453d4dc4a2e87402ca4c8d4a', 'object': 'text_completion', 'created': 1761526371, 'model': 'Mem-Lab/Qwen2.5-7B-RL-RAG-Q2-EM-Release', 'choices': [{'index': 0, 'text': ' \n<think>\nThe information provided indicates that "The Wonders" was directed by Alice Rohrwacher. It was selected to compete for the Palme d\'Or at the 2014 Cannes Film Festival and won the Grand Prix. Therefore, the answer to the first question is Alice Rohrwacher.\n\nFor the second question, "Who was the director of Fear?," I would search for the director of the film "Fear."\n\n<search> Director of "Fear" </search>\n</think>\n', 'logprobs': None, 'finish_reason': 'stop', 'stop_reason': '</answer>', 'prompt_logprobs': None}], 'usage': {'prompt_tokens': 834, 'total_tokens': 941, 'completion_tokens': 107}}

Here is a data point from Quick Start for your information.

[BobbyZhouZijian]

Hi. Sorry for the late responses. The issue seems to be with the eval.py script. We have updated the script in the latest commit 6f22061.

Please try it out and let me know if the issue persists.

[TheNewBeeKing]

Hi. Sorry for the late responses. The issue seems to be with the eval.py script. We have updated the script in the latest commit 6f22061.

Please try it out and let me know if the issue persists.

Thanks for reply, I also tried the ppo training script at that time, and I found that it could not converge, which showed high entropy and gradnorm even with low learning rate, but I did not change any core hyper-params. Can you give me some advice?

[lujiarui-iie]

Same issue.
Have you successfully reproduced the results?

Hi. Sorry for the late responses. The issue seems to be with the eval.py script. We have updated the script in the latest commit 6f22061.
Please try it out and let me know if the issue persists.

Thanks for reply, I also tried the ppo training script at that time, and I found that it could not converge, which showed high entropy and gradnorm even with low learning rate, but I did not change any core hyper-params. Can you give me some advice?

[isyuhaochen]

Hi. Sorry for the late responses. The issue seems to be with the eval.py script. We have updated the script in the latest commit 6f22061.

Please try it out and let me know if the issue persists.

We used this latest script, but the same issue still occurs.