Create a task specification and verifier function for the medical reasoning dataset/benchmark of
HuatuoGPT-01 : https://arxiv.org/abs/2412.18925
The main idea is that we need more tasks specifications... What does that mean?
A task is a dataset consisting of prompts. each task has a build_dataset method, which sends a dataset of prompts to the episode generator.
An LLM will generate solutions to these, and the episode generator will invoke the task class to verify their correctness.
luckily, the dataset of prompts is already available on the HuatuoGPT-o1 github. they probably have an answer grader aswell, but it would need to get retrofitted in a relign task specification.
If this is implemented, we could then evaluate other models/rl algorithms on this specific benchmark!
Create a task specification and verifier function for the medical reasoning dataset/benchmark of
HuatuoGPT-01 : https://arxiv.org/abs/2412.18925
The main idea is that we need more tasks specifications... What does that mean?
A
taskis a dataset consisting of prompts. each task has abuild_datasetmethod, which sends a dataset of prompts to the episode generator.An LLM will generate solutions to these, and the episode generator will invoke the task class to verify their correctness.
luckily, the dataset of prompts is already available on the HuatuoGPT-o1 github. they probably have an answer grader aswell, but it would need to get retrofitted in a relign task specification.
If this is implemented, we could then evaluate other models/rl algorithms on this specific benchmark!