Previously, simpleQA was used as a benchmark - with its help, they tried to form a general understanding of how the framework works as a whole. Now, based on new categories, we need to create our own lightweight benchmark, which will make it possible, among other things, to automatically check the integrity and quality.
You can use SilkQA, SimpleQA, etc. as a basis. Ultimately, the benchmark should not be too voluminous and expensive in terms of various resources.
Categories - #114
The expected volume at the first stage is 100+ tasks and no more than 300
Previously, simpleQA was used as a benchmark - with its help, they tried to form a general understanding of how the framework works as a whole. Now, based on new categories, we need to create our own lightweight benchmark, which will make it possible, among other things, to automatically check the integrity and quality.
You can use SilkQA, SimpleQA, etc. as a basis. Ultimately, the benchmark should not be too voluminous and expensive in terms of various resources.
Categories - #114
The expected volume at the first stage is 100+ tasks and no more than 300