[cool project / potential collaboration] Weval

[simonaszilinskas]

Just had a call with people behind Weval. As we're talking more and more about automating benchmarks on top of crowdsourced data, it could be interesting to propose a collaboration with them around our datasets. For example, we could take part of our dataset, let's say on French cuisine, study the human preferences and then construct a benchmark out of them.

Their team was great and their product is very cool !

[KennethEnevoldsen]

EuroEval might also be worth a consideration here - generally though sampling datasets from comparia is somehting we have also looked into (probably more sample a problem space and then generative, revise and publish).

e.g. we had a linguist look at common mistakes in danish in the leaderboard responses

[KennethEnevoldsen]

So far, Nathalie Norman (from DFM, I don't know her GitHub handle, but will send this message to her) has done a deep dive into what grammatical errors and odd language structure are common in LLMs. We plan to use this to construct evaluation sets targeting these capabilites.

[KennethEnevoldsen]

A follow up on that is probably also to examine if people prefer model based on these attributes or whether it is more factual, stylistics or other errors than are in the forefront.

[Linguistcoder]

Hello! I'm Nathalie. So what I am working on is: a) a deep analysis of non-fluent language in LLMs, b) a preference dataset based on that analysis with corrected examples. I think it could be relevant to split such datasets into domains/topics.

[simonaszilinskas]

Nice to meet you @Linguistcoder ! I think the Danish datasets will be very interesting for this

A follow up on that is probably also to examine if people prefer model based on these attributes or whether it is more factual, stylistics or other errors than are in the forefront.

We're starting to explore this on #273 and potentially with a research partnership with Dauphine-PSL.

[KennethEnevoldsen]

Ahh right, we could probably do a similar thing but with "fluent vs non-fluent", with the central question being "how much does people care about fluency?" (I suspect it is a tier and fluency is not at the top)