[Feature Request] Support unordered list of function calls for `expectedCall`

[beaufortfrancois]

Currently, expectedCall appears to be optimized for single-tool call validation. In more complex scenarios (like agents that trigger multiple parallel tool calls or tasks that can be solved using different tools in any order), a strict one-to-one or ordered comparison causes evaluation failures even when the model's logic is correct.

Example: If a prompt is: "Check the weather in London and Paris", the model might call get_weather(city='London') then get_weather(city='Paris'), or vice versa. Both should be considered a "Pass."

[swissspidy]

I thought @Kulikowski had already converted expectedCall to an array; all that's left to be done is to rename the property.

[Kulikowski]

expectedCall is an array now yes, I understand requirement as "having array of expectedCalls allow for ordered and unordered match" - There are cases where we want to ensure events happen in specific sequence (ordered) and cases where we don't care about order of calls, just completeness - like example of getWeather(London), getWeather(Paris) - it doesn't matter for me as eval creator in which order it will happen, and currently if we structure it in eval as London,Paris and model decides to call Paris,London the eval will fail while being correct in its intent.

TL;DR - allow for ordered vs unordered check of expectedCalls

[beaufortfrancois]

Something like matchStrategy would be nice!

 export type Eval = {
   messages: Message[];
   expectedCall: FunctionCall[] | null;
+  matchStrategy?: "ordered" | "unordered";
 };

[swissspidy]

Eventually we will need a more complex solution for this. I just discussed with @Kulikowski a scenario like this:

"Go to my store page, add products A, B, C, and then proceed to check out".

The order of function calls could be something like this:

1. Navigate to store page
2. Add product B
3. Add product A
4. Add product C
5. Check out

The order of steps 2-4 doesn't really matter, as the final result is the same.

In my own mcp-eval tool I solved this with an imperative zod-schema based validation where you could do any validation you would like, including verifying the state in the browser.

For a declarative approach like here, we could perhaps introduce nested sub-groups. Something like this:

matchStrategy: ordered,
expectedCalls: [
	navigateToPage,
	{
		matchStrategy: unordered,
		expectedCalls: [
			buyProductA,
			buyProductB,
			buyProductC,
		]
	},
	checkout
]

Edit: example updated, thanks!

[beaufortfrancois]

Your proposal looks great!

Nit: I believe you meant matchStrategy: "ordered" first, then matchStrategy: "unordered" in your example