The supported tasks of nemo-evaluator-launcher are much less than nemo-evaluator

[mianzhang]

Take bfcl as an example:

>>> nemo-evaluator-launcher ls
=====================================================
harness: bfcl
container: nvcr.io/nvidia/eval-factory/bfcl:25.10
                                                     
task                               endpoint_type     
-----------------------------------------------------
bfclv2_ast_prompting               chat              
bfclv3_ast_prompting               chat              
-----------------------------------------------------
  2 tasks available
=====================================================

However, if I enter the bfcl container:

>>> nemo-evaluator ls
bfcl: 
  * bfclv3
  * bfclv3_ast
  * bfclv3_ast_prompting
  * bfclv2
  * bfclv2_ast
  * bfclv2_ast_prompting

Are these expected? Or how to make nemo-evaluator-launcher also support the full list of tasks?

Thanks!

[marta-sd]

Hi @mianzhang , our goal is to have a curated set of tasks in the launcher. We wanted to avoid confusing users with too many variants of a given benchmark.

However, the current approach might not provide enough flexibility for more advanced users who prefer to run a different variant. We are working on a solution that would give more freedom to the advanced users, without introducing noise for the rest.

As a temporary work-around you could try modifying this section:

Evaluator/packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/resources/mapping.toml

Lines 260 to 267 in e5207b4

	[bfcl]
	container = "nvcr.io/nvidia/eval-factory/bfcl:25.10"
	[bfcl.tasks.chat.bfclv2_ast_prompting]
	required_env_vars = []
	[bfcl.tasks.chat.bfclv3_ast_prompting]
	required_env_vars = []

to expose the task you need. If e.g. you'd like to run bfclv3_ast, you should add:

[bfcl.tasks.chat.bfclv3_ast]
required_env_vars = []

Let me know if that unblocks you. I'll update this thread once we have a proper solution.

[mianzhang]

Thanks for the quick reply! And it works by adding tasks to the mapping.toml file. Looking forward to the updates.

[marta-sd]

Hi @mianzhang, we have introduced the needed functionality in PR #523 :)

Starting from nemo-evaluator-launcher==0.1.50 we have full parity between tasks from the underlying container and the launcher. Also, you can now check the tasks parameters with a new command:

nemo-evaluator-launcher ls task <task_name>

which should simplify configuring the tasks.

cc @agronskiy

[mianzhang]

Cool, will try. Thanks for the effort.