The supported tasks of nemo-evaluator-launcher are much less than nemo-evaluator

[mianzhang]

Take bfcl as an example:

>>> nemo-evaluator-launcher ls
=====================================================
harness: bfcl
container: nvcr.io/nvidia/eval-factory/bfcl:25.10
                                                     
task                               endpoint_type     
-----------------------------------------------------
bfclv2_ast_prompting               chat              
bfclv3_ast_prompting               chat              
-----------------------------------------------------
  2 tasks available
=====================================================

However, if I enter the bfcl container:

>>> nemo-evaluator ls
bfcl: 
  * bfclv3
  * bfclv3_ast
  * bfclv3_ast_prompting
  * bfclv2
  * bfclv2_ast
  * bfclv2_ast_prompting

Are these expected? Or how to make nemo-evaluator-launcher also support the full list of tasks?

Thanks!

[marta-sd]

Hi @mianzhang , our goal is to have a curated set of tasks in the launcher. We wanted to avoid confusing users with too many variants of a given benchmark.

However, the current approach might not provide enough flexibility for more advanced users who prefer to run a different variant. We are working on a solution that would give more freedom to the advanced users, without introducing noise for the rest.

As a temporary work-around you could try modifying this section:

Evaluator/packages/nemo-evaluator-launcher/src/nemo_evaluator_launcher/resources/mapping.toml

Lines 260 to 267 in e5207b4

	[bfcl]
	container = "nvcr.io/nvidia/eval-factory/bfcl:25.10"
	[bfcl.tasks.chat.bfclv2_ast_prompting]
	required_env_vars = []
	[bfcl.tasks.chat.bfclv3_ast_prompting]
	required_env_vars = []

to expose the task you need. If e.g. you'd like to run bfclv3_ast, you should add:

[bfcl.tasks.chat.bfclv3_ast]
required_env_vars = []

Let me know if that unblocks you. I'll update this thread once we have a proper solution.

[mianzhang]

Thanks for the quick reply! And it works by adding tasks to the mapping.toml file. Looking forward to the updates.

[marta-sd]

Hi @mianzhang, we have introduced the needed functionality in PR #523 :)

Starting from nemo-evaluator-launcher==0.1.50 we have full parity between tasks from the underlying container and the launcher. Also, you can now check the tasks parameters with a new command:

nemo-evaluator-launcher ls task <task_name>

which should simplify configuring the tasks.

cc @agronskiy

[mianzhang]

Cool, will try. Thanks for the effort.

[pruprakash]

QA RCCA Analysis

Date: 2026-04-20
Analyst: AI QA Agent (issues-rca skill)
Issue: #436 — The supported tasks of nemo-evaluator-launcher are much less than nemo-evaluator

---

1. Fix Reference

No single fix PR was identified. The issue was closed on 2026-01-07 as completed. Based on the issue body, the gap was that nemo-evaluator-launcher ls showed far fewer tasks than nemo-evaluator itself supported — the launcher's task registry was not kept in sync with the evaluator's benchmark registry. The fix involved expanding the launcher's task manifest to include missing benchmarks (e.g., bfcl).

---

2. Root Cause

The nemo-evaluator-launcher maintained its own task registry that was a manual subset of the full benchmark set supported by nemo-evaluator. When new benchmarks were added to the evaluator (e.g., BFCL, additional harnesses), they were not automatically reflected in the launcher's task list. The nemo-evaluator-launcher ls command only enumerated tasks defined in the launcher's own config directory, not by querying the evaluator's registry directly. This created a growing divergence over time where users could run tasks directly via the evaluator CLI that were not accessible via the launcher.

---

3. Trigger Config

Trigger conditions:

• User runs nemo-evaluator-launcher ls to discover available tasks
• A benchmark supported by nemo-evaluator (e.g., BFCL variants) is missing from the launcher's task registry
• User tries to submit a task via the launcher that they found via nemo-evaluator directly

Deterministic? Yes for any affected benchmark. Once a task is missing from the launcher registry, it is always missing until the registry is updated.

---

4. Nature of Bug

Primary classification: API/interface bug — missing parameter, breaking change (incomplete task registry in the launcher layer)

Impact scope: Users who rely on nemo-evaluator-launcher to orchestrate evaluation jobs on SLURM. Any benchmark supported by nemo-evaluator but absent from the launcher's registry is silently unavailable via the launcher workflow.

NOT affected: Users calling nemo-evaluator directly (without the launcher). TRT-LLM and vLLM deployment paths are unaffected.

---

5. Functional Test Coverage

Verdict: PARTIAL

Test	File	Key Config	What it covers
test_executor_interface_parity	evaluator/testcases/rcca/launcher/test_launcher_task_parity.py	No SLURM needed	Both executors expose the same required methods
test_env_var_forwarding_keys_identical	same	No SLURM needed	Both executors have env-forwarding capability
test_task_payload_required_keys	same	No SLURM needed	Task payload has required keys (benchmark, model, output_dir)

The tests verify structural parity between local and SLURM executors. They do not enumerate or compare the full task registry between launcher and evaluator.

---

6. Gaps and Limitations

Gap 1 — No registry enumeration test:
The most direct test would query nemo-evaluator-launcher ls output and compare it against the list of benchmarks registered in nemo-evaluator. This requires both packages installed and is not currently covered.

Gap 2 — Dynamic registry comparison (out of scope for unit test):
A complete parity test would need to dynamically import both the launcher and evaluator task registries and assert set(launcher_tasks) == set(evaluator_tasks). This is feasible but sensitive to version skew between the two packages.

Overall gap assessment: Medium regression risk. The structural parity tests guard executor interface consistency but do not directly prevent task registry divergence from recurring.

---

7. New Test Added

Field	Value
Test file	evaluator/testcases/rcca/launcher/test_launcher_task_parity.py
Test functions	test_executor_interface_parity, test_env_var_forwarding_keys_identical, test_task_payload_required_keys
QA repo	nmfw_tests (local: evaluator/testcases/rcca/launcher/)
PR	Pending
What it validates	Executor interface parity (local vs SLURM), env-var forwarding, task payload required keys
How it would catch a regression	If a future refactor removes required methods from one executor but not the other, test_executor_interface_parity fails immediately.

---

8. Conclusion

Issue #436 was caused by the launcher's task registry being a manually maintained subset of the evaluator's full benchmark set, creating silent divergence when new benchmarks were added to the evaluator but not the launcher. The fix expanded the launcher's task manifest to include missing tasks. A new structural regression test has been added to verify executor interface parity and task payload completeness; a full registry parity test remains an open gap with medium regression risk.

---

Auto-generated by the issues-rca skill — QA RCCA Analysis v2