Nemotron AIME prompt and math-oai.yaml file

[MLKoz2]

Hello, local_nemotron-3-super-120b-a12b.yaml and local_nvidia_nemotron_3_nano_30b_a3b.yaml requires file math-oai.yaml file for AIME eval, nevertheless I cannot find this file, so the run of these files does not work.

args: '++prompt_config="/nemo_run/code/eval_factory_prompts/math-oai.yaml"'

Edit:
After the analysis, it works, from nemotron 3 super:

  - name: nemo_skills.ns_aime2025
    nemo_evaluator_config:
      config:
        params:
          extra:
            num_repeats: 64
            args: '++prompt_config="/nemo_run/code/eval_factory_prompts/math-oai.yaml"'
  - name: nemo_skills.ns_aime2026
    nemo_evaluator_config:
      config:
        params:
          extra:
            num_repeats: 64
            args: ++prompt_config=/nemo_run/code/eval_factory_prompts/math-oai.yaml ++inference.tokens_to_generate=null

but there is the problem with it:

  - name: ns_aime2025
    nemo_evaluator_config:
      config:
        params:
          extra:
            num_repeats: 64
            args: '++prompt_config="/nemo_run/code/eval_factory_prompts/math-oai.yaml"'

[marta-sd]

Hi @MLKoz2 , what kind of problem do you hit with the last snippet? Please share your command and the error log

[marta-sd]

Hi @MLKoz2 , do you still face issues when trying to run the evaluation? Please let us know how we can help

[MLKoz2]

I already solved the problem, using Nemotron 3 Super eval. files is ok as it is recent model, probably files for Nem.3-Nano may be outdated, thanks for your interest.😊

I would like to ask; do you have any information whether we can expect SWE-Bench or Terminal-Bench in NeMo Evaluator?
SWE-Bench can be found in NeMo Skills, but in Evaluator it would be more useful.

[marta-sd]

Happy to hear that you're unblocked 🙂

When it comes to benchmarks: both are high priority for us, we're already working on Terminal-Bench and SWE-Bench is in our roadmap too, so stay tuned 🙂

[pruprakash]

QA RCCA Analysis

Date: 2026-04-20
Analyst: AI QA Agent (issues-rca skill)
Issue: #880 — Nemotron AIME prompt and math-oai.yaml file

---

1. Fix Reference

No dedicated fix PR was created. The issue was closed on 2026-04-07. The reporter self-resolved after further analysis: the math-oai.yaml prompt config file is present in the NeMo Skills repository at /nemo_run/code/eval_factory_prompts/math-oai.yaml inside the container. The YAML config references it via an absolute container path (++prompt_config="/nemo_run/code/eval_factory_prompts/math-oai.yaml"), which is correct — it is not a missing file but a path that only exists inside the mounted NeMo run container. Users need to ensure the NeMo Skills container path is properly mounted.

The launcher config files (local_nemotron-3-super-120b-a12b.yaml, local_nvidia_nemotron_3_nano_30b_a3b.yaml) were confirmed to work correctly once the container path is available.

---

2. Root Cause

The math-oai.yaml prompt config file exists inside the NeMo Skills container at /nemo_run/code/eval_factory_prompts/math-oai.yaml, not as a standalone downloadable file. Users running evaluations expected to find this file in the repo or install directory, but it is only accessible inside the specific container image. The launcher YAML configs correctly reference the absolute container path — the root cause was insufficient documentation explaining that this path is container-internal and requires the NeMo Skills container to be mounted correctly in the evaluation job setup.

---

3. Trigger Config

Trigger conditions (AND — all must be present):

• Using local_nemotron-3-super-120b-a12b.yaml or local_nvidia_nemotron_3_nano_30b_a3b.yaml launcher config
• AIME evaluation task (nemo_skills.ns_aime2025)
• NeMo Skills container not mounted or /nemo_run/code/eval_factory_prompts/ path not accessible
• ++prompt_config="/nemo_run/code/eval_factory_prompts/math-oai.yaml" arg in the config

Deterministic? Yes. If the container path is not mounted, every AIME run fails with a file-not-found error.

---

4. Nature of Bug

Primary classification: Documentation bug — missing documentation about container path requirements for Nemotron AIME evaluation

Impact scope: Users running Nemotron AIME benchmarks (ns_aime2025) via the launcher YAML configs without the NeMo Skills container properly set up. No data corruption or incorrect results — evaluation simply fails to start.

NOT affected: Other benchmarks that do not reference math-oai.yaml. Users with the correct container mount in place are unaffected.

---

5. Functional Test Coverage

Verdict: PARTIAL

Test	File	Key Config	What it covers
test_task_model_declares_required_fields	evaluator/testcases/rcca/launcher/test_launcher_config_completeness.py	No container needed	TaskModel declares container and endpoint_type fields
test_evaluation_model_declares_tasks_field	same	No container needed	EvaluationModel declares tasks field
test_task_model_minimal_valid_instance	same	No container needed	TaskModel accepts minimal valid kwargs without error

These tests verify the launcher config schema is complete. They do not verify the presence of math-oai.yaml or the container mount — which is the actual failure scenario from this issue.

---

6. Gaps and Limitations

Gap 1 — math-oai.yaml path not validated (out of scope — container-internal):
The file lives at /nemo_run/code/eval_factory_prompts/math-oai.yaml inside the NeMo Skills container. Testing for its presence requires the container to be running with the correct mounts. This is out of CI smoke-test scope. A proper test would mount the container and assert Path("/nemo_run/code/eval_factory_prompts/math-oai.yaml").exists().

Gap 2 — AIME config YAML not schema-validated:
The launcher YAML configs for Nemotron AIME (local_nemotron-3-super-120b-a12b.yaml) are not loaded and validated in the tests. A schema validation test that parses these YAML files and asserts all required fields (including prompt_config path format) would close this gap.

Overall gap assessment: Medium regression risk for the specific AIME path. The config completeness tests guard the launcher schema but not the container path dependency.

---

7. New Test Added

Field	Value
Test file	evaluator/testcases/rcca/launcher/test_launcher_config_completeness.py
Test functions	test_task_model_declares_required_fields, test_evaluation_model_declares_tasks_field, test_task_model_minimal_valid_instance
QA repo	nmfw_tests (local: evaluator/testcases/rcca/launcher/)
PR	Pending
What it validates	Launcher config schema declares container, endpoint_type, and tasks fields; minimal TaskModel instance can be constructed
How it would catch a regression	If container or endpoint_type is removed from TaskModel, test_task_model_declares_required_fields fails — preventing silent field omissions that caused issues like #880.

---

8. Conclusion

Issue #880 was a documentation gap: the math-oai.yaml prompt config file for Nemotron AIME evaluation exists inside the NeMo Skills container at a specific absolute path, but this was not documented, causing users to search for the file outside the container. The reporter self-resolved after discovering the file at its correct container path. A launcher config completeness test has been added to guard the schema fields that must be present for jobs like AIME to run; testing the container-internal file path remains out of scope for automated CI.

---

Auto-generated by the issues-rca skill — QA RCCA Analysis v2