Support logprob benchmarks for automodel evaluation

[athitten]

Initial support in 25.09 just added generation benchmarks, extend the support for logprob benchmarks.

[athitten]

Will be prioritized after #269

[athitten]

This is being actively worked on. HF's model.generate does not natively support returning logprobs. It only returns scores which needs to be post processed to get logprobs and top_logprobs. Added the support for this, however the score is slightly off with the baseline. Investigating this issue.

Next will check for logprobs and top_logprobs for HF deployment via vllm and add missing support.

[athitten]

PR for enabling logprob benchmarks for HF in-fw (wo vllm): NVIDIA-NeMo/Export-Deploy#452.

For eval via vllm path, code is ready in this branch: https://github.com/NVIDIA-NeMo/Export-Deploy/tree/athitten/automodel_vllm_logprobs. Depends on NVIDIA-NeMo/Export-Deploy#452 to be merged.

[athitten]

PR for vllm path merged: NVIDIA-NeMo/Export-Deploy#479

[athitten]

Docs need to be added

[athitten]

Docs PR: #380

[athitten]

QA please use the docs here: https://github.com/NVIDIA-NeMo/Evaluator/blob/athitten/hf_logprob_docs/docs/nemo-fw/evaluation-doc.md with latest rc container

[athitten]

Incorrect link above, docs are here: https://github.com/NVIDIA-NeMo/Evaluator/blob/main/docs/nemo-fw/evaluation-hf.md