Update for NeMo Evaluator 25.03

[shengnuo]

• add evaluation images at .spec.evaluationImages to NeMo Evaluator
• add NeMo entity store endpoint at .spec.entitystore to NeMo Evaluator
• rename DATA_STORE_HOST environment variable to DATA_STORE_URL

Tests

1. Create an evaluation target that points to the llama-3.1-8b-instruct NIMService

res = requests.post(
    url="http://nemoevaluator-sample.nemo.svc.cluster.local:7331/v1/evaluation/targets",
    json={
        "type": "model",
        "name": "foo",
        "namespace": "my-organization",
        "model": {
            "api_endpoint": {
                "url": "http://meta-llama3-8b-instruct.nemo.svc.cluster.local:8000/v1/completions",
                "model_id": "meta/llama-3.1-8b-instruct"
            }
        }
    }
)

2. Create a custom evaluation config

simple_eval_config = {
    "type": "custom",
    "params": {"parallelism": 8},
    "tasks": {
        "qa": {
            "type": "completion",
            "params": {
                "template": {
                    "prompt": "{{prompt}}",
                    "max_tokens": 20,
                    "temperature": 0.7,
                    "top_p": 0.9,
                },
            },
            "dataset": {"files_url": f"hf://datasets/{repo_id}/testing/testing.jsonl"},
            "metrics": {
                "bleu": {
                    "type": "bleu",
                    "params": {"references": ["{{ideal_response}}"]},
                },
                "string-check": {
                    "type": "string-check",
                    "params": {"check": ["{{ideal_response | trim}}", "equals", "{{output_text | trim}}"]},
                },
            },
        }
    },
}

3. Create an evaluation job for the above target and configuration

>> res = requests.post(
    f"http://nemoevaluator-sample.nemo.svc.cluster.local:7331/v1/evaluation/jobs",
    json={
        "config": simple_eval_config,
        "target": "my-organization/foo",
    },
)
>> res.json()["id"]
eval-NgZ8AAbQvfihVsvy3Pn1Pv

4. After few minutes, check if the evaluation job is completed

$ curl "http://${EVALUATOR_HOSTNAME}/v1/evaluation/jobs/eval-NgZ8AAbQvfihVsvy3Pn1Pv/status"
{"message":null,"task_status":{"qa":"completed"},"progress":100.0}

5. Download the evaluation result

$ curl -X 'GET' \
"http://${EVALUATOR_HOSTNAME}/v1/evaluation/jobs/eval-NgZ8AAbQvfihVsvy3Pn1Pv/download-results" \
-H 'accept: application/json' \
-o result.zip

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  474k  100  474k    0     0  3313k      0 --:--:-- --:--:-- --:--:-- 3321k

$ echo $?
0

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shivamerla]

Looks good with the initial review