You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: lmeval-evaluation-job.adoc
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,7 @@ After the `LMEvalJob` is created, the LM-Eval service runs the evaluation job.
13
13
14
14
[NOTE]
15
15
--
16
-
TrustyAI does not support non-tabular models. Deploying the TrustyAI custom resource (CR) in a namespace that contains non-tabular models can cause errors within the TrustyAI service.
16
+
Other TrustyAI features (such as bias and drift metrics) do not support non-tabular models (including LLMs). Deploying the TrustyAIService custom resource (CR) in a namespace that contains non-tabular models (such as the namespace where an evaluation job is being executed) can cause errors within the TrustyAI service.
| Creates an operator-managed PVC to store this job's results. The PVC is named `<job-name>-pvc` and is owned by the `LMEvalJob`. After the job finishes, the PVC is still be available, but it is deleted with the `LMEvalJob`. Supports the following fields:
180
180
181
-
* `size`: The PVC's size, compatible with standard PVC syntax (e.g. 5Gi)
181
+
* `size`: The PVC's size, compatible with standard PVC syntax (for example, 5Gi)
182
182
183
183
| `outputs.pvcName`
184
184
| Binds an existing PVC to a job by specifying its name. The PVC must be created separately and must already exist when creating the job.
The procedures below outline different example scenarios which you may find useful for your ML-Eval setup.
10
10
11
-
== Configuring the ML-eval environment
11
+
== Configuring the LM-eval environment
12
12
13
13
If the `LMEvalJob` needs to access a model on HuggingFace with the access token, you can set up the `HF_TOKEN` as one of the environment variables for the `lm-eval` container.
14
14
@@ -20,7 +20,7 @@ If the `LMEvalJob` needs to access a model on HuggingFace with the access token,
20
20
21
21
.Procedure
22
22
23
-
Enter the following code:
23
+
To start an evaluation job for a `huggingface` model, apply the following YAML file:
24
24
25
25
[source]
26
26
----
@@ -61,6 +61,8 @@ env:
61
61
62
62
== Custom Unitxt Card
63
63
64
+
You can also run evaluations using custom unitxt cards. To do this, include the custom unitxt card in JSON format within the LMEvalJob YAML.
65
+
64
66
.Prerequisites
65
67
* You have logged in to Red Hat OpenShift AI.
66
68
@@ -251,14 +253,19 @@ spec:
251
253
252
254
== Using an InferenceService
253
255
254
-
This example assumes that the vLLM model is already deployed in your cluster.
256
+
To run an evaluation job on an InferenceService which is already deployed and running in your namespace, define your LMEvalJob CR, then apply this CR into the same namespace as your model.
257
+
255
258
.Prerequisites
256
259
* You have logged in to Red Hat OpenShift AI.
257
260
258
261
* Your OpenShift cluster administrator has installed OpenShift AI and enabled the TrustyAI service for the data science project where the models are deployed.
259
262
263
+
* You have a namespace that contains an InferenceService with a vLLM model. This example assumes that the vLLM model is already deployed in your cluster.
0 commit comments