Skip to content

Commit 12225bb

Browse files
authored
Updates from Adolfo (QE) re text (#556)
* Updates from Adolfo * small typo update * updated to address peer review feedback from Chris * removing extra space
1 parent 47733a7 commit 12225bb

File tree

2 files changed

+12
-5
lines changed

2 files changed

+12
-5
lines changed

lmeval-evaluation-job.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ After the `LMEvalJob` is created, the LM-Eval service runs the evaluation job.
1313

1414
[NOTE]
1515
--
16-
TrustyAI does not support non-tabular models. Deploying the TrustyAI custom resource (CR) in a namespace that contains non-tabular models can cause errors within the TrustyAI service.
16+
Other TrustyAI features (such as bias and drift metrics) do not support non-tabular models (including LLMs). Deploying the TrustyAIService custom resource (CR) in a namespace that contains non-tabular models (such as the namespace where an evaluation job is being executed) can cause errors within the TrustyAI service.
1717
--
1818

1919
.Sample LMEvalJob object
@@ -178,7 +178,7 @@ textsynth (TextSynth APIs): Check link:https://github.com/EleutherAI/lm-evaluati
178178
| `outputs.pvcManaged`
179179
| Creates an operator-managed PVC to store this job's results. The PVC is named `<job-name>-pvc` and is owned by the `LMEvalJob`. After the job finishes, the PVC is still be available, but it is deleted with the `LMEvalJob`. Supports the following fields:
180180

181-
* `size`: The PVC's size, compatible with standard PVC syntax (e.g. 5Gi)
181+
* `size`: The PVC's size, compatible with standard PVC syntax (for example, 5Gi)
182182
183183
| `outputs.pvcName`
184184
| Binds an existing PVC to a job by specifying its name. The PVC must be created separately and must already exist when creating the job.

lmeval-scenarios.adoc

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ ifdef::context[:parent-context: {context}]
88

99
The procedures below outline different example scenarios which you may find useful for your ML-Eval setup.
1010

11-
== Configuring the ML-eval environment
11+
== Configuring the LM-eval environment
1212

1313
If the `LMEvalJob` needs to access a model on HuggingFace with the access token, you can set up the `HF_TOKEN` as one of the environment variables for the `lm-eval` container.
1414

@@ -20,7 +20,7 @@ If the `LMEvalJob` needs to access a model on HuggingFace with the access token,
2020

2121
.Procedure
2222

23-
Enter the following code:
23+
To start an evaluation job for a `huggingface` model, apply the following YAML file:
2424

2525
[source]
2626
----
@@ -61,6 +61,8 @@ env:
6161

6262
== Custom Unitxt Card
6363

64+
You can also run evaluations using custom unitxt cards. To do this, include the custom unitxt card in JSON format within the LMEvalJob YAML.
65+
6466
.Prerequisites
6567
* You have logged in to Red Hat OpenShift AI.
6668

@@ -251,14 +253,19 @@ spec:
251253

252254
== Using an InferenceService
253255

254-
This example assumes that the vLLM model is already deployed in your cluster.
256+
To run an evaluation job on an InferenceService which is already deployed and running in your namespace, define your LMEvalJob CR, then apply this CR into the same namespace as your model.
257+
255258
.Prerequisites
256259
* You have logged in to Red Hat OpenShift AI.
257260

258261
* Your OpenShift cluster administrator has installed OpenShift AI and enabled the TrustyAI service for the data science project where the models are deployed.
259262

263+
* You have a namespace that contains an InferenceService with a vLLM model. This example assumes that the vLLM model is already deployed in your cluster.
264+
260265
.Procedure
266+
261267
* Define your LMEvalJob CR:
268+
262269
[source]
263270
----
264271
apiVersion: trustyai.opendatahub.io/v1alpha1

0 commit comments

Comments
 (0)