Skip to content

Commit d1d220c

Browse files
Add 'useful commands' section to LMEval docs (#50)
1 parent 29fb640 commit d1d220c

File tree

1 file changed

+17
-2
lines changed

1 file changed

+17
-2
lines changed

docs/modules/ROOT/pages/lm-eval-tutorial.adoc

+17-2
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ trustyai:
2121
----
2222
====
2323

24-
== Global settings for LM-Eval
24+
== Global settings for LM-Eval [[global_settings]]
2525

2626
There are some configurable global settings for LM-Eval services and they are stored in the TrustyAI's operator global `ConfigMap`, `trustyai-service-operator-config`, located in the same namespace as the operator. Here is a list of properties for LM-Eval:
2727

@@ -174,7 +174,7 @@ A list of paired name and value arguments for the model type. Each model type or
174174

175175

176176
|`taskList.taskNames`
177-
|Specify a list of tasks supported by lm-evaluation-harness.
177+
|Specify a list of tasks supported by lm-evaluation-harness. See the link:#list_tasks[useful commands section] of this page to get a list of built-in tasks.
178178

179179

180180
|`taskList.taskRecipes`
@@ -615,3 +615,18 @@ Set `suspend` to `false` and verify job's pod getting created and running:
615615
----
616616
oc patch lmevaljob evaljob-sample --patch '{"spec":{"suspend":false}}' --type merge
617617
----
618+
619+
== Useful Commands & References
620+
621+
=== List all available tasks [[list_tasks]]
622+
623+
As mentioned above, LMEvalJob supports running both Unitxt tasks (via recipes or custom JSON) **and** the built-in tasks that are available out-of-the-box with `lm-evaluation-harness`. To see a list of available built-in tasks, run the following command. `LMES-POD-IMAGE` is the same as listed link:#global_settings[here].
624+
625+
[source,shell]
626+
----
627+
oc exec <<LMES-POD-IMAGE>> -- bash -c "lm_eval --tasks list"
628+
----
629+
630+
It is recommended to pipe this output to a file as there are several thousand built-in tasks.
631+
632+
Note: the output of the task list command will include some Unitxt tasks that have been directly contributed to `lm-evaluation-harness`. You can run these tasks by specifying the task name in the `taskList` property of the job definition as you would with any built-in task.

0 commit comments

Comments
 (0)