fix tests part 1 - also removes fewshot truncation in the task name because it's no longer used anywhere in the code logically

Author: clefourrier

README.md

@@ -90,7 +90,7 @@ Here’s a quick command to evaluate using the Accelerate backend:
 ```shell
 lighteval accelerate \
     "model_name=gpt2" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 ## 🙏 Acknowledgements

community_tasks/custom_task_classification_grammar_task.py

@@ -32,7 +32,7 @@
 Example usage:
     TGI endpoint evaluation:
     ```bash
-    uv run --active --extra litellm --extra tgi lighteval endpoint tgi examples/model_configs/tgi_model.yaml "custom|emotion_classification|0|0"
+    uv run --active --extra litellm --extra tgi lighteval endpoint tgi examples/model_configs/tgi_model.yaml "custom|emotion_classification|0"
     --custom-tasks examples/custom_tasks_templates/custom_task_classification_grammar_task.py
     --output-dir results
     --save-details
@@ -449,8 +449,8 @@ def get_emotion_classification_grammar() -> TextGenerationInputGrammarType:
 
     print("\nUsage Examples:")
     print(
-        f"  TGI: uv run lighteval endpoint tgi config/tgi/tgi.yaml 'custom|{task.name}|0|0' --custom-tasks {__file__} --output-dir results --override-batch-size 1 --use-chat-template --save-details --no-public-run --max-samples 10"
+        f"  TGI: uv run lighteval endpoint tgi config/tgi/tgi.yaml 'custom|{task.name}|0' --custom-tasks {__file__} --output-dir results --override-batch-size 1 --use-chat-template --save-details --no-public-run --max-samples 10"
     )
     print(
-        f"  Full: uv run lighteval endpoint tgi config/tgi/tgi.yaml 'custom|{task.name}|5|1' --custom-tasks {__file__} --output-dir results --override-batch-size 1 --use-chat-template --save-details --no-public-run"
+        f"  Full: uv run lighteval endpoint tgi config/tgi/tgi.yaml 'custom|{task.name}|5' --custom-tasks {__file__} --output-dir results --override-batch-size 1 --use-chat-template --save-details --no-public-run"
     )

docs/source/adding-a-custom-task.mdx

@@ -126,6 +126,6 @@ Once your file is created you can then run the evaluation with the following com
 ```bash
 lighteval accelerate \
     "model_name=HuggingFaceH4/zephyr-7b-beta" \
-    "community|{custom_task}|{fewshots}|{truncate_few_shot}" \
+    "community|{custom_task}|{fewshots}" \
     --custom-tasks {path_to_your_custom_task_file}
 ```

docs/source/evaluating-a-custom-model.mdx

@@ -56,7 +56,7 @@ You can evaluate your custom model using either the command line interface or th
 lighteval custom \
     "google-translate" \
     "examples/custom_models/google_translate_model.py" \
-    "lighteval|wmt20:fr-de|0|0" \
+    "lighteval|wmt20:fr-de|0" \
     --max-samples 10
 ```
 
@@ -91,7 +91,7 @@ model_config = CustomModelConfig(
 
 # Create and run the pipeline
 pipeline = Pipeline(
-    tasks="leaderboard|truthfulqa:mc|0|0",
+    tasks="leaderboard|truthfulqa:mc|0",
     pipeline_parameters=pipeline_params,
     evaluation_tracker=evaluation_tracker,
     model_config=model_config

docs/source/quicktour.mdx

@@ -27,7 +27,7 @@ To evaluate `GPT-2` on the Truthful QA benchmark with [🤗
 ```bash
 lighteval accelerate \
      "model_name=openai-community/gpt2" \
-     "leaderboard|truthfulqa:mc|0|0"
+     "leaderboard|truthfulqa:mc|0"
 ```
 
 Here, we first choose a backend (either `accelerate`, `nanotron`, `endpoint`, or `vllm`), and then specify the model and task(s) to run.
@@ -38,12 +38,9 @@ Valid key-value pairs correspond with the backend configuration, and are detaile
 The syntax for the task specification might be a bit hard to grasp at first. The format is as follows:
 
 ```txt
-{suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a truncation if context size is too small}
+{suite}|{task}|{num_few_shot}
 ```
 
-If the fourth value is set to 1, lighteval will check if the prompt (including the few-shot examples) is too long for the context size of the task or the model.
-If so, the number of few shot examples is automatically reduced.
-
 Tasks have a function applied at the sample level and one at the corpus level. For example,
 - an exact match can be applied per sample, then averaged over the corpus to give the final score
 - samples can be left untouched before applying Corpus BLEU at the corpus level
@@ -52,7 +49,7 @@ etc.
 If the task you are looking at has a sample level function (`sample_level_fn`) which can be parametrized, you can pass parameters in the CLI.
 For example
 ```txt
-{suite}|{task}@{parameter_name1}={value1},{parameter_name2}={value2},...|0|0
+{suite}|{task}@{parameter_name1}={value1}@{parameter_name2}={value2},...|0
 ```
 
 All officially supported tasks can be found at the [tasks_list](available-tasks) and in the
@@ -71,7 +68,7 @@ When specifying a path to file, it should start with `./`.
 lighteval accelerate \
      "model_name=openai-community/gpt2" \
      ./path/to/lighteval/examples/tasks/recommended_set.txt
-# or, e.g., "leaderboard|truthfulqa:mc|0|0|,leaderboard|gsm8k|3|1"
+# or, e.g., "leaderboard|truthfulqa:mc|0,leaderboard|gsm8k|3"
 ```
 
 ## Evaluate a model on one or more GPUs
@@ -90,7 +87,7 @@ You can then evaluate a model using data parallelism on 8 GPUs like follows:
 accelerate launch --multi_gpu --num_processes=8 -m \
     lighteval accelerate \
     "model_name=openai-community/gpt2" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 Here, `--override_batch_size` defines the batch size per device, so the effective
@@ -103,7 +100,7 @@ To evaluate a model using pipeline parallelism on 2 or more GPUs, run:
 ```bash
 lighteval accelerate \
     "model_name=openai-community/gpt2,model_parallel=True" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 This will automatically use accelerate to distribute the model across the GPUs.
@@ -134,7 +131,7 @@ think tokens.
 ```bash
 lighteval vllm \
     "model_name=mistralai/Magistral-Small-2507,dtype=float16,data_parallel_size=4" \
-    "lighteval|aime24|0|0" \
+    "lighteval|aime24|0" \
     --remove-reasoning-tags \
     --reasoning-tags="[('[THINK]','[/THINK]')]"
 ```

docs/source/saving-and-reading-results.mdx

@@ -203,12 +203,9 @@ The detail file contains the following columns:
         "hash_input_tokens": "29916e7afe5cb51d",
         "hash_cont_tokens": "37f91ce23ef6d435"
       },
-      "truncated": 2,
-      "non_truncated": 0,
       "padded": 0,
       "non_padded": 2,
       "effective_few_shots": 0.0,
-      "num_truncated_few_shots": 0
     }
   },
   "summary_general": {
@@ -218,11 +215,8 @@ The detail file contains the following columns:
       "hash_input_tokens": "ac933feb14f96d7b",
       "hash_cont_tokens": "9d03fb26f8da7277"
     },
-    "truncated": 2,
-    "non_truncated": 0,
     "padded": 0,
     "non_padded": 2,
-    "num_truncated_few_shots": 0
   }
 }
 ```

docs/source/use-inference-providers-as-backend.mdx

@@ -12,7 +12,7 @@ Lighteval allows to use Hugging Face's Inference Providers to evaluate llms on s
 ```bash
 lighteval endpoint inference-providers \
     "model_name=deepseek-ai/DeepSeek-R1,provider=hf-inference" \
-    "lighteval|gsm8k|0|0"
+    "lighteval|gsm8k|0"
 ```
 
 ## Using a config file
@@ -22,7 +22,7 @@ You can use config files to define the model and the provider to use.
 ```bash
 lighteval endpoint inference-providers \
     examples/model_configs/inference_providers.yaml \
-    "lighteval|gsm8k|0|0"
+    "lighteval|gsm8k|0"
 ```
 
 with the following config file:

docs/source/use-litellm-as-backend.mdx

@@ -11,7 +11,7 @@ Documentation for available APIs and compatible endpoints can be found [here](ht
 ```bash
 lighteval endpoint litellm \
     "provider=openai,model_name=gpt-3.5-turbo" \
-    "lighteval|gsm8k|0|0" \
+    "lighteval|gsm8k|0" \
 ```
 
 ## Using a config file

docs/source/use-sglang-as-backend.mdx

@@ -6,7 +6,7 @@ To use, simply change the `model_args` to reflect the arguments you want to pass
 ```bash
 lighteval sglang \
     "model_name=HuggingFaceH4/zephyr-7b-beta,dtype=float16" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 `sglang` is able to distribute the model across multiple GPUs using data
@@ -18,15 +18,15 @@ For example if you have 4 GPUs you can split it across using `tp_size`:
 ```bash
 lighteval sglang \
     "model_name=HuggingFaceH4/zephyr-7b-beta,dtype=float16,tp_size=4" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 Or, if your model fits on a single GPU, you can use `dp_size` to speed up the evaluation:
 
 ```bash
 lighteval sglang \
     "model_name=HuggingFaceH4/zephyr-7b-beta,dtype=float16,dp_size=4" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 ## Use a config file
@@ -37,7 +37,7 @@ An example of a config file is shown below and can be found at `examples/model_c
 ```bash
 lighteval sglang \
     "examples/model_configs/sglang_model_config.yaml" \
-    "leaderboard|truthfulqa:mc|0|0"
+    "leaderboard|truthfulqa:mc|0"
 ```
 
 > [!TIP]

docs/source/use-vllm-as-backend.mdx

@@ -10,7 +10,7 @@ To use, simply change the `model_args` to reflect the arguments you want to pass
 ```bash
 lighteval vllm \
     "model_name=HuggingFaceH4/zephyr-7b-beta" \
-    "extended|ifeval|0|0"
+    "extended|ifeval|0"
 ```
 
 `vllm` is able to distribute the model across multiple GPUs using data
@@ -22,15 +22,15 @@ For example if you have 4 GPUs you can split it across using `tensor_parallelism
 ```bash
 export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm \
     "model_name=HuggingFaceH4/zephyr-7b-beta,tensor_parallel_size=4" \
-    "extended|ifeval|0|0"
+    "extended|ifeval|0"
 ```
 
 Or, if your model fits on a single GPU, you can use `data_parallelism` to speed up the evaluation:
 
 ```bash
 export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm \
     "model_name=HuggingFaceH4/zephyr-7b-beta,data_parallel_size=4" \
-    "extended|ifeval|0|0"
+    "extended|ifeval|0"
 ```
 
 ## Use a config file
@@ -41,7 +41,7 @@ An example of a config file is shown below and can be found at `examples/model_c
 ```bash
 lighteval vllm \
     "examples/model_configs/vllm_model_config.yaml" \
-    "extended|ifeval|0|0"
+    "extended|ifeval|0"
 ```
 
 ```yaml