last fixes

Author: NathanHB

docs/source/adding-a-new-metric.mdx

@@ -8,9 +8,9 @@ There are two types of metrics in Lighteval:
 
 #### Sample-Level Metrics
 - **Purpose**: Evaluate individual samples/predictions
-- **Input**: Takes a `Doc` (document/sample) and `ModelResponse` (model's prediction)
+- **Input**: Takes a `Doc` and `ModelResponse` (model's prediction)
 - **Output**: Returns a float or boolean value for that specific sample
-- **Example**: Checking if a model's answer matches the correct answer for one question
+- **Example**: Checking if a model's answer matches the correct answer for one sample
 
 #### Corpus-Level Metrics
 - **Purpose**: Compute final scores across the entire dataset/corpus
@@ -94,7 +94,7 @@ with [`~metrics.utils.metric_utils.SampleLevelMetric`]:
 my_custom_metric = SampleLevelMetric(
     metric_name="custom_accuracy",
     higher_is_better=True,
-    category="accuracy",
+    category=SamplingMethod.GENERATIVE,
     sample_level_fn=custom_metric,
     corpus_level_fn=agg_function,
 )
@@ -113,7 +113,7 @@ custom_metric = SampleLevelMetricGrouping(
         "response_length": False,  # Shorter responses might be better
         "confidence": True
     },
-    category="comprehensive",
+    category=SamplingMethod.GENERATIVE,
     sample_level_fn=custom_metric,
     corpus_level_fn={
         "accuracy": np.mean,

docs/source/evaluating-a-custom-model.mdx

@@ -13,7 +13,6 @@ Create a Python file containing your custom model implementation. The model must
 Here's a basic example:
 
 ```python
-from typing import List
 from lighteval.models.abstract_model import LightevalModel
 from lighteval.models.model_output import ModelResponse
 from lighteval.tasks.requests import Doc
@@ -28,17 +27,17 @@ class MyCustomModel(LightevalModel):
         self._cache = SampleCache(config)
 
     @cached("predictions")  # Enable caching for better performance
-    def greedy_until(self, docs: List[Doc]) -> List[ModelResponse]:
+    def greedy_until(self, docs: list[Doc]) -> list[ModelResponse]:
         # Implement generation logic
         pass
 
     @cached("predictions")  # Enable caching for better performance
-    def loglikelihood(self, docs: List[Doc]) -> List[ModelResponse]:
+    def loglikelihood(self, docs: list[Doc]) -> list[ModelResponse]:
         # Implement loglikelihood computation
         pass
 
     @cached("predictions")  # Enable caching for better performance
-    def loglikelihood_rolling(self, docs: List[Doc]) -> List[ModelResponse]:
+    def loglikelihood_rolling(self, docs: list[Doc]) -> list[ModelResponse]:
         # Implement rolling loglikelihood computation
         pass
 ```
@@ -113,15 +112,15 @@ Your custom model must implement these core methods:
 For generating text until a stop sequence or max tokens is reached. This is used for generative evaluations.
 
 ```python
-def greedy_until(self, docs: List[Doc]) -> List[ModelResponse]:
+def greedy_until(self, docs: list[Doc]) -> list[ModelResponse]:
     """
     Generate text until stop sequence or max tokens.
 
     Args:
-        docs: List of documents containing prompts and generation parameters
+        docs: list of documents containing prompts and generation parameters
 
     Returns:
-        List of model responses with generated text
+        list of model responses with generated text
     """
     pass
 ```
@@ -130,15 +129,15 @@ def greedy_until(self, docs: List[Doc]) -> List[ModelResponse]:
 For computing log probabilities of specific continuations. This is used for multiple choice logprob evaluations.
 
 ```python
-def loglikelihood(self, docs: List[Doc]) -> List[ModelResponse]:
+def loglikelihood(self, docs: list[Doc]) -> list[ModelResponse]:
     """
     Compute log probabilities of continuations.
 
     Args:
-        docs: List of documents containing context and continuation pairs
+        docs: list of documents containing context and continuation pairs
 
     Returns:
-        List of model responses with log probabilities
+        list of model responses with log probabilities
     """
     pass
 ```
@@ -147,15 +146,15 @@ def loglikelihood(self, docs: List[Doc]) -> List[ModelResponse]:
 For computing rolling log probabilities of sequences. This is used for perplexity metrics.
 
 ```python
-def loglikelihood_rolling(self, docs: List[Doc]) -> List[ModelResponse]:
+def loglikelihood_rolling(self, docs: list[Doc]) -> list[ModelResponse]:
     """
     Compute rolling log probabilities of sequences.
 
     Args:
-        docs: List of documents containing text sequences
+        docs: list of documents containing text sequences
 
     Returns:
-        List of model responses with rolling log probabilities
+        list of model responses with rolling log probabilities
     """
     pass
 ```
@@ -184,7 +183,7 @@ def __init__(self, config):
 Add cache decorators to your prediction methods:
 ```python
 @cached("predictions")
-def greedy_until(self, docs: List[Doc]) -> List[ModelResponse]:
+def greedy_until(self, docs: list[Doc]) -> list[ModelResponse]:
     # Your implementation...
 ```
 

docs/source/use-huggingface-inference-endpoints-or-tgi-as-backend.mdx

@@ -98,15 +98,15 @@ model_parameters:
 ```bash
 lighteval endpoint inference-endpoint \
     "configs/endpoint_model.yaml" \
-    "lighteval|gsm8k|0|0"
+    "lighteval|gsm8k|0"
 ```
 
 ### Using an Existing TGI Server
 
 ```bash
 lighteval endpoint tgi \
     "configs/tgi_server.yaml" \
-    "lighteval|gsm8k|0|0"
+    "lighteval|gsm8k|0"
 ```
 
 ### Reusing an Existing Endpoint

docs/source/use-inference-providers-as-backend.mdx

@@ -66,28 +66,4 @@ model_parameters:
   org_to_bill: "my-organization"
 ```
 
-## Troubleshooting
-
-### Common Issues
-
-1. **API Key Errors**: Ensure your `HF_TOKEN` is set correctly
-2. **Provider Not Available**: Check that the provider supports your model
-3. **Billing Issues**: Verify you have sufficient credits or organization permissions
-4. **Timeout Errors**: Increase the timeout parameter for large models
-
-### Performance Tips
-
-- Use appropriate `parallel_calls_count` based on your needs
-- Monitor your usage to avoid unexpected costs
-- Consider using organization billing for team projects
-- Use appropriate timeout values for your model size
-
-### Error Handling
-
-Common error messages and solutions:
-- **"Provider not found"**: Check the provider name is correct
-- **"Model not available"**: Verify the model is supported by the provider
-- **"Insufficient credits"**: Add credits to your account or organization
-- **"Timeout"**: Increase the timeout parameter or use a smaller model
-
 For more detailed error handling and provider-specific information, refer to the [Hugging Face Inference Providers documentation](https://huggingface.co/docs/inference-endpoints/guides/inference_providers).

docs/source/use-litellm-as-backend.mdx

@@ -69,28 +69,4 @@ model_parameters:
     api_key: ""
 ```
 
-
-## Troubleshooting
-
-### Common Issues
-
-1. **API Key Errors**: Ensure your API key is correct and has sufficient permissions
-2. **Connection Errors**: Check that the base URL is accessible and the service is running
-3. **Model Not Found**: Verify the model name is correct for the specified provider
-4. **Rate Limiting**: Implement retry logic or reduce request frequency
-
-### Performance Tips
-
-- Implement retry logic for transient failures
-- Consider using batch processing for multiple evaluations
-- Monitor API usage and costs for cloud providers
-
-### Error Handling
-
-LiteLLM provides detailed error messages for common issues:
-- Invalid API keys
-- Model not available
-- Rate limiting
-- Network connectivity issues
-
 For more detailed error handling and debugging, refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/).

docs/source/use-sglang-as-backend.mdx

@@ -110,41 +110,3 @@ model_parameters:
 - `repetition_penalty`: Penalty for repeating tokens
 - `presence_penalty`: Penalty for token presence
 - `frequency_penalty`: Penalty for token frequency
-
-## Advanced Features
-
-### Backend Selection
-SGLang supports different backends for sampling and attention:
-
-```yaml
-model_parameters:
-    sampling_backend: "vllm"  # or "hf", "hf_legacy"
-    attention_backend: "flash"  # or "xformers", "native"
-```
-
-### Optimizations
-- `pairwise_tokenization`: Enable pairwise tokenization for multiple choice tasks
-- `add_special_tokens`: Add special tokens during tokenization
-- `skip_tokenizer_init`: Skip tokenizer initialization for faster startup
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Out of Memory Errors**: Reduce `mem_fraction_static` or `chunked_prefill_size`
-2. **Model Loading Errors**: Check that the model name and configuration are correct
-3. **Performance Issues**: Ensure appropriate backend selection for your hardware
-
-### Performance Tips
-
-- Use tensor parallelism for large models that don't fit on a single GPU
-- Use data parallelism for smaller models to increase throughput
-- Adjust `mem_fraction_static` based on your GPU memory capacity
-- Consider using `float16` or `bfloat16` for faster inference
-- Use appropriate sampling and attention backends for your hardware
-
-### Memory Optimization
-
-- Reduce `mem_fraction_static` if you encounter OOM errors
-- Adjust `chunked_prefill_size` based on your memory constraints
-- Use `skip_tokenizer_init=True` for faster startup if tokenizer is not needed

docs/source/use-vllm-as-backend.mdx

@@ -112,10 +112,3 @@ model_parameters:
 1. **Out of Memory Errors**: Reduce `gpu_memory_utilization` or `max_model_length`
 2. **Worker Process Issues**: Ensure `VLLM_WORKER_MULTIPROC_METHOD=spawn` is set for multi-GPU setups
 3. **Model Loading Errors**: Check that the model name and revision are correct
-
-### Performance Tips
-
-- Use tensor parallelism for large models that don't fit on a single GPU
-- Use data parallelism for smaller models to increase throughput
-- Adjust `gpu_memory_utilization` based on your GPU memory capacity
-- Consider using `bfloat16` or `float16` for faster inference

docs/source/using-the-python-api.mdx

@@ -41,7 +41,7 @@ def main():
         dtype="float16",
     )
 
-    task = "helm|mmlu|5"
+    task = "lighteval|gsm8k|5"
 
     pipeline = Pipeline(
         tasks=task,
@@ -78,7 +78,7 @@ You can evaluate on multiple tasks by providing a comma-separated list or a file
 
 ```python
 # Multiple tasks as comma-separated string
-tasks = "helm|mmlu|5|1,helm|truthfulqa:mc|0"
+tasks = "lighteval|aime24|0,lighteval|aime25|0"
 
 # Or load from a file
 tasks = "./path/to/tasks.txt"