update from suggestions

Author: NathanHB

docs/source/adding-a-custom-task.mdx

@@ -43,7 +43,7 @@ dataset to a document to be used for evaluation.
 from lighteval.tasks.requests import Doc
 
 # Define as many as you need for your different tasks
-def prompt_fn(line, task_name: str = None):
+def prompt_fn(line: dict, task_name: str):
     """Defines how to go from a dataset line to a doc object.
     Follow examples in src/lighteval/tasks/default_prompts.py, or get more info
     about what this function should do in the README.
@@ -99,7 +99,7 @@ task = LightevalTaskConfig(
     name="myothertask",
     prompt_function=prompt_fn,  # Must be defined in the file or imported
     suite=["community"],
-    hf_repo="your_dataset_name",
+    hf_repo="your_dataset_repo_on_hf",
     hf_subset="default",
     hf_avail_splits=["train", "test"],
     evaluation_splits=["test"],
@@ -157,6 +157,11 @@ TASKS_TABLE = SUBSET_TASKS
 # TASKS_TABLE = [task]
 ```
 
+### Step 6: Creating a requirement file
+
+If your task has requirements, you need to create a `requirement.txt` file with
+only the required dependencies so that anyone can run your task.
+
 ## Running Your Custom Task
 
 Once your file is created, you can run the evaluation with the following command:

docs/source/adding-a-new-metric.mdx

@@ -1,9 +1,25 @@
 # Adding a New Metric
 
-Lighteval provides a flexible system for creating custom metrics to evaluate model performance. This guide explains how to create and integrate new metrics into the evaluation framework.
-
 ## Before You Start
 
+### Two different types of metrics
+
+There are two types of metrics in Lighteval:
+
+#### Sample-Level Metrics
+- **Purpose**: Evaluate individual samples/predictions
+- **Input**: Takes a `Doc` (document/sample) and `ModelResponse` (model's prediction)
+- **Output**: Returns a float or boolean value for that specific sample
+- **Example**: Checking if a model's answer matches the correct answer for one question
+
+#### Corpus-Level Metrics
+- **Purpose**: Compute final scores across the entire dataset/corpus
+- **Input**: Takes the results from all sample-level evaluations
+- **Output**: Returns a single score representing overall performance
+- **Examples**:
+  - Simple aggregation: Calculating average accuracy across all test samples
+  - Complex metrics: BLEU score where sample-level metric prepares data (tokenization, etc.) and corpus-level metric computes the actual BLEU score
+
 ### Check Existing Metrics
 
 First, check if you can use one of the parameterized functions in

docs/source/available-tasks.mdx

@@ -1,7 +1,5 @@
 # Available Tasks
 
-Lighteval provides a comprehensive collection of evaluation tasks covering various aspects of language model capabilities. You can explore and use these tasks to evaluate your models across different domains and skills.
-
 ## Discovering Available Tasks
 
 ### List All Tasks

docs/source/contributing-to-multilingual-evaluations.mdx

@@ -1,6 +1,6 @@
 # Contributing to Multilingual Evaluations
 
-Lighteval supports multilingual evaluations through a comprehensive system of translation literals and language-adapted templates. This guide explains how to contribute translations and create new multilingual tasks.
+Lighteval supports multilingual evaluations through a comprehensive system of translation literals and language-adapted templates.
 
 ## Contributing Translation Literals
 
@@ -53,14 +53,6 @@ Language.ENGLISH: TranslationLiterals(
 )
 ```
 
-### Translation Guidelines
-
-- **Accuracy**: Ensure translations are accurate and contextually appropriate
-- **Consistency**: Use consistent terminology across all literals
-- **Cultural Sensitivity**: Consider cultural differences in language usage
-- **Punctuation**: Only adjust punctuation if your language uses different conventions
-- **Indices**: Update the indices list if your language uses different alphabetical characters
-
 ## Contributing New Multilingual Tasks
 
 ### Prerequisites
@@ -81,12 +73,59 @@ For multilingual evaluations, the `prompt_function` should be implemented using
 
 #### Template Types
 Available template types include:
-- **XNLI**: Natural language inference tasks
-- **COPA**: Causal reasoning tasks
-- **Multiple Choice**: Standard multiple choice questions
-- **Question Answering**: Open-ended question answering
+- **XNLI**: Natural language inference tasks - [`get_nli_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/nli.py#L162)
+- **COPA**: Causal reasoning tasks - [`get_copa_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/copa.py#L76)
+- **Multiple Choice**: Standard multiple choice questions - [`get_mcq_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/multichoice.py#L81)
+- **Question Answering**: Open-ended question answering - [`get_qa_prompt_function`](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/templates/qa.py#L46)
 - **Custom**: Specialized task templates
 
+#### Formulation Types
+
+##### Multiple Choice Formulation (MCF)
+Used for standard multiple choice questions where the model selects from lettered options:
+```python
+MCFFormulation()
+```
+
+**Example output:**
+```
+Question: What is the capital of France?
+A. London
+B. Paris
+C. Berlin
+D. Rome
+Answer: | A/B/C/D
+```
+
+##### Classification Formulation (CF)
+Used for classification tasks where the model generates the answer directly:
+```python
+CFFormulation()
+```
+
+**Example output:**
+```
+Question: What is the capital of France?
+Answer: | Paris
+```
+
+##### Hybrid Formulation
+Used for tasks that present choices but expect the full answer text:
+```python
+HybridFormulation()
+```
+
+**Example output:**
+```
+Question: What is the capital of France?
+A. London
+B. Paris
+C. Berlin
+D. Rome
+Answer: | Paris
+```
+
+
 ### Creating Your Multilingual Task
 
 #### Step 1: Create the Task File
@@ -158,102 +197,15 @@ Follow the custom task guide to test if your task is correctly implemented.
 > [!TIP]
 > All [`~tasks.lighteval_task.LightevalTaskConfig`] parameters are strongly typed, including the inputs to the template function. Make sure to take advantage of your IDE's functionality to make it easier to correctly fill these parameters.
 
-### Formulation Types
-
-#### Multiple Choice Formulation (MCF)
-Used for standard multiple choice questions:
-```python
-MCFFormulation()
-```
-
-#### Classification Formulation (CF)
-Used for classification tasks:
-```python
-CFFormulation()
-```
-
-#### Hybrid Formulation
-Used for tasks that combine multiple approaches:
-```python
-HybridFormulation()
-```
-
-### Language Support
-
-Currently supported languages include:
-- **English** (ENGLISH)
-- **Spanish** (SPANISH)
-- **French** (FRENCH)
-- **German** (GERMAN)
-- **Italian** (ITALIAN)
-- **Portuguese** (PORTUGUESE)
-- **Russian** (RUSSIAN)
-- **Chinese** (CHINESE)
-- **Japanese** (JAPANESE)
-- **Korean** (KOREAN)
-- **Arabic** (ARABIC)
-- **Hindi** (HINDI)
-- **Turkish** (TURKISH)
-- **Dutch** (DUTCH)
-- **Polish** (POLISH)
-- **Swedish** (SWEDISH)
-- **Norwegian** (NORWEGIAN)
-- **Danish** (DANISH)
-- **Finnish** (FINNISH)
-
-### Best Practices
-
-#### Translation Literals
-- **Test your translations** with native speakers
-- **Consider regional variations** in language usage
-- **Maintain consistency** with existing translations
-- **Document any special considerations** in your PR
-
-#### Multilingual Tasks
-- **Choose appropriate templates** for your task type
-- **Test across multiple languages** to ensure consistency
-- **Consider cultural differences** in task interpretation
-- **Validate dataset quality** for each language
-- **Use appropriate metrics** for your task formulation
-
-#### Code Quality
-- **Follow existing code style** and conventions
-- **Add comprehensive documentation** for your task
-- **Include example usage** in your PR description
-- **Test thoroughly** before submitting
-
-### Testing Your Contribution
-
-#### Local Testing
-```bash
-# Test your task locally
-lighteval accelerate \
-    "model_name=openai-community/gpt2" \
-    "community|your_task_name|0|0" \
-    --max-samples 10
-```
-
-#### Validation Checklist
+### Validation Checklist
 - [ ] Translation literals are accurate and complete
 - [ ] Task works correctly across all target languages
 - [ ] Metrics are appropriate for the task type
 - [ ] Documentation is clear and comprehensive
 - [ ] Code follows project conventions
 
-### Submitting Your Contribution
-
-1. **Create a feature branch** for your changes
-2. **Add comprehensive tests** for your task
-3. **Update documentation** as needed
-4. **Open a pull request** with a clear description
-5. **Respond to review feedback** promptly
-
-Once everything is good, open a PR, and we'll be happy to review it!
-
 ### Getting Help
 
 - **GitHub Issues**: Report bugs or ask questions
 - **Discussions**: Join community discussions
 - **Documentation**: Review existing guides and examples
-
-For more detailed information about multilingual evaluations, see the [Multilingual Tasks Reference](package_reference/tasks).

docs/source/index.mdx

@@ -1,35 +1,36 @@
 # Lighteval
 
-🤗 Lighteval is your all-in-one toolkit for evaluating Large Language Models (LLMs) across multiple backends—whether it's
-[Transformers](https://github.com/huggingface/transformers),
-[Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference),
-[Inference Providers](https://huggingface.co/docs/huggingface_hub/en/guides/inference),
-[VLLM](https://github.com/vllm-project/vllm), or
-[Nanotron](https://github.com/huggingface/nanotron)—with
-ease. Dive deep into your model's performance by saving and exploring detailed,
-sample-by-sample results to debug and see how your models stack up.
+🤗 Lighteval is your all-in-one toolkit for evaluating Large Language Models
+(LLMs) across multiple backends with ease. Dive deep into your model's
+performance by saving and exploring detailed, sample-by-sample results to debug
+and see how your models stack up.
 
 ## Key Features
 
 ### 🚀 **Multi-Backend Support**
 Evaluate your models using the most popular and efficient inference backends:
-- **Transformers**: Standard Hugging Face models with full customization
-- **VLLM**: High-performance inference with optimized memory usage
-- **SGLang**: Fast and efficient model serving
-- **TGI**: Production-ready text generation inference
-- **Inference Endpoints**: Cloud-based model deployment
-- **LiteLLM**: Unified interface for multiple LLM providers
-- **Nanotron**: Distributed training and evaluation
+- `transformers`: Evaluate models on CPU or one or more GPUs using [🤗
+  Accelerate](https://github.com/huggingface/transformers)
+- `nanotron`: Evaluate models in distributed settings using [⚡️
+  Nanotron](https://github.com/huggingface/nanotron)
+- `vllm`: Evaluate models on one or more GPUs using [🚀
+  VLLM](https://github.com/vllm-project/vllm)
+- `custom`: Evaluate custom models (can be anything)
+- `sglang`: Evaluate models using [SGLang](https://github.com/sgl-project/sglang) as backend
+- `inference-endpoint`: Evaluate models using Hugging Face's [Inference Endpoints API](https://huggingface.co/inference-endpoints/dedicated)
+- `tgi`: Evaluate models using [🔗 Text Generation Inference](https://huggingface.co/docs/text-generation-inference/en/index) running locally
+- `litellm`: Evaluate models on any compatible API using [LiteLLM](https://www.litellm.ai/)
+- `inference-providers`: Evaluate models using [HuggingFace's inference providers](https://huggingface.co/docs/inference-providers/en/index) as backend**: Distributed training and evaluation
 
 ### 📊 **Comprehensive Evaluation**
-- **Extensive Task Library**: 100+ pre-built evaluation tasks
+- **Extensive Task Library**: 1000s pre-built evaluation tasks
 - **Custom Task Creation**: Build your own evaluation tasks
 - **Flexible Metrics**: Support for custom metrics and scoring
 - **Detailed Analysis**: Sample-by-sample results for deep insights
 
 ### 🔧 **Easy Customization**
-Customization at your fingertips: create [new tasks](adding-a-custom-task) and
-[metrics](adding-a-new-metric) tailored to your needs, or browse all our existing tasks and metrics.
+Customization at your fingertips: create [new tasks](adding-a-custom-task),
+[metrics](adding-a-new-metric) or [model](evaluating-a-custom-model) tailored to your needs, or browse all our existing tasks and metrics.
 
 ### ☁️ **Seamless Integration**
 Seamlessly experiment, benchmark, and store your results on the Hugging Face Hub, S3, or locally.
@@ -49,11 +50,6 @@ pip install lighteval
 lighteval accelerate \
     "model_name=openai-community/gpt2" \
     "leaderboard|truthfulqa:mc|0|0"
-
-# Evaluate using VLLM for better performance
-lighteval vllm \
-    "model_name=HuggingFaceH4/zephyr-7b-beta,dtype=float16" \
-    "leaderboard|gsm8k|3|1"
 ```
 
 ### Save Results

docs/source/use-inference-providers-as-backend.mdx

@@ -47,29 +47,6 @@ Optionally, you can charge them to an organization by setting `org_to_bill="<you
 Hugging Face Inference Providers supports a wide range of LLM providers see the [Inference Providers documentation](https://huggingface.co/docs/inference-providers/en/index) for the complete list.
 
 
-## Key Parameters
-
-### Provider Configuration
-- `model_name`: The Hugging Face model ID to use
-- `provider`: The inference provider (together, fireworks, novita, etc.)
-- `timeout`: Request timeout in seconds (null for default)
-- `proxies`: Proxy configuration (null for default)
-- `parallel_calls_count`: Number of parallel API calls (default: 10)
-- `org_to_bill`: Organization name for billing (optional)
-
-### Generation Parameters
-- `temperature`: Controls randomness in generation (0.0 = deterministic, 1.0 = random)
-- `max_new_tokens`: Maximum number of tokens to generate
-- `top_k`: Top-k sampling parameter
-- `top_p`: Nucleus sampling parameter
-- `repetition_penalty`: Penalty for repeating tokens
-- `stop_tokens`: List of tokens that stop generation
-
-### Advanced Configuration
-- `trust_remote_code`: Whether to trust remote code from the model
-- `use_auth_token`: Custom authentication token
-- `device_map`: Device mapping for model loading
-
 ## Billing and Costs
 
 ### Personal Account Billing

docs/source/use-litellm-as-backend.mdx

@@ -42,12 +42,8 @@ model_parameters:
 LiteLLM supports a wide range of LLM providers:
 
 ### Cloud Providers
-- **OpenAI**: GPT models (GPT-3.5, GPT-4, etc.)
-- **Azure OpenAI**: Azure-hosted OpenAI models
-- **Google Vertex AI**: PaLM, Gemini models
-- **AWS Bedrock**: Claude, Llama, and other models
-- **Anthropic**: Claude models
-- **Together AI**: Various open-source models
+
+all cloud providers can be found in the [litellm documentation](https://docs.litellm.ai/docs/providers).
 
 ### Local/On-Premise
 - **VLLM**: Local VLLM servers

src/lighteval/logging/evaluation_tracker.py

@@ -94,7 +94,7 @@ class EvaluationTracker:
 
     The EvaluationTracker coordinates multiple specialized loggers to track different aspects of model evaluation:
 
-    - Details Logger (DetailsLogger): Records per-instance evaluation details and predictions
+    - Details Logger (DetailsLogger): Records per-sample evaluation details and predictions
     - Metrics Logger (MetricsLogger): Tracks aggregate evaluation metrics and scores
     - Versions Logger (VersionsLogger): Records task and dataset versions
     - General Config Logger (GeneralConfigLogger): Stores overall evaluation configuration