-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Add tutorial for optimizer tracking #8277
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
TomeHirata
merged 4 commits into
stanfordnlp:main
from
TomeHirata:docs/optimizer_tracking
May 29, 2025
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,138 @@ | ||
# Tracking DSPy Optimizers with MLflow | ||
|
||
This tutorial demonstrates how to use MLflow to track and analyze your DSPy optimization process. MLflow's built-in integration for DSPy provides traceability and debuggability for your DSPy optimization experience. It allows you to understand the intermediate trials during the optimization, store the optimized program and its results, and provides observability into your program execution. | ||
|
||
Through the autologging capability, MLflow tracks the following information: | ||
|
||
* **Optimizer Parameters** | ||
* Number of few-shot examples | ||
* Number of candidates | ||
* Other configuration settings | ||
|
||
* **Program States** | ||
TomeHirata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* Initial instuctions and few-shot examples | ||
* Optimized instuctions and few-shot examples | ||
* Intermediate instuctions and few-shot examples during optimization | ||
|
||
* **Datasets** | ||
* Training data used | ||
* Evaluation data used | ||
|
||
* **Performance Progression** | ||
* Overall metric progression | ||
* Performance at each evaluation step | ||
|
||
* **Traces** | ||
* Program execution traces | ||
* Model responses | ||
* Intermediate prompts | ||
|
||
## Getting Started | ||
|
||
### 1. Install MLflow | ||
First, install MLflow (version 2.21.1 or later): | ||
|
||
```bash | ||
pip install mlflow>=2.21.1 | ||
``` | ||
|
||
### 2. Start MLflow Tracking Server | ||
|
||
Let's spin up the MLflow tracking server with the following command. This will start a local server at `http://127.0.0.1:5000/`: | ||
|
||
```bash | ||
TomeHirata marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# It is highly recommended to use SQL store when using MLflow tracing | ||
mlflow server --backend-store-uri sqlite:///mydb.sqlite | ||
``` | ||
|
||
### 3. Enable Autologging | ||
|
||
Configure MLflow to track your DSPy optimization: | ||
|
||
```python | ||
import mlflow | ||
import dspy | ||
|
||
# Enable autologging with all features | ||
mlflow.dspy.autolog( | ||
log_compiles=True, # Track optimization process | ||
log_evals=True, # Track evaluation results | ||
log_traces_from_compile=True # Track program traces during optimization | ||
) | ||
|
||
# Configure MLflow tracking | ||
mlflow.set_tracking_uri("http://localhost:5000") # Use local MLflow server | ||
mlflow.set_experiment("DSPy-Optimization") | ||
``` | ||
|
||
### 4. Optimizing Your Program | ||
|
||
Here's a complete example showing how to track the optimization of a math problem solver: | ||
|
||
```python | ||
import dspy | ||
from dspy.datasets.gsm8k import GSM8K, gsm8k_metric | ||
|
||
# Configure your language model | ||
lm = dspy.LM(model="openai/gpt-4o") | ||
dspy.configure(lm=lm) | ||
|
||
# Load dataset | ||
gsm8k = GSM8K() | ||
trainset, devset = gsm8k.train, gsm8k.dev | ||
|
||
# Define your program | ||
program = dspy.ChainOfThought("question -> answer") | ||
|
||
# Create and run optimizer with tracking | ||
teleprompter = dspy.teleprompt.MIPROv2( | ||
metric=gsm8k_metric, | ||
auto="light", | ||
) | ||
|
||
# The optimization process will be automatically tracked | ||
optimized_program = teleprompter.compile( | ||
program, | ||
trainset=trainset, | ||
) | ||
``` | ||
|
||
### 5. Viewing Results | ||
|
||
Once your optimization is complete, you can analyze the results through MLflow's UI. Let's walk through how to explore your optimization runs. | ||
|
||
#### Step 1: Access the MLflow UI | ||
Navigate to `http://localhost:5000` in your web browser to access the MLflow tracking server UI. | ||
|
||
#### Step 2: Understanding the Experiment Structure | ||
When you open the experiment page, you'll see a hierarchical view of your optimization process. The parent run represents your overall optimization process, while the child runs show each intermediate version of your program that was created during optimization. | ||
|
||
 | ||
|
||
#### Step 3: Analyzing the Parent Run | ||
Clicking on the parent run reveals the big picture of your optimization process. You'll find detailed information about your optimizer's configuration parameters and how your evaluation metrics progressed over time. The parent run also stores your final optimized program, including the instructions, signature definitions, and few-shot examples that were used. Additionally, you can review the training data that was used during the optimization process. | ||
|
||
 | ||
|
||
#### Step 4: Examining Child Runs | ||
Each child run provides a detailed snapshot of a specific optimization attempt. When you select a child run from the experiment page, you can explore several aspects of that particular intermediate program. | ||
On the run parameter tab or artifact tab, you can review the instructions and few-shot examples used for the intermediate program. | ||
One of the most powerful features is the Traces tab, which provides a step-by-step view of your program's execution. Here you can understand exactly how your DSPy program processes inputs and generates outputs. | ||
|
||
 | ||
|
||
### 6. Loading Models for Inference | ||
You can load the optimized program directly from the MLflow tracking server for inference: | ||
|
||
```python | ||
model_path = mlflow.artifacts.download_artifacts("mlflow-artifacts:/path/to/best_model.json") | ||
program.load(model_path) | ||
``` | ||
|
||
## Troubleshooting | ||
|
||
- If traces aren't appearing, ensure `log_traces_from_compile=True` | ||
- For large datasets, consider setting `log_traces_from_compile=False` to avoid memory issues | ||
- Use `mlflow.get_run(run_id)` to programmatically access MLflow run data | ||
|
||
For more features, explore the [MLflow Documentation](https://mlflow.org/docs/latest/llms/dspy). |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.