Prompt Refinement

Iterative prompt engineering, solved.

Prompt Refinement is an automated framework that evolves your LLM prompts using an iterative optimization loop. It improves prompt quality by minimizing cost and maximizing accuracy against your ground-truth dataset, and it now works across providers instead of being tied to a single SDK.

Instead of guessing what works, let Prompt Refinement treat your prompt as a hyperparameter to optimize.

Why Prompt Refinement?

🔄 Iterative Evolution: Uses a generator model to improve prompts based on the current bottleneck.
🎯 Multi-Objective Scoring: Balances Accuracy, Novelty, and Cost.
🌐 Provider Agnostic Core: Bring your own client or use LiteLLM to target OpenAI, Anthropic, Groq, and other supported backends.
📊 Traceable History: Tracks prompt versions, scores, and token usage.
🖥️ Streamlit UI: Includes an interactive web interface with live progress tracking.

Quick Start

1. Install

git clone https://github.com/yourusername/prompt-refinement.git
cd prompt-refinement
uv sync

2. Configure a Provider

The included CLI and UI use litellm, so you can point them at many providers with the same interface.

Examples:

export OPENAI_API_KEY=your_key_here
export PROMPT_REFINEMENT_EXECUTOR_MODEL=openai/gpt-4.1-mini
export PROMPT_REFINEMENT_GENERATOR_MODEL=openai/gpt-4.1

export ANTHROPIC_API_KEY=your_key_here
export PROMPT_REFINEMENT_EXECUTOR_MODEL=anthropic/claude-3-7-sonnet-latest
export PROMPT_REFINEMENT_GENERATOR_MODEL=anthropic/claude-3-7-sonnet-latest

You can also use LLM_API_KEY and LLM_API_BASE for custom OpenAI-compatible endpoints.

3. Run the Demo

Option A: Command Line

uv run python prompt_refinement/cli/main.py

Option B: Streamlit UI

uv run streamlit run prompt_refinement/ui/app.py

The UI lets you provide:

API key and optional base URL
Executor and generator model names
Optimizer weights and iteration settings
Dataset configuration

Architecture

graph LR
    A[Initial Prompt] --> B(Executor)
    B --> C{Evaluator}
    C -->|Score & Bottleneck| D[Generator]
    D -->|New Variations| B

Module	Component	Role
`Executor`	`PromptExecutor`	Runs the current prompt against your evaluation set.
`Evaluator`	`Evaluator`	Scores outputs on Accuracy, Novelty, and Cost.
`Generator`	`PromptGenerator`	Proposes improved prompt variations.
`Optimizer`	`PromptOptimizer`	Manages the full optimization loop.

Programmatic Usage

Option 1: Use LiteLLM

from datasets import load_dataset
from prompt_refinement import (
    EvaluatorConfig,
    OptimizerConfig,
    PromptOptimizer,
    make_litellm_completion_fn,
)

dataset = load_dataset("your_dataset", split="train[:50]")
completion_fn = make_litellm_completion_fn()

config = OptimizerConfig(
    executor_model_name="openai/gpt-4.1-mini",
    executor_completion_fn=completion_fn,
    generator_model_name="openai/gpt-4.1",
    generator_completion_fn=completion_fn,
    max_iter_numb=5,
    improvement_threshold=0.01,
    questions="question_column",
    answers="answer_column",
    num_variations=3,
    task_description="Extract entities from financial reports.",
)

optimizer = PromptOptimizer(config, EvaluatorConfig(), dataset)
result = optimizer.optimize("Extract the required entities.")

print(result.final_prompt)

Option 2: Bring Your Own Provider Adapter

If you already have a provider client, pass it directly. The framework can work with:

a client exposing client.chat.complete(...)
a client exposing client.chat.completions.create(...)
a custom callable passed through executor_completion_fn / generator_completion_fn

That means the core optimization loop is no longer coupled to one provider SDK.

Configuration

`EvaluatorConfig`

Parameter	Default	Description
`weights`	`{"accuracy": 0.7, "novelty": 0.2, "cost": 0.1}`	Relative importance of each scoring metric.
`baseline_tokens`	`500`	Reference token budget for cost scoring.
`model`	`"model-name"`	Optional model label used for evaluation context.

`OptimizerConfig`

Parameter	Description
`executor_model_name`	Model used to execute prompts against the dataset.
`generator_model_name`	Model used to generate prompt variations.
`executor_client` / `generator_client`	Optional raw provider clients.
`executor_completion_fn` / `generator_completion_fn`	Optional custom completion callbacks.
`executor_response_text_fn` / `generator_response_text_fn`	Optional response text extractors for custom payloads.
`executor_token_count_fn`	Optional token extractor for custom payloads.
`max_iter_numb`	Maximum number of optimization iterations.
`improvement_threshold`	Minimum score improvement required to continue.
`num_variations`	Number of prompt variants generated per round.
`task_description`	Natural-language task description used by the generator.

Project Structure

prompt-refinement/
├── assets/
├── notebooks/
├── prompt_refinement/
│   ├── core/
│   ├── ui/
│   ├── cli/
│   ├── providers.py
│   └── __init__.py
├── pyproject.toml
└── tests/

Contributing

Run the tests before opening a PR:

uv run pytest tests/

FilesExpand file tree

README.md

Latest commit

History