Skip to content

version-prompts.mdx

Latest commit

 

History

History
354 lines (254 loc) · 11.6 KB

version-prompts.mdx

File metadata and controls

354 lines (254 loc) · 11.6 KB
subtitle Create and manage different versions of your prompts
slug llm-evaluation/prompt-management/version-prompts

Overview

Prompt versioning allows you to optimize and test different versions of your prompts. Managing prompts on Confident AI allows you to:

  1. Collaborate and centralize where prompts are stored and edited, even for non-technical team members
  2. Pinpoint which version, or even combination of your prompt versions, performed best
  3. Optionally co-locate model settings, output type, and tools with prompts to version them as a single unit

There are a million places you can keep your prompts - on GitHub, CSV files, in memory in code, Google Sheets, Notion, or even written in a diary hidden under your table drawer. But only by keeping prompts on Confident AI can you fully leverage Confident AI's evaluation features.

Prompts are a type of hyperparameter on Confident AI. Others include things like models, embedders, top-K, and max tokens. When you run evals against prompts kept on Confident AI, we can tell you which version performs best, and later automatically optimize it for you. **Prompts vs Prompts + Model Config:** You can use Confident AI purely for prompt versioning — pull your prompts and use them with whatever model you configure in your code. Alternatively, if you want to manage prompts and model configurations together as a single versioned unit, you can attach model settings, output type, and tools to prompt versions.

Different Types of Prompts

There are two types of prompts you can create:

  • (Single) Text Prompt: Use this when you need a straightforward, one-off prompt for simple completions.
  • Prompt Message List: Use this when you need to define multiple messages with specific roles (system, user, assistant) in an OpenAI messages format. This format is ideal for few-shot prompting, where you can start with a system message that sets the context.
If you ever see a prompt being mentioned without any mention of "message" or "list", assume it is a single prompt we're talking about.

Understanding Prompt Versions

In Confident AI, each prompt is identified by a unique alias. This alias acts as a unique identifier and refers to a single, specific prompt. Different aliases refer to completely separate prompts.

Suppose you have a prompt with the alias `MyPrompt`. This prompt can have multiple versions, such as `0.0.1`, `0.0.2`, ..., up to `1.0.0`. 
flowchart TD
    MyPrompt["Alias: MyPrompt"]
    MyPrompt --> V001["Version 0.0.1"]
    MyPrompt --> V002["Version 0.0.2"]
    MyPrompt --> V003[" ... "]
    MyPrompt --> V100["Version 1.0.0"]
Loading

A prompt version represents a different variation of the same prompt. Each version is an iteration or improvement of the original prompt, serving the same purpose but with adjustments—similar to how software versions evolve over time.

Create a Prompt Version

You can create a prompt version in Project > Prompt Studio through two simple steps:

  1. Create a text or messages prompt
  2. Edit and commit an initial prompt version in the prompt editor

A prompt cannot be both a text and message prompt at the same time.

Don't forget to commit an initial version of your prompt after you're done editing it. Alternately, you can create a prompt version from code.

You can manage your prompts in any project by configuring a `CONFIDENT_API_KEY`.
  • For default usage, set CONFIDENT_API_KEY as an environment variable.
  • To target a specific project, pass a confident_api_key directly when creating the Prompt object.
from deepeval.prompt import Prompt, PromptMessage

prompt = Prompt(
  alias="YOUR-PROMPT-ALIAS",
  messages_template=[PromptMessage(role="...", content="...")],
  confident_api_key="confident_us...",
)
prompt.push()

When both are provided, the confident_api_key passed to Prompt always takes precedence over the environment variable.

```python from deepeval.prompt import Prompt from deepeval.prompt.api import PromptMessage

prompt = Prompt(alias="YOUR-PROMPT-ALIAS") prompt.push(messages=[PromptMessage(role="...", content="...")])

</Tab>

<Tab title="Text" language="python">
```python
from deepeval.prompt import Prompt

prompt = Prompt(alias="YOUR-PROMPT-ALIAS")
prompt.push(text="...")
For more advanced push options including model settings, output type, and tools, see [Automate Prompt Management](/llm-evaluation/prompt-management/automate-prompt-management).

Templating Options

Dynamic variables

You can include variables that can be interpolated dynamically in your LLM application later on. There are five interpolation types available:

Type Syntax Example
FSTRING {variable} Hello, {name}!
MUSTACHE {{variable}} Hello, {{name}}!
MUSTACHE_WITH_SPACE {{ variable }} Hello, {{ name }}!
DOLLAR_BRACKETS ${variable} Hello, ${name}!
JINJA {% ... %} {% if admin %}Hello!{% endif %}

Variable names must not contain spaces:

# ✅ Correct usage:
"Hi, my name is {name}."
"The temperature is {temperature} degrees."
"User input: {user_input}"

# ❌ Incorrect usage:
"Hi, my name is {variable name}." # Spaces in variable name

Conditional logic

Conditional logic can be added when using JINJA interpolation. JINJA supports jinja templates, which allows you to render more complex logic such as conditional if/else blocks:

{% if is_admin %}
Welcome back, mighty admin {{ name }}!
{% else %}
Hello {{ name }}, you have regular access.
{% endif %}

As well as for loops:

Shopping List:
{% for item in items %}
- {{ item }}
{% endfor %}

Jinja interpolated prompts are only available for Python users.

Model Configs

Beyond creating and editing prompt versions, you can also configure model settings, output type, and tools associated with your prompt. This allows you to:

Keeping model configs for prompt versions on Confident AI does no harm but it doesn't mean you have to use it - for both in code and on the platform in the Arena or when running experiments. However, if model configs confuses you, feel free to leave them out.

Model settings

You can configure the model provider, model name, and model parameters for each prompt version. This ensures that when you pull a prompt in code, you also get the exact model configuration needed to run it.

Field Description Example
Provider The LLM provider (e.g., OpenAI, Anthropic, Azure) openai
Model The specific model name gpt-4.1
Parameters Model-specific parameters like temperature, max tokens, etc. {"temperature": 0.7, "max_tokens": 1024}
For an OpenAI GPT-4.1 configuration with custom temperature and max tokens:
  • Provider: openai
  • Model: gpt-4.1
  • Parameters: 
    {
  "temperature": 0.7,
  "max_tokens": 1024
}

Output type

You can specify the expected output format for your prompt by selecting an output type:

  • Text Output: Standard text response (default)
  • JSON Output: Structured JSON response
  • Schema Output: Structured response conforming to a defined schema

When using Schema Output, you can define a custom schema that your LLM response should conform to. This is useful for ensuring structured, predictable outputs.

To configure a schema:

  1. Click on Schema Output in the output type dropdown
  2. Enter a Schema Name (required)
  3. Add Schema Fields with their property names and types (String, Number, Boolean, etc.)
  4. Click Save Schema

The schema will be previewed as a Pydantic BaseModel class, making it easy to visualize how your structured output will look.

Attach tools

You can attach tools to your prompt version for function calling capabilities. This allows your LLM to invoke external tools like web search, APIs, or custom functions.

To attach tools:

  1. Click on the Tools button in the prompt editor
  2. Search for available tools
  3. Select the tools you want to enable for this prompt version

Assign Prompt Labels

You can also label different versions of your prompt in the Version History page so no code changes are required to "deploy" a new version into a certain environment.