Skip to content

README.md

Latest commit

 

History

History
335 lines (255 loc) · 9.77 KB

README.md

File metadata and controls

335 lines (255 loc) · 9.77 KB
Raw

prompt-vcs

PyPI version Python 3.10+ License: MIT

Git-native prompt management library for LLM applications

A lightweight, code-first Python library for managing LLM prompts using Git and the file system — no external database required.

中文文档

✨ Features

  • 🚀 Zero Configuration - Define prompts directly in code, no extra setup needed
  • 📦 Git Native - Version control through file system and Git
  • 📄 Single-File Mode - All prompts in one prompts.yaml (default, clean and simple)
  • 📂 Multi-File Mode - Separate files per prompt (for large projects)
  • 🔄 Lockfile Mechanism - Lock specific versions for production, use code strings in development
  • 🛠️ Auto Migration - One-click conversion of hardcoded prompts to managed format
  • 🧪 Testing Framework - Define and run test cases for prompts with YAML-based test suites
  • ✅ Output Validation - Validate prompt outputs with JSON schema, regex, length checks, and custom rules
  • 🔬 A/B Testing - Compare different prompt versions and analyze LLM output effectiveness
  • 🎯 Type Safe - Full type hints support

📦 Installation

pip install prompt-vcs

# Optional: statistically tested A/B winner detection for small samples
pip install "prompt-vcs[analysis]"

🚀 Quick Start

1. Initialize Project

# Single-file mode (default) - creates prompts.yaml
pvcs init

# Multi-file mode - creates prompts/ directory
pvcs init --split

2. Inline Mode

from prompt_vcs import p

# Uses code string by default, switches to locked version when specified
msg = p("user_greeting", "Hello {name}", name="Developer")

3. Decorator Mode

from prompt_vcs import prompt

@prompt(id="system_core", default_version="v1")
def get_system_prompt(role: str):
    """
    You are a helpful assistant playing the role of {role}.
    """
    pass

4. Extract Prompts to YAML

pvcs scaffold src/

5. Switch Versions

pvcs switch user_greeting v2

6. Auto-Migrate Existing Code

Automatically convert hardcoded prompt strings to p() calls:

# Preview changes
pvcs migrate src/ --dry-run

# Interactive migration (confirm each change)
pvcs migrate src/

# Apply all changes automatically
pvcs migrate src/ --yes

# Clean mode: extract prompts to YAML and remove from code
# - If prompts.yaml exists → writes to prompts.yaml (single-file mode)
# - Otherwise → creates prompts/{id}/v1.yaml (multi-file mode)
pvcs migrate src/ --clean -y

Supported Conversions:

# Before
prompt = f"Hello {user.name}, price: {price:.2f}"

# After (default mode) - keeps template in code
from prompt_vcs import p
prompt = p("demo_prompt", "Hello {user_name}, price: {price:.2f}", 
           user_name=user.name, price=price)

# After (--clean mode) - extracts template to YAML
from prompt_vcs import p
prompt = p("demo_prompt", user_name=user.name, price=price)
# Template is stored in prompts.yaml or prompts/demo_prompt/v1.yaml

Features:

  • ✅ F-string variable extraction
  • ✅ Format spec preservation (:.2f)
  • ✅ Attribute/dict access sanitization (user.name → user_name)
  • ✅ Automatic import statement insertion
  • ✅ Smart skipping of short strings and complex expressions
  • ✅ Clean mode: Extract to YAML, keep only ID in code
  • ✅ Auto-detects storage mode: single-file (prompts.yaml) or multi-file (prompts/)

📁 Project Structure

Single-File Mode (Default)

your-project/
├── .prompt_lock.json     # Version lock file
├── prompts.yaml          # All prompts in one file
└── src/
    └── your_code.py

prompts.yaml format:

user_greeting:
  description: "Greeting template"
  template: |
    Hello, {name}!

system_core:
  description: "System prompt"
  template: |
    You are a helpful assistant.

Multi-File Mode (--split)

your-project/
├── .prompt_lock.json     # Version lock file
├── prompts/              # Prompt YAML files
│   ├── user_greeting/
│   │   ├── v1.yaml
│   │   └── v2.yaml
│   └── system_core/
│       └── v1.yaml
└── src/
    └── your_code.py

🎯 Core Principles

  • No Database - File system is the database
  • Git Native - Version control relies on file naming conventions and Git commits
  • Code First - Developers define prompts in code first
  • Zero Latency Dev - Development mode uses code strings, production reads from Lockfile

🧪 Testing Framework

Define test cases in YAML and validate prompt outputs:

# tests/prompts_test.yaml
name: "Prompt Tests"
tests:
  - name: "greeting_test"
    prompt_id: "user_greeting"
    inputs:
      name: "Developer"
    expected_output: "Hello, Developer!"
    validation:
      - type: contains
        substring: "Hello"
      - type: length
        max_length: 100
from prompt_vcs.testing import PromptTestRunner, load_test_suite_from_yaml

# Load and run tests
suite = load_test_suite_from_yaml("tests/prompts_test.yaml")
runner = PromptTestRunner()
results = runner.run_suite(suite)

Validation Types:

  • json_schema - Validate JSON structure (requires pip install prompt-vcs[validation])
  • regex - Match patterns
  • length - Check min/max length
  • contains - Verify substring presence
  • custom - Custom validation functions

🔬 A/B Testing

Compare different prompt versions and analyze their effectiveness:

from prompt_vcs import ABTestManager, ABTestConfig, ABTestVariant

# Create an experiment
manager = ABTestManager.get_instance()
config = ABTestConfig(
    name="greeting_test",
    prompt_id="user_greeting",
    variants=[
        ABTestVariant("v1", weight=1.0),
        ABTestVariant("v2", weight=1.0),
    ],
)
manager.create_experiment(config)

# Run experiment
with manager.experiment("greeting_test") as exp:
    prompt = exp.get_prompt(name="Alice")
    response = my_llm.generate(prompt)  # Your LLM call
    exp.record(output=response, score=0.8)

# Analyze results
result = manager.analyze("greeting_test")
print(result.summary())

CLI Commands:

# Create an A/B test experiment
pvcs ab create my_test user_greeting --variants v1,v2

# List all experiments
pvcs ab list

# View experiment status
pvcs ab status my_test

# Manually record a result
pvcs ab record my_test v1 --score 0.8

# Analyze results
pvcs ab analyze my_test

Winner detection requires at least five scored records per variant and 95% confidence. Install prompt-vcs[analysis] for Welch's t-test on small samples; without SciPy, a conservative normal approximation is used only for groups of at least 30 records.

📖 CLI Commands

Command Description
pvcs init Initialize project (single-file mode, creates prompts.yaml)
pvcs init --split Initialize project (multi-file mode, creates prompts/ dir)
pvcs scaffold <dir> Scan code and generate prompts (auto-detects mode)
pvcs switch <id> <version> Switch prompt version
pvcs status View current lock status
pvcs list List prompt IDs, locked versions, and available versions
pvcs add <id> <template> Add a prompt or prompt version
pvcs delete <id> Delete a prompt and remove its lock
pvcs unlock <id> Remove a prompt from the lockfile
pvcs migrate <path> Auto-migrate hardcoded prompts
pvcs migrate <path> --clean Migrate and extract prompts to YAML files
pvcs test <suite.yaml> Run prompt tests from YAML suite
pvcs validate <id> <output> --config <file> Validate output against configured rules
pvcs diff <id> <v1> <v2> Compare two versions of a prompt
pvcs log <id> Show Git commit history for a prompt
pvcs export --format <json|openai|langchain> Export prompts to a portable format
pvcs ab create <name> <id> Create an A/B test experiment
pvcs ab list List all A/B test experiments
pvcs ab status <name> View experiment status and variants
pvcs ab analyze <name> Analyze experiment results
pvcs ab record <name> <v> Manually record a test result
pvcs ab clear <name> Clear records for an experiment

Lockfiles are enforced strictly: if a locked version is missing or the lockfile is malformed, prompt resolution fails instead of silently falling back to another template.

✅ Repository Verification

Run the cross-platform verification entrypoint from the repository root:

python scripts/verify.py            # Full local regression suite
python scripts/verify.py --quick    # Fast feedback while developing
python scripts/verify.py --release  # Audit, package build, and wheel smoke test

Install the extension dependencies once before running the combined verification:

npm --prefix vscode-extension ci

🧰 Runnable Test Project

The repository includes an offline customer-support test project that covers prompt rendering, version locking and switching, YAML test suites, output validation, and Python unit tests. It requires no API key:

python -m pip install -e ".[dev]"
powershell -ExecutionPolicy Bypass -File .\examples\customer-support-demo\run_all.ps1

See examples/customer-support-demo/README.md for the complete walkthrough and expected output.

🤝 Contributing

Issues and Pull Requests are welcome!

📄 License

MIT License - See LICENSE file for details

👤 Author

emerard - @emerardd