Added a prompt optimization example

[htahir1]

Describe changes

This PR adds a new example demonstrating ZenML's artifact
management capabilities through a two-stage AI prompt optimization
workflow using Pydantic AI for exploratory data analysis.

Key Features:

• Two-stage workflow: optimization → production with artifact
  sharing
• Exclusive artifact tagging for best prompt selection
• Cross-pipeline artifact retrieval using ZenML's registry
• AI-powered EDA with multiple LLM providers (OpenAI/Anthropic)
• Support for multiple data sources (HuggingFace, local CSV)

What This Example Demonstrates

ZenML Core Capabilities:

• Artifact Management: Exclusive tagging, versioning,
  cross-pipeline sharing
• Pipeline Orchestration: Multi-stage workflows with artifact
  passing
• Client Library: Programmatic artifact registry access
• Lineage Tracking: End-to-end workflow traceability

Real-World Use Case:

• Systematic prompt testing and optimization for production AI
  systems
• Seamless transition from experimentation to production
  deployment
• Performance-based prompt selection with quality metrics

Usage

Complete workflow (default)

python run.py

Individual stages

python run.py --optimization-pipeline # Find best prompt
python run.py --production-pipeline # Use optimized prompt

Custom data sources

python run.py --data-source "local:data.csv"
python run.py --data-source "hf:scikit-learn/wine"

Technical Highlights

Artifact Management Pattern:

Stage 1: Tag best prompt with exclusive tag

add_tags(tags=[Tag(name="optimized", exclusive=True)],
infer_artifact=True)

Stage 2: Retrieve tagged prompt

artifacts = client.list_artifact_versions(tags=["optimized"],
size=1)
optimized_prompt = artifacts.items[0] if artifacts.items else
default_prompt

Simplified Architecture:

• Removed over-engineered components (quality gates, complex
  experiment tracking)
• Focused on core ZenML concepts with practical AI use case
• Clean separation of concerns with typed artifacts
• Defensive error handling with meaningful fallbacks

Code Quality

• Simplified from original: 75% reduction in ingest step (374→93
  lines)
• Concise documentation: Practical README focused on getting users
  started
• Sane defaults: python run.py runs complete workflow
• Multiple LLM support: Auto-detects available API keys
• Type safety: Full Pydantic models with proper annotations

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch doc/pydanticaiexample

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

✅ No broken links found!

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	numpy@​2.0.0rc2

View full report