Verdict Framework Demo

A comprehensive demonstration project showcasing the Verdict autonomous agent framework. This demo provides hands-on examples of pipelines, agents, guards, tools, auditing, evaluation, budgeting, and real LLM integration.

Quick Start

Prerequisites

• Rust 1.70+ installed
• Verdict library available at ../verdict (relative to this project)
• (Optional) An LLM endpoint for the live demo

Building

cargo build

Running Demos

All subcommands are invoked via:

cargo run -- <command>

---

Demo Subcommands

1. pipeline — Pipeline Structure & Guards

What it demonstrates:

• How to define a multi-step pipeline
• Guard conditions (guard_in, guard_out)
• Verdict types and failure modes
• Step dependencies and parallel execution flags
• Tool scoping per step

How to run:

cargo run -- pipeline

What to expect:

• Prints a 2-step analysis pipeline
• Shows guard conditions on each step
• Attempts to run the pipeline without an LLM client
• Prints the expected error: "no LLM client configured"

Output example:

Pipeline Name: DemoAnalysisPipeline
Steps: 2
Max Retries: 2

Step 1 - Plan:
  Guard In:  None
  Guard Out: NonEmptyOutput
  Tools:     ReadOnly
  ...

[Expected behavior] Running without LLM client configured:
✗ Expected error (no LLM client): step 'Plan' failed: ...

---

2. agents — Agent Registry & Built-in Agents

What it demonstrates:

• Agent registration in a registry
• Built-in agent definitions (coder, reviewer, planner)
• Agent introspection: name, description, policies
• Pipeline composition per agent

How to run:

cargo run -- agents

What to expect:

• Lists all registered agents
• Shows detailed policy constraints for the planner agent
• Displays each agent's pipeline steps

Output example:

Registered Agents:
  - coder
  - reviewer
  - planner

[Planner Agent Details]
  Name: planner
  Description: Plans and orchestrates agent workflows
  Pipeline Steps: 2
    - understand_goal
    - create_plan
  Policy - Max Steps: 20
  Policy - Max Retries: 3
  Policy - Max Delegation Depth: 5

---

3. guards — Guard Evaluations

What it demonstrates:

• All 8 core guard types in action
• Pass/fail test cases
• Content validation (JSON, TOML, YAML)
• Security guards (secrets, empty output)

How to run:

cargo run -- guards

What to expect:

• 8 sequential tests, each printing ✓ PASS or ✗ FAIL
• Tests cover:
  1. NonEmptyOutput on "hello world" → PASS
  2. NonEmptyOutput on "" → FAIL
  3. ValidJson on valid JSON → PASS
  4. ValidJson on invalid text → FAIL
  5. ValidToml on valid TOML → PASS
  6. ValidYaml on valid YAML → PASS
  7. NoSecretsInOutput on suspicious text → depends on scanner
  8. NoSecretsInOutput on clean text → PASS

Output example:

Running 8 guard test cases...

✓ Test 1: NonEmptyOutput on 'hello world' -> PASS
✓ Test 2: NonEmptyOutput on '' -> FAIL
✓ Test 3: ValidJson on valid JSON -> PASS
✓ Test 4: ValidJson on 'not json' -> FAIL
✓ Test 5: ValidToml on valid TOML -> PASS
✓ Test 6: ValidYaml on valid YAML -> PASS
✓ Test 7: NoSecretsInOutput on 'hunter2' -> PASS (no secret pattern matched)
✓ Test 8: NoSecretsInOutput on 'the sky is blue' -> PASS

---

4. tools — Function Tools & Tool Sets

What it demonstrates:

• Creating async function-based tools
• Tool schema validation
• Tool registry and execution
• ToolSet permission models: ReadOnly, Allow, Deny, Full

How to run:

cargo run -- tools

What to expect:

• Creates two custom tools: word_count and to_uppercase
• Demonstrates tool invocation with parameters
• Shows tool output as JSON
• Tests ToolSet permission checking

Output example:

Creating function tools...

Testing 'word_count' tool:
  ✓ Result: {"count":4}

Testing 'to_uppercase' tool:
  ✓ Result: {"result":"HELLO WORLD"}

[ToolSet Permission Checking]
  ReadOnly.contains('word_count'): false
  Allow(['word_count']).contains('word_count'): true
  Allow(['word_count']).contains('to_uppercase'): false

---

5. audit — Audit Logging & Security Scanning

What it demonstrates:

• Audit log creation and entry appending
• Different AuditEvent types
• Prompt injection detection (high-risk patterns)
• Secret scanning (API keys, credentials)

How to run:

cargo run -- audit

What to expect:

• Creates an audit log with 5 sample entries
• Prints entries as a formatted table: timestamp | pipeline | step | event
• Scans two text samples for injection patterns (one malicious, one clean)
• Scans two text samples for secrets (one with API key pattern, one clean)

Output example:

Audit Log Entries:
Timestamp                 Pipeline             Step            Event
──────────────────────────────────────────────────────────────────────────────
14:23:45.123             main_pipeline        step_1          StepStarted
14:23:45.124             main_pipeline        step_1          GuardPassed(NonEmptyOutput)
14:23:45.125             main_pipeline        step_1          ToolCall(word_count)
14:23:45.126             main_pipeline        step_2          StepCompleted(PASS)
14:23:45.127             main_pipeline        final           PipelineCompleted(2/0)

[Injection Scanner Tests]
  Scan 'ignore all previous...': detected=true, risk=Some(Critical)
  Scan 'the cat sat on the mat': detected=false, risk=None

[Secret Scanner Tests]
  Scan 'my openai key...': found 1 matches
    - Pattern: openai_api_key, Position: 23
  Scan 'nothing secret here': found 0 matches

---

6. eval — Evaluation Suite

What it demonstrates:

• Building an evaluation suite with multiple test cases
• Running evaluation against a pipeline
• Different evaluation strategies:
  • Guard-based validation
  • Custom closure validation
• Scoring and pass/fail results

How to run:

cargo run -- eval

What to expect:

• Creates a pipeline with a Custom action (no LLM needed)
• Defines 3 evaluation cases
• Runs the suite and reports per-case scores
• Displays overall suite score and pass/fail status

Output example:

Running evaluation suite...

Evaluation Results:
  Suite: TextGeneration
  Overall Score: 100.00%
  Passed: true

  Case Results:
    ✓ Contains 'agent' (score: 100.00)
    ✓ Valid output structure (score: 100.00)
    ✓ Custom validation (score: 100.00)

---

7. budget — Budget Tracking & Rate Limiting

What it demonstrates:

• Creating a budget tracker with cost and call limits
• Recording LLM calls and checking limits
• Rate limiting: preventing calls exceeding a threshold per minute
• Remaining budget calculations

How to run:

cargo run -- budget

What to expect:

• Demonstrates budget tracking with a 3-call limit and $10 budget
• Records calls until limit is exceeded
• Shows remaining calls and USD after each call
• Demonstrates rate limiting with 2 calls per minute
• Shows ALLOWED/BLOCKED status for each rate-limit check

Output example:

[Budget Tracker Demo]
Initial budget: 3 LLM calls, $10.00
  Call 1: cost=$0.50, remaining calls: Some(2), remaining USD: Some(9.5)
  Call 2: cost=$0.50, remaining calls: Some(1), remaining USD: Some(9.0)
  Call 3: cost=$0.50, remaining calls: Some(0), remaining USD: Some(8.5)
  Call 4: cost=$0.50, remaining calls: None, remaining USD: None
    Budget limit exceeded: LLM call limit exceeded: 4 calls, max 3

[Rate Limiter Demo]
Rate limit: 2 calls per minute

  Call 1: ✓ ALLOWED
  Call 2: ✓ ALLOWED
  Call 3: ✗ BLOCKED (Rate limit exceeded: ...)

---

8. monitor — Monitoring Server

What it demonstrates:

• Starting a monitoring server on a local port
• Creating audit logs and execution traces
• Exposing monitoring endpoints: /audit, /trace, /ui
• Real-time observability infrastructure

How to run:

cargo run -- monitor

What to expect:

• Starts a monitoring server on 127.0.0.1:9001
• Populates sample audit and trace data
• Prints available endpoints
• Waits 10 seconds then exits
• (Note: actual HTTP endpoints are not available in this demo version)

Output example:

Starting monitoring server on 127.0.0.1:9001...

✓ Server started

Monitoring endpoints:
  - http://127.0.0.1:9001/audit - Audit log JSON
  - http://127.0.0.1:9001/trace - Pipeline trace JSON
  - http://127.0.0.1:9001/ui   - Live monitoring UI

Waiting 10 seconds for demo...
✓ Demo complete

---

9. live — Real LLM Integration

What it demonstrates:

• Configuring an LLM client with real API credentials
• Building a 2-step poetry-themed pipeline
• Running the pipeline end-to-end with LLM calls
• Error handling when LLM is unavailable

How to run:

cargo run -- live

LLM Configuration:

The demo uses hardcoded defaults:

• Base URL: http://192.168.178.166:4141/v1
• API Key: sk-llmp-239b82f7192fd75bff9300d1391bacafe049144a19f84d6d26f9cbc5cfb944d9.MainPC
• Model: gpt-4o-mini

What to expect:

1. If LLM is available:

   • ✓ LLM client configured successfully
   • Pipeline runs 2 steps:
     • Step 1: Compose a haiku about Rust
     • Step 2: Critique the haiku
   • Prints output from both steps (first 200 chars)
   • Shows timing and step counts

2. If LLM is unavailable:

   • ✗ Failed to configure LLM client: [network error]
   • Suggests alternative demos that don't require LLM

Output example (success):

✓ LLM client configured successfully

Running 2-step poetry pipeline...

✓ Pipeline completed successfully

Results:

Step: compose_haiku
Output (first 200 chars):
  Rust code flows so clean,
  Memory safe without the fear,
  Ownership reigns true

Step: critique
Output (first 200 chars):
  This haiku effectively captures Rust's core value proposition through natural imagery...

Steps Passed: 2
Steps Failed: 0

Output example (failure):

✗ Failed to configure LLM client: Network error

Note: This demo requires an LLM endpoint running at:
  http://192.168.178.166:4141/v1

For testing without a real LLM endpoint, run other demos:
  cargo run -- pipeline
  cargo run -- guards
  cargo run -- tools

---

Project Structure

verdict-demo/
├── Cargo.toml          # Project manifest with dependencies
├── README.md           # This file
└── src/
    └── main.rs         # Single-file demo implementation with all 9 subcommands

Key Verdict Concepts Demonstrated

Concept	Demo	Location
Pipeline	Multi-step workflow definition	demo_pipeline()
Guards	Pre/post-condition validation	demo_guards()
Verdicts	Step execution results	demo_pipeline()
Tools	Custom function-based tools	demo_tools()
Agents	Named autonomous entities	demo_agents()
Audit Log	Execution history & events	demo_audit()
Injection Scanning	Prompt injection detection	demo_audit()
Secret Scanning	Credential leak detection	demo_audit()
Evaluation Suite	Test agents against criteria	demo_eval()
Budget Tracking	Cost & call limits	demo_budget()
Rate Limiting	Call frequency control	demo_budget()
Monitoring	Real-time observability	demo_monitor()
LLM Integration	OpenAI-compatible API calls	demo_live()

Dependencies

• verdict: The Verdict framework library
• tokio: Async runtime and utilities
• serde_json: JSON serialization/deserialization
• async-trait: Async trait support
• chrono: Datetime handling with serde support

Troubleshooting

Build Issues

Error: cannot find crate 'verdict'

• Ensure the verdict library is at ../verdict relative to this project
• Run cargo check to validate the path

Error: failed to compile 'chrono'

• Ensure chrono is in Cargo.toml with serde feature enabled

Runtime Issues

live demo fails with "Connection refused"

• The LLM endpoint is not running or not accessible
• Check the configured URL and API key in main.rs
• Try running other demos first (pipeline, guards, tools)

monitor demo shows empty endpoints

• The monitoring server is started but endpoints are not fully wired in this demo
• This is expected; it demonstrates the server startup process

Future Extensions

This demo can be extended to showcase:

• Multi-agent delegation and collaboration
• Skill-based pipelines with prompt injection
• Custom guard implementations
• Tool orchestration and MCP integration
• Self-update mechanics and approval workflows
• Advanced evaluation metrics and benchmarking

License

Part of the Verdict framework project.