QUICKSTART.md

Quick Start

This quickstart guide demonstrates how to use the xk6-ai extension to test AI agents using automated scenarios. The extension provides a JavaScript/TypeScript API for k6 test scripts to interact with AI tools, generate tasks, and evaluate responses.

Basic Usage

1. Create a Simple Test Script

Create a file examples/basic.test.js:

import { Agent, User, Validator } from 'k6/x/ai';

// Configuration
const llmConfig = {
  type: 'openai',
  apiKey: __ENV.OPENAI_API_KEY,
  model: 'gpt-4',
  maxTokens: 1000,
  temperature: 0.7,
  timeout: 30,
  retryCount: 3
};

const agentConfig = {
  url: __ENV.AGENT_URL,
  name: 'Test Agent',
  description: 'A test AI agent',
  capabilities: ['text', 'analysis'],
  timeout: 60,
  retryCount: 3
};

const userPrompt = `
Generate a prompt for testing an AI agent's ability to analyze text sentiment.
The task should be clear, specific, and testable.
`;

const expectations = [
 { name: "Accuracy", criteria: "Does the response correctly identify sentiment?", weihgt: .60},
 { name: "Completeness", criteria: "Is the response complete and detailed?", weight: .20 },
 { name: "Clarity", criteria: "Is the response clear and well-structured?", weight: .10 },
 { name: "Relevance": criteria: "Does the response address the task requirements?", weight: .10 }
]

export default async function() {
  // Create user with prompt
  const user = new User(llmConfig, userPrompt);
  
  // Create agent
  const agent = new Agent(agentConfig);
  
  // Create validator
  const validator = new Validator(llmConfig, expectations);
  
  // Generate task
  const task = await user.request();
  console.log('Generated task:', task);
  
  // Execute task against agent
  const response = await agent.perform(task);

  console.log('Agent response:', response.content);
    
  // Assert minimum score
  check(evaluation.overallScore, {
    'evaluation score is above threshold': (score) => score >= 0.8
  });
}

2. Run the Test

# Set environment variables
export OPENAI_API_KEY="your-openai-api-key"
export AGENT_URL="https://your-agent-endpoint.com"

# Run the test
./k6 run examples/basic.test.js

Multiple LLM Providers

import { Agent, User, Validator } from 'k6/x/ai';

export default async function() {
  // Create different LLM providers
  const openaiLLM = {
    provider: 'openai',
    apiKey: __ENV.OPENAI_API_KEY,
    model: 'gpt-4'
  };
  
  const anthropicLLM = {
    provider: 'anthropic',
    apiKey: __ENV.ANTHROPIC_API_KEY,
    model: 'claude-3-sonnet'
  };
  
  // Use OpenAI for task generation
  const user = new User(openaiLLM, userPrompt);
  
  // Use Anthropic for validation
  const validator = new Validator(anthropicLLM, expectations);
  
  // Rest of the test...
}

Custom evaluation criteria

import { Agent, User, Validator } from 'k6/x/ai';

export default async function() {
  const agent = new Agent(agentConfig);
  
  // Create user with specific prompt
  const user = new User(llmConfig, `
    Generate a prompt for testing code generation capabilities.
    The prompt should require the agent to write a function that:
    - Takes a list of numbers as input
    - Returns the sum of all even numbers
    - Handles edge cases (empty list, non-numeric values)
    - Includes proper error handling
  `);
  
  // create detailed expectations
  const expectations = [
    { 
      name "functionality",
      criteria: `
       - Does the function work correctly?
       - Does it handle edge cases?
       - Is error handling implemented?
      `
    },
    { 
      name: "quality",
      criteria: `
       - Is the code readable and well-structured?
       - Are variable names meaningful?
       - Is the code efficient?
      `
    },
    { 
      name: "completeness",
      criteria: `
       - Is the function complete?
       - Are all requirements addressed?
       - Is documentation included?
    `
    },
    {
      name: "Practices",
      criteria: `
       - Does the code follow language conventions?
       - Are there any security issues?
       - Is the code maintainable?
       `
    }
  ]

  // Create validator with detailed criteria
  const validator = new Validator(llmConfig, expectations);
  
  const task = await user.request();
  const response = agent.perform(task);
  const evaluation = validator.evaluate(task, response);
  
  // Check individual criteria
  check(evaluation, {
    'functionality score is high': (evaluation) => 
      evaluation.results.find(c => r.name === 'functionality').score >= 0.8,
    'code quality score is acceptable': (criteria) => 
      evaluation.result.find(c => r.name === 'quality').score >= 0.7,
    'overall score meets threshold': (evaluation) => 
      evaluation.overallScore >= 0.75
  });
}

Best Practices

1. Test Design

• Clear Prompts: Write specific, testable prompts
• Realistic Scenarios: Use real-world use cases
• Edge Cases: Test boundary conditions
• Performance: Monitor response times and resource usage

2. Evaluation Criteria

• Specific Metrics: Define clear, measurable criteria
• Weighted Scoring: Use appropriate weights for different criteria
• Thresholds: Set realistic score thresholds
• Documentation: Document evaluation reasoning

3. Load Testing

• Gradual Ramp-up: Start with low VU count and increase gradually
• Realistic Load: Use realistic load patterns
• Monitoring: Monitor both performance and accuracy metrics
• Error Handling: Implement proper error handling and retry logic

4. Security

• API Keys: Never hardcode API keys in test scripts
• Environment Variables: Use environment variables for sensitive data
• Input Validation: Validate all inputs from external sources
• Error Messages: Don't expose sensitive information in error messages

Common Issues

1. Extension Not Found

   Error: module 'k6/x/ai' not found
   

   Solution: Ensure you built k6 with the xk6-ai extension

2. API Key Issues

   Error: invalid API key
   

   Solution: Check environment variables and API key validity

3. Agent Timeout

   Error: agent request timeout
   

   Solution: Increase timeout settings or check agent availability

4. LLM Rate Limits

   Error: rate limit exceeded
   

   Solution: Implement retry logic with exponential backoff