CLAUDE.md

AI Assistant Instructions for DreamOps Oncall Agent Codebase

This document provides comprehensive instructions for AI assistants (like Claude, GPT-4, etc.) on how to effectively work with this codebase.

⚠️ CRITICAL: PagerDuty Test Incident Rules ⚠️

EXTREMELY IMPORTANT - READ THIS FIRST:

When testing PagerDuty webhooks or creating test incidents:

• ⚠️ ALWAYS RESOLVE TEST INCIDENTS IMMEDIATELY AFTER CREATING THEM
• ⚠️ Test incidents disturb the on-call engineer and trigger real alerts
• ⚠️ Never leave test incidents open - resolve them within seconds

How to resolve an incident:

# Always use the same dedup_key you used to trigger
curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "<YOUR_ROUTING_KEY>",
    "event_action": "resolve",
    "dedup_key": "<SAME_DEDUP_KEY_USED_TO_TRIGGER>"
  }'

Best Practice:

1. Generate a unique dedup_key: test-$(date +%s)
2. Trigger the test incident
3. IMMEDIATELY send the resolve event with the same dedup_key
4. Include "(TEST BY SKY)" in summaries so engineers know it's a test

CRITICAL: Git Commit Message Rules

IMPORTANT: When creating git commits, follow these strict rules:

• ❌ NEVER EVER add co-author credits in commit messages
• ❌ DO NOT include any "Co-Authored-By" lines
• ❌ DO NOT add "Generated with Claude Code" or similar attribution
• ❌ DO NOT add emoji or branding in commit messages
• ✅ Keep commit messages clean, professional, and focused on the changes only
• ✅ Use conventional commit format: type(scope): description
• ✅ Be concise and descriptive about what changed and why

Example of a GOOD commit message:

refactor(auth): remove authentication in favor of Authentik proxy

- Remove all Firebase authentication code from backend and frontend
- Delete auth routers, models, and security modules
- Update deployment docs for bare-metal K3s with Authentik
- Remove firebase-admin, pyjwt, and bcrypt dependencies

Documentation Structure

IMPORTANT: All project documentation has been consolidated into the main README.md file. Before making any changes, read the relevant sections in README.md:

• Database Setup - Database configuration for all environments
• Configuration - PagerDuty setup and webhook configuration
• Features & Integrations - YOLO mode, MCP integrations, and safety mechanisms
• Deployment - AWS deployment options (Terraform, Amplify)
• Architecture & Technical Details - Implementation details and fixes
• CI/CD - GitHub Actions workflows and deployment automation

Always check the README.md for the latest information before implementing changes.

CRITICAL: Documentation Guidelines

IMPORTANT: Follow these strict rules when creating or updating documentation:

Documentation Location Rules

1. ALWAYS write documentation in the /docs directory ONLY

   • ❌ DO NOT create markdown files in the root directory (except README.md and CLAUDE.md)
   • ❌ DO NOT create documentation files in subdirectories (backend/, frontend/, etc.)
   • ✅ ALL documentation files MUST go in /docs/
   • ✅ Use clear, descriptive filenames (e.g., kubernetes-integration.md, not k8s-stuff.md)

2. ALWAYS combine similar documentation whenever possible

   • If you're creating a new doc about Kubernetes, check if /docs/kubernetes-integration.md exists
   • If similar documentation exists, UPDATE the existing file instead of creating a new one
   • Only create a new file if the topic is distinctly different
   • Example: Combine "Kubernetes Setup", "Kubernetes Configuration", and "Kubernetes Troubleshooting" into ONE file

3. Keep documentation organized and consolidated

   • Prefer fewer, comprehensive files over many small files
   • Use sections and subsections within files to organize content
   • Link between docs when necessary using relative paths
   • Maintain a clear structure: Overview → Setup → Configuration → Usage → Troubleshooting

Documentation File Structure

Organize files in /docs by topic:

/docs
├── README.md                       # Index of all docs
├── deployment.md                   # All deployment guides (AWS, K3s, Docker, etc.)
├── kubernetes-integration.md       # All Kubernetes-related docs
├── grafana-integration.md          # Grafana setup and configuration
├── pagerduty-integration.md        # PagerDuty webhook and API
└── troubleshooting.md             # Common issues and solutions

What NOT to Document

• ❌ DO NOT create separate files for minor features
• ❌ DO NOT duplicate information across multiple files
• ❌ DO NOT create README files in subdirectories
• ❌ DO NOT write docs in code comments when they belong in /docs

Documentation Best Practices

1. Before creating a new doc: Check /docs directory for existing files on that topic
2. Before updating a doc: Read the entire file to understand its structure
3. After updating: Ensure links are updated if file names changed
4. Keep it DRY: Don't Repeat Yourself - link to existing documentation instead
5. Use consistent formatting: Follow the style of existing documentation

Example: Good vs Bad Documentation Approach

❌ BAD Approach:

Creating separate files:
- /docs/k8s-setup.md
- /docs/kubernetes-config.md
- /docs/kubernetes-troubleshooting.md
- /docs/k8s-deployment.md
- /backend/kubernetes-notes.md

✅ GOOD Approach:

One comprehensive file:
- /docs/kubernetes-integration.md
  - Section: Setup
  - Section: Configuration
  - Section: Deployment
  - Section: Troubleshooting
  - Section: MCP Integration

Project Overview

DreamOps is an intelligent AI-powered incident response and infrastructure management platform built with:

• Claude API: For intelligent incident analysis and decision making
• MCP (Model Context Protocol): For integrating with external tools and services
• FastAPI: Web framework for the REST API and webhooks
• Python AsyncIO: For concurrent operations and real-time processing
• uv: Modern Python package manager for dependency management
• Next.js: Frontend SaaS interface with real-time dashboard
• Terraform: AWS infrastructure deployment and management
• Neon: PostgreSQL database with environment separation
• Docker: Containerization for consistent deployments

Key Architecture Decisions

1. Individual User Model: The platform operates on an individual user basis WITHOUT teams. Users register and use the platform as individuals, not as part of teams or organizations.
2. Authentication via Authentik: Authentication is handled by Authentik reverse proxy, not built into the application. User identity is provided via headers.
3. Modular MCP Integrations: All integrations extend MCPIntegration base class in src/oncall_agent/mcp_integrations/base.py
4. Async-First: All operations are async to handle concurrent MCP calls efficiently
5. Configuration-Driven: Uses pydantic for config validation and environment variables
6. Type-Safe: Extensive use of type hints throughout the codebase
7. Retry Logic: Built-in exponential backoff for network operations (configurable via MCP_MAX_RETRIES)
8. Singleton Config: Global configuration instance accessed via get_config()
9. Environment Separation: Complete database and configuration isolation between local/staging/production
10. YOLO Mode: Autonomous remediation mode that executes fixes without human approval

Project Structure

backend/
├── src/oncall_agent/
│   ├── agent.py              # Core agent logic
│   ├── agent_enhanced.py     # Enhanced agent with YOLO mode
│   ├── agent_executor.py     # Command execution engine
│   ├── api.py                # FastAPI REST API
│   ├── config.py             # Configuration management
│   ├── mcp_integrations/     # MCP integration modules
│   │   ├── base.py          # Base integration class
│   │   ├── kubernetes.py    # Kubernetes integration
│   │   ├── github_mcp.py    # GitHub integration
│   │   ├── notion_direct.py # Notion integration
│   │   └── pagerduty.py     # PagerDuty integration
│   ├── strategies/          # Resolution strategies
│   │   ├── kubernetes_resolver.py
│   │   └── deterministic_k8s_resolver.py
│   └── utils/               # Utility functions
│       └── logger.py        # Logging configuration
├── tests/                    # All test files
├── examples/                 # Example scripts and demos
├── scripts/                  # Utility scripts
├── main.py                   # CLI entry point
└── Dockerfile.prod          # Production Docker image

frontend/
├── app/                     # Next.js app router
│   ├── (dashboard)/        # Dashboard pages
│   ├── (login)/            # Authentication pages
│   └── api/                # API routes
├── components/             # React components
│   ├── ui/                # UI components
│   └── incidents/         # Incident-specific components
├── lib/                   # Utilities and configurations
│   ├── db/               # Database utilities
│   ├── auth/             # Authentication
│   └── hooks/            # React hooks
└── scripts/              # Build and deployment scripts

Working with the Codebase

When Adding New MCP Integrations

1. Always extend the base class:

   from src.oncall_agent.mcp_integrations.base import MCPIntegration

2. Follow the established pattern:

   • Implement all abstract methods
   • Use the retry mechanism for network operations
   • Log all operations appropriately
   • Handle errors gracefully

3. File location: Create new integrations in src/oncall_agent/mcp_integrations/

4. Required methods to implement:

   async def connect() -> bool
   async def disconnect() -> None
   async def fetch_context(params: Dict[str, Any]) -> Dict[str, Any]
   async def execute_action(action: str, params: Dict[str, Any]) -> Dict[str, Any]
   def get_capabilities() -> List[str]
   async def health_check() -> bool

Code Style Guidelines

• Use descriptive variable names
• Add type hints to all functions
• Include docstrings for all public methods
• Follow PEP 8 conventions
• Use async/await for all I/O operations
• Handle exceptions gracefully with proper logging
• Use dataclasses for structured data
• Implement proper JSON serialization for API responses

File Management Guidelines

IMPORTANT: DO NOT CREATE UNNECESSARY FILES

This is a production codebase. When working with this codebase, strictly follow these file management rules:

1. NEVER create test files - We will write tests later in a structured manner:

   • No test_*.py files
   • No *_test.py files
   • No test directories
   • No pytest files

2. NEVER create demo/example files:

   • No demo_*.py files
   • No example_*.py files
   • No simulation scripts
   • No mock implementations

3. NEVER create temporary or experimental files:

   • No temp_*.py files
   • No tmp_*.py files
   • No experiment_*.py files
   • No quick test scripts
   • No validation scripts unless explicitly requested

4. NEVER create duplicate implementations:

   • Check if functionality already exists before creating new files
   • Use existing integrations rather than creating alternatives
   • Extend existing classes rather than creating new versions

5. Keep the codebase clean:

   • Only create files that are essential for production functionality
   • Remove any generated log files, temporary files, or backup files
   • Do not commit .bak, .tmp, .log, or similar files

6. When asked to test or validate:

   • Use the existing main.py or api_server.py to test functionality
   • Modify existing files rather than creating new test files
   • Use the Python REPL or existing scripts for quick checks

Remember: This is a production codebase that should remain clean and focused. Every file should serve a clear production purpose.

Common Patterns in This Codebase

1. Error Handling:

   try:
       result = await operation()
   except Exception as e:
       self.logger.error(f"Operation failed: {e}")
       return {"error": str(e)}

2. Configuration Access:

   from src.oncall_agent.config import get_config
   config = get_config()

3. Logging:

   from src.oncall_agent.utils.logger import get_logger
   logger = get_logger(__name__)

4. JSON Serialization (IMPORTANT):

   # Always convert dataclasses to dict for JSON serialization
   execution_context = {
       "action": action.to_dict() if hasattr(action, 'to_dict') else action.__dict__,
       "result": result
   }

Important Files to Understand

1. src/oncall_agent/agent_enhanced.py: Enhanced agent with YOLO mode capabilities
2. src/oncall_agent/agent_executor.py: Command execution engine for Kubernetes operations
3. src/oncall_agent/mcp_integrations/base.py: Base class defining the MCP interface
4. src/oncall_agent/config.py: Configuration schema and defaults
5. src/oncall_agent/api.py: FastAPI application with REST endpoints
6. main.py: Entry point showing how to use the agent
7. src/oncall_agent/strategies/deterministic_k8s_resolver.py: Kubernetes resolution strategies

Commands and Development Workflow

Development Commands

• Install dependencies: uv sync
• Run the agent: uv run python main.py
• Run API server: uv run python api_server.py
• Add dependency: uv add <package>
• Add dev dependency: uv add --dev <package>

Environment Setup

# Backend setup
cd backend
uv sync
cp .env.example .env
# Edit .env with your API keys

# Frontend setup
cd frontend
npm install
npm run dev

Testing and Validation

• Run demo: uv run python main.py
• Run API server: uv run uvicorn src.oncall_agent.api:app --reload
• Run tests: uv run pytest tests/
• Run linter: uv run ruff check . --fix
• Run type checker: uv run mypy . --ignore-missing-imports

Database Operations

# Frontend database operations
cd frontend
npm run db:migrate:local      # Migrate local database
npm run db:migrate:staging    # Migrate staging database
npm run db:migrate:production # Migrate production (requires confirmation)
npm run db:studio            # Open Drizzle Studio
npm run test:db             # Test all database connections

IMPORTANT: Pre-commit Checklist

ALWAYS run these commands before finishing any task:

# 1. Run linter and fix any issues
uv run ruff check . --fix

# 2. Run type checker
uv run mypy . --ignore-missing-imports

# 3. Run all tests
uv run pytest tests/

# 4. Verify the application still runs
uv run python main.py

# 5. Test API server
uv run python api_server.py

If any of these fail, fix the issues before considering the task complete.

Testing Approach

When asked to test changes:

1. Check if .env exists, if not copy from .env.example
2. Ensure ANTHROPIC_API_KEY is set
3. Run with: uv run python main.py
4. For specific integrations, create mock implementations first
5. Test database connections: cd frontend && npm run test:db
6. Test API endpoints: curl -X GET http://localhost:8000/health

Common Tasks

Adding a New MCP Integration

1. Create file in src/oncall_agent/mcp_integrations/
2. Extend MCPIntegration base class
3. Implement all abstract methods
4. Add configuration to .env.example
5. Update README.md with usage instructions
6. Add integration to the main config file

Modifying Alert Handling

1. Look at PagerAlert model in agent_enhanced.py
2. Modify handle_pager_alert method
3. Update the prompt sent to Claude if needed
4. Test with different alert types
5. Ensure proper JSON serialization

Adding New Configuration Options

1. Add to Config class in config.py
2. Add to .env.example with description
3. Document in README.md
4. Handle in frontend configuration if needed

Implementing YOLO Mode Features

1. Understand the YOLO mode philosophy: execute ALL actions without human approval
2. Add new resolution strategies in strategies/ directory
3. Implement corresponding action executors in agent_executor.py
4. Ensure proper error handling and logging
5. Test with fuck_kubernetes.sh scenarios

Database Schema Changes

1. Modify schema in frontend/lib/db/schema.ts
2. Generate migration: npm run db:generate
3. Apply to local: npm run db:migrate:local
4. Test with Drizzle Studio: npm run db:studio
5. Apply to staging/production when ready

Testing Kubernetes Alerting

The project includes fuck_kubernetes.sh for simulating Kubernetes failures:

# Usage from project root
./fuck_kubernetes.sh [1-5|all|random|clean]

# Simulates:
# 1 - Pod crashes (CrashLoopBackOff)
# 2 - Image pull errors (ImagePullBackOff)  
# 3 - OOM kills
# 4 - Deployment failures
# 5 - Service unavailability

# The script:
# - Creates issues in 'fuck-kubernetes-test' namespace
# - Triggers CloudWatch alarms within 60 seconds
# - Sends alerts through SNS → PagerDuty → Slack
# - Helps verify the entire alerting pipeline

API Development

The project includes a FastAPI REST API in src/oncall_agent/api.py:

API Endpoints

• GET /health - Health check endpoint
• POST /alerts - Submit new alerts for processing
• GET /alerts/{alert_id} - Get alert details
• GET /integrations - List available MCP integrations
• POST /integrations/{name}/health - Check specific integration health
• POST /webhook/pagerduty - PagerDuty webhook handler
• GET /agent/config - Get current agent configuration
• POST /agent/config - Update agent configuration

Running the API

# Development with auto-reload
uv run uvicorn src.oncall_agent.api:app --reload

# Production
uv run uvicorn src.oncall_agent.api:app --host 0.0.0.0 --port 8000

API Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Deployment

The project includes comprehensive AWS deployment configuration:

Infrastructure

• Backend: ECS Fargate with ALB
• Frontend: S3 + CloudFront or AWS Amplify
• Database: Neon PostgreSQL with environment separation
• Monitoring: CloudWatch dashboards and alarms
• Security: Secrets Manager for sensitive data

CI/CD Pipelines

• Backend: .github/workflows/backend-ci.yml
• Frontend: .github/workflows/frontend-ci.yml
• Security: .github/workflows/security-scan.yml
• Markdown Check: .github/workflows/check-markdown-files.yml

Environment Management

The project uses strict environment separation:

• Local: For development with local database
• Staging: For testing with staging database
• Production: For production with production database

Each environment has its own:

• Database instance
• Configuration files
• Environment variables
• Deployment pipeline

GitHub Secrets Required

1. AWS_ACCESS_KEY_ID - AWS access key for deployment
2. AWS_SECRET_ACCESS_KEY - AWS secret key
3. ANTHROPIC_API_KEY - Your Anthropic API key
4. CLOUDFRONT_DISTRIBUTION_ID - CloudFront ID (after initial deployment)
5. AMPLIFY_APP_ID - Amplify app ID for frontend deployment
6. NEON_DATABASE_URL_STAGING - Staging database connection string
7. NEON_DATABASE_URL_PROD - Production database connection string

MCP Integration Interface

Every MCP integration must implement these methods from the base class:

• async def connect(): Establish connection to the service
• async def disconnect(): Gracefully close connections
• async def fetch_context(params: Dict[str, Any]): Retrieve information
• async def execute_action(action: str, params: Dict[str, Any]): Perform remediation
• def get_capabilities() -> List[str]: List available actions
• async def health_check() -> bool: Verify connection status

Kubernetes Integration

The Kubernetes MCP integration (src/oncall_agent/mcp_integrations/kubernetes.py) provides:

Features

• Pod management (list, logs, describe, restart)
• Deployment operations (status, scale, rollback)
• Service monitoring
• Event retrieval
• Automated resolution strategies
• Memory limit adjustments
• Resource constraint analysis

YOLO Mode Kubernetes Operations

When YOLO mode is enabled (K8S_ENABLE_DESTRUCTIVE_OPERATIONS=true), the agent can:

• Automatically restart failed pods
• Increase memory limits for OOM issues
• Scale deployments up/down
• Apply configuration patches
• Delete problematic resources

Kubernetes Action Types

• identify_error_pods: Find pods in error states
• restart_error_pods: Delete pods to force restart
• check_resource_constraints: Run kubectl top commands
• identify_oom_pods: Find OOM killed pods
• increase_memory_limits: Patch deployments with higher memory
• scale_deployment: Scale deployments up or down

PagerDuty Integration

Configuration

PAGERDUTY_API_KEY=your-api-key-here
PAGERDUTY_USER_EMAIL=your-email@company.com
PAGERDUTY_WEBHOOK_SECRET=your-webhook-secret

Webhook Handling

The PagerDuty webhook handler (/webhook/pagerduty) processes:

• Incident creation
• Incident updates
• Alert acknowledgments
• Incident resolutions

YOLO Mode Behavior

In YOLO mode, the PagerDuty integration:

• Always attempts to acknowledge incidents
• Automatically resolves incidents after successful remediation
• Ignores API errors and continues execution
• Logs warnings but doesn't fail the remediation process

Database Management

Schema Design

The database schema is defined in frontend/lib/db/schema.ts using Drizzle ORM:

• Users: Individual user accounts (NO TEAMS - users operate independently)
• Incidents: Incident tracking and history
• Metrics: Performance and analytics data
• Logs: Agent execution logs
• Integrations: User-specific integration configurations
• API Keys: User-specific API key management

IMPORTANT: This application uses an individual user model. There are NO teams, organizations, or multi-tenancy features. Each user has their own isolated data and integrations.

Migration Strategy

1. Local Development: Auto-migrate with npm run db:migrate:local
2. Staging: Manual migration with npm run db:migrate:staging
3. Production: Confirmation required with npm run db:migrate:production

Database Testing

Use the included database testing utilities:

cd frontend
node test-db-connections.mjs  # Test all connections
npm run db:studio            # Visual database exploration

Troubleshooting Common Issues

JSON Serialization Errors

Problem: TypeError: Object of type ResolutionAction is not JSON serializable

Solution: Always convert dataclasses to dictionaries:

# Add to_dict() method to dataclasses
def to_dict(self) -> dict[str, Any]:
    return asdict(self)

# Use in execution context
execution_context = {
    "action": action.to_dict(),
    "result": result
}

PagerDuty API Errors

Problem: "Requester User Not Found"

Solution: Ensure PAGERDUTY_USER_EMAIL is a valid user in your PagerDuty account.

Database Connection Issues

Problem: Connection timeouts or SSL errors

Solution:

1. Verify connection strings include ?sslmode=require
2. Check Neon project is active (not suspended)
3. Ensure no &channel_binding=require in connection string

YOLO Mode Not Executing

Problem: Agent not executing remediation actions

Solution:

1. Verify K8S_ENABLE_DESTRUCTIVE_OPERATIONS=true
2. Check K8S_ENABLED=true
3. Ensure kubectl is configured and working
4. Verify API server is running with correct configuration

Frontend Build Issues

Problem: Next.js build failures

Solution:

1. Check all environment variables are set
2. Verify database connection strings
3. Ensure API URL is accessible
4. Check for TypeScript errors

Security Considerations

Environment Variables

• Never commit .env files to version control
• Use different secrets for each environment
• Rotate API keys regularly
• Use least-privilege access for service accounts

Database Security

• Each environment uses completely separate databases
• Connection strings include SSL requirements
• Database users have minimal required permissions
• Regular security updates for database instances

Kubernetes Security

• RBAC permissions are minimally scoped
• Destructive operations require explicit enablement
• All kubectl commands are logged
• Namespace isolation for testing

API Security

• Authentication required for sensitive endpoints
• Request validation using Pydantic models
• Rate limiting implemented
• Comprehensive audit logging

Performance Optimization

Async Operations

• All I/O operations use async/await
• Concurrent processing of multiple alerts
• Connection pooling for database operations
• Efficient resource cleanup

Caching Strategy

• Configuration cached in memory
• Database query results cached where appropriate
• Static assets served from CDN
• Browser caching for frontend resources

Monitoring and Alerting

• CloudWatch integration for metrics
• Custom dashboards for system health
• Alerting on error rates and performance
• Real-time log streaming

Best Practices for AI Assistants

When Working with This Codebase

1. Always check the latest README.md and this CLAUDE.md for current instructions
2. Test all changes locally before suggesting them
3. Consider environment separation when making database changes
4. Respect YOLO mode safety mechanisms when adding new features
5. Follow the established patterns for error handling and logging
6. Document any new integrations or features thoroughly
7. Test with multiple scenarios including edge cases
8. Ensure JSON serialization compatibility for all API responses

Code Review Checklist

Before submitting any changes:

• All tests pass
• Type checking passes
• Linting passes
• Application runs successfully
• API endpoints respond correctly
• Database migrations work
• Documentation is updated
• Environment variables are documented
• Security considerations are addressed

This document should be your primary reference when working with the DreamOps codebase. Always refer back to it when implementing new features or making modifications.

Final Reminder: Production Code Standards

This is a PRODUCTION CODEBASE. Do not create:

• Test files (we'll add proper tests later)
• Demo or example files
• Temporary or experimental scripts
• Mock implementations
• Duplicate functionality
• Any "circus files" that don't serve production purposes

Keep the codebase clean, focused, and production-ready at all times.

• EVERY TIME U OPEN INCIDENT U NEED TO CLOSE IT IMMEDIATELY AS IT DISTURBS ONCALL ENGINEER