audit.txt

================================================================================
                    AXON REPOSITORY FULL AUDIT REPORT
================================================================================
Generated: 2026-03-14
Scope: backend/ and agents/ directories
Purpose: Complete system analysis and architecture documentation

================================================================================
                         1. SYSTEM OVERVIEW
================================================================================

AXON is a multi-agent AI orchestration platform built on FastAPI that 
coordinates specialized agents to execute complex tasks through a pipeline 
architecture. The system integrates with DigitalOcean's Gradient AI Platform
for LLM inference and agent deployment.

Core Architecture:
- Backend: FastAPI-based orchestration layer with PostgreSQL persistence
- Agents: Four specialized ADK agents deployed on DigitalOcean infrastructure
- Memory: ChromaDB vector store for semantic memory and context retrieval
- Skills: Pluggable execution modules for agent capabilities
- Evolution: Self-improving system that generates new skills from failures

Operating Modes:
- mock: Local testing with simulated LLM responses
- real: Production mode routing all inference through deployed ADK agents

Technology Stack:
- Backend: Python 3.11+, FastAPI, SQLAlchemy 2.0, AsyncPG
- Agents: gradient-adk, LangGraph, Gradient SDK
- Database: PostgreSQL with async support
- Vector Store: ChromaDB with sentence-transformers embeddings
- Observability: Loguru structured logging, WebSocket event streaming

================================================================================
                    2. FULL DIRECTORY STRUCTURE
================================================================================

backend/
├── .chroma/
│   └── chroma.sqlite3                    # Vector database persistence
├── .pytest_cache/                        # Test cache
├── .venv/                                # Python virtual environment
├── alembic/                              # Database migrations
│   ├── versions/
│   │   └── 0001_initial.py              # Initial schema migration
│   └── env.py                           # Alembic configuration
├── scripts/
│   ├── dev_test.py                      # Development testing utilities
│   ├── evaluation_dataset_example.csv   # Agent evaluation dataset
│   └── run_agent_evaluation.py          # Automated evaluation runner
├── src/
│   ├── agents/                          # Agent implementations
│   │   ├── base_agent.py               # Abstract base agent class
│   │   ├── planning_agent.py           # Task planning agent
│   │   ├── research_agent.py           # Research and information gathering
│   │   ├── reasoning_agent.py          # Strategy evaluation agent
│   │   └── builder_agent.py            # Solution generation agent
│   ├── ai/                              # LLM service layer
│   │   ├── llm_service.py              # Unified LLM interface
│   │   ├── gradient_client.py          # DigitalOcean Gradient client
│   │   └── huggingface_client.py       # HuggingFace inference client
│   ├── api/                             # HTTP API layer
│   │   ├── controllers/
│   │   │   ├── task_controller.py      # Task CRUD operations
│   │   │   ├── skill_controller.py     # Skill listing
│   │   │   └── evolution_controller.py # Evolution triggers
│   │   ├── middleware/
│   │   │   └── logging_middleware.py   # HTTP request logging
│   │   ├── routes/
│   │   │   ├── tasks.py                # Task endpoints
│   │   │   ├── skills.py               # Skill endpoints
│   │   │   ├── evolution.py            # Evolution endpoints
│   │   │   └── system.py               # System status endpoints
│   │   └── websocket/
│   │       └── event_stream.py         # Real-time event streaming
│   ├── config/
│   │   ├── config.py                   # Settings and environment variables
│   │   ├── agents_config.py            # Agent URL configuration
│   │   └── dependencies.py             # FastAPI dependency injection
│   ├── core/
│   │   ├── agent_orchestrator.py       # Multi-agent pipeline coordinator
│   │   ├── task_manager.py             # Async task queue manager
│   │   ├── event_bus.py                # Pub/sub event system
│   │   ├── evolution_engine.py         # Skill generation engine
│   │   ├── version_manager.py          # Version control utilities
│   │   └── debug_tools.py              # Debugging utilities
│   ├── db/
│   │   ├── models.py                   # SQLAlchemy ORM models
│   │   └── session.py                  # Database session management
│   ├── memory/
│   │   ├── vector_store.py             # ChromaDB vector operations
│   │   └── embeddings.py               # Sentence transformer embeddings
│   ├── providers/
│   │   └── digitalocean/
│   │       ├── digitalocean_agent_router.py    # Agent routing logic
│   │       ├── digitalocean_agent_client.py    # HTTP client for agents
│   │       └── digitalocean_agent_types.py     # Request/response types
│   ├── schemas/
│   │   ├── task.py                     # Task Pydantic models
│   │   ├── skill.py                    # Skill Pydantic models
│   │   ├── evolution.py                # Evolution Pydantic models
│   │   └── system.py                   # System status models
│   ├── services/
│   │   ├── task_service.py             # Task business logic
│   │   ├── skill_service.py            # Skill business logic
│   │   └── evolution_service.py        # Evolution business logic
│   ├── skills/
│   │   ├── core_skills/
│   │   │   ├── planning.py             # Planning skill implementation
│   │   │   ├── coding.py               # Code generation skill
│   │   │   └── reasoning.py            # Reasoning skill
│   │   ├── generated_skills/
│   │   │   └── web_search.py           # Generated web search skill
│   │   ├── templates/
│   │   │   └── skill_template.py       # Skill template
│   │   ├── registry.py                 # Skill discovery and registration
│   │   └── executor.py                 # Skill execution engine
│   ├── storage/
│   │   └── artifact_store.py           # File artifact storage
│   ├── utils/
│   │   ├── logger.py                   # Loguru configuration
│   │   ├── audit_logger.py             # Runtime audit generation
│   │   ├── validators.py               # Input validation
│   │   └── code_parser.py              # Code parsing utilities
│   └── main.py                          # FastAPI application entrypoint
├── tests/
│   ├── conftest.py                      # Pytest configuration
│   ├── test_agent_pipeline.py           # Agent orchestration tests
│   ├── test_api_tasks.py                # API endpoint tests
│   └── test_task_service.py             # Service layer tests
├── .env.example                         # Environment variable template
├── alembic.ini                          # Alembic configuration
├── pyproject.toml                       # Project dependencies
└── audit.log                            # Runtime audit log

agents/
├── planner_agent/
│   ├── main.py                          # Planner agent implementation
│   ├── requirements.txt                 # Agent dependencies
│   └── .env.example                     # Agent environment template
├── research_agent/
│   ├── main.py                          # Research agent implementation
│   ├── requirements.txt                 # Agent dependencies
│   └── .env.example                     # Agent environment template
├── reasoning_agent/
│   ├── main.py                          # Reasoning agent implementation
│   ├── requirements.txt                 # Agent dependencies
│   └── .env.example                     # Agent environment template
├── builder_agent/
│   ├── main.py                          # Builder agent implementation
│   ├── requirements.txt                 # Agent dependencies
│   └── .env.example                     # Agent environment template
└── README.md                            # Agent deployment documentation

================================================================================
                         3. MODULE INDEX
================================================================================

────────────────────────────────────────────────────────────────────────────
BACKEND CORE MODULES
────────────────────────────────────────────────────────────────────────────

File: backend/src/main.py
Purpose: FastAPI application entrypoint with lifespan management
Key Components:
  - FastAPI app initialization
  - CORS middleware configuration
  - Route registration (tasks, skills, evolution, system, websocket)
  - Database initialization on startup
  - TaskManager lifecycle management
  - Audit log generation in test mode
Dependencies:
  Internal: config, db.session, api.routes, utils.logger, utils.audit_logger
  External: fastapi, fastapi.middleware.cors
Execution Role: Application bootstrap and HTTP server configuration

────────────────────────────────────────────────────────────────────────────

File: backend/src/core/agent_orchestrator.py
Purpose: Coordinates multi-agent pipeline execution for task processing
Key Classes:
  - AgentOrchestrator
Methods:
  - run_pipeline(task, session): Executes 4-stage agent pipeline
  - _record_step(session, task, agent_name, input, output): Persists execution
Dependencies:
  Internal: agents.*, ai.llm_service, memory.vector_store, core.event_bus,
            providers.digitalocean.digitalocean_agent_router, db.models
  External: sqlalchemy, time.perf_counter
Execution Role: Central orchestration hub for agent coordination
Pipeline Stages: Planning → Research → Reasoning → Builder
Performance: Tracks execution time per agent with microsecond precision

────────────────────────────────────────────────────────────────────────────

File: backend/src/core/task_manager.py
Purpose: Async task queue manager with background worker
Key Classes:
  - TaskManager
Methods:
  - start(): Launches background worker task
  - stop(): Gracefully shuts down worker
  - create_task(session, title, description): Enqueues new task
  - list_tasks(session): Retrieves all tasks
  - get_task(session, task_id): Retrieves single task
  - run(): Background worker loop processing queue
  - queue_size(): Returns pending task count
  - status(): Returns worker status (running/stopped)
Dependencies:
  Internal: core.agent_orchestrator, core.event_bus, db.models, db.session
  External: asyncio, sqlalchemy, time.perf_counter
Execution Role: Manages async task execution queue with single worker
Concurrency: Single worker processes tasks sequentially
Error Handling: Catches exceptions, marks tasks as failed, continues processing

────────────────────────────────────────────────────────────────────────────

File: backend/src/core/event_bus.py
Purpose: Pub/sub event system for real-time notifications
Key Classes:
  - EventBus
Methods:
  - subscribe(callback): Registers event subscriber
  - unsubscribe(callback): Removes event subscriber
  - publish(event): Broadcasts event to all subscribers
  - subscriber_count(): Returns active subscriber count
Dependencies:
  Internal: None
  External: asyncio
Execution Role: Decouples event producers from consumers
Thread Safety: Uses asyncio.Lock for subscriber list protection
Event Types: task.created, task.progress, task.result, task.error, agent.step

────────────────────────────────────────────────────────────────────────────

File: backend/src/core/evolution_engine.py
Purpose: Self-improving system that generates skills from failed tasks
Key Classes:
  - EvolutionEngine
Methods:
  - evolve(session): Analyzes failures and generates new skill
  - get_status(session): Returns evolution statistics
  - _sanitize_name(name): Cleans skill names for Python modules
  - _failed_tasks_count(session): Counts failed tasks
Dependencies:
  Internal: ai.llm_service, skills.registry, core.event_bus, db.models
  External: sqlalchemy, importlib, re, datetime
Execution Role: Automated skill generation from failure patterns
Algorithm:
  1. Count failed tasks
  2. Generate skill description via LLM
  3. Create Python module with SKILL metadata
  4. Write to generated_skills/ directory
  5. Reload skill registry
  6. Persist to database
  7. Emit evolution.generated event

────────────────────────────────────────────────────────────────────────────
AGENT IMPLEMENTATIONS
────────────────────────────────────────────────────────────────────────────

File: backend/src/agents/base_agent.py
Purpose: Abstract base class for all agent implementations
Key Classes:
  - BaseAgent (ABC)
Attributes:
  - agent_name: Agent identifier
  - llm: LLM service instance
  - skills: Skill executor instance
  - memory: Vector store instance
  - event_bus: Event bus instance
  - digitalocean_router: Optional ADK agent router
Methods:
  - _load_context(task_text, task_id): Retrieves semantic context
  - _remember(task_id, content, memory_type): Stores memory
  - _emit(event_type, payload): Publishes event
  - execute(task): Abstract method for agent execution
Dependencies:
  Internal: ai.llm_service, skills.executor, memory.vector_store,
            core.event_bus, providers.digitalocean
  External: abc
Execution Role: Provides common functionality for all agents

────────────────────────────────────────────────────────────────────────────

File: backend/src/agents/planning_agent.py
Purpose: Breaks down tasks into actionable steps
Key Classes:
  - PlanningAgent(BaseAgent)
Methods:
  - execute(task): Generates task plan
Execution Flow:
  1. Load semantic context from vector store
  2. If real mode: Route to DigitalOcean ADK planner agent
  3. If mock mode: Execute planning skill + LLM refinement
  4. Store result in memory
  5. Emit agent.step event
  6. Return plan structure
Dependencies:
  Internal: agents.base_agent, config.config
  External: json
Input Schema: {"id": str, "title": str, "description": str}
Output Schema: {"agent": "planning", "plan": dict, "llm_refinement": str}

────────────────────────────────────────────────────────────────────────────

File: backend/src/agents/research_agent.py
Purpose: Conducts research and gathers information
Key Classes:
  - ResearchAgent(BaseAgent)
Methods:
  - execute(task): Performs research
Execution Flow:
  1. Load semantic context
  2. If real mode: Route to ADK research agent (with KB retrieval)
  3. If mock mode: Execute web_search skill + LLM synthesis
  4. Store research in memory
  5. Emit agent.step event
  6. Return research notes
Dependencies:
  Internal: agents.base_agent, config.config
  External: json
Input Schema: {"id": str, "title": str, "description": str, "plan": dict}
Output Schema: {"agent": "research", "research": dict, "synthesis": str}
Special Features: Integrates with DigitalOcean Knowledge Base in real mode

────────────────────────────────────────────────────────────────────────────

File: backend/src/agents/reasoning_agent.py
Purpose: Evaluates strategies and provides logical analysis
Key Classes:
  - ReasoningAgent(BaseAgent)
Methods:
  - execute(task): Analyzes plan and provides reasoning
Execution Flow:
  1. Load semantic context
  2. If real mode: Route to ADK reasoning agent
  3. If mock mode: Execute reasoning skill + LLM rationale
  4. Store reasoning in memory
  5. Emit agent.step event
  6. Return analysis
Dependencies:
  Internal: agents.base_agent, config.config
  External: json
Input Schema: {"id": str, "title": str, "plan": dict}
Output Schema: {"agent": "reasoning", "analysis": dict, "rationale": str}

────────────────────────────────────────────────────────────────────────────

File: backend/src/agents/builder_agent.py
Purpose: Generates final solutions based on plan and reasoning
Key Classes:
  - BuilderAgent(BaseAgent)
Methods:
  - execute(task): Produces final solution
Execution Flow:
  1. If real mode: Route to ADK builder agent
  2. If mock mode: Execute coding skill + LLM response
  3. Store build output in memory
  4. Emit agent.step event
  5. Return solution
Dependencies:
  Internal: agents.base_agent, config.config
  External: json
Input Schema: {"id": str, "title": str, "description": str, "plan": dict,
               "reasoning": dict}
Output Schema: {"agent": "builder", "build": dict, "final": str}

────────────────────────────────────────────────────────────────────────────
AI SERVICE LAYER
────────────────────────────────────────────────────────────────────────────

File: backend/src/ai/llm_service.py
Purpose: Unified LLM interface with provider fallback chain
Key Classes:
  - LLMService
Methods:
  - complete(prompt): Single-turn completion
  - chat(messages): Multi-turn conversation
  - stream(messages): Streaming response generator
  - _test_mode_response(messages): Deterministic mock responses
  - _retry_call(func, *args, **kwargs): Retry logic with exponential backoff
  - _extract_text(response): Normalizes provider responses
  - _log_usage(response, provider): Logs token usage
  - _local_complete(messages): Fallback to local model
Dependencies:
  Internal: ai.gradient_client, ai.huggingface_client, config.config
  External: tenacity, transformers (optional)
Execution Role: Abstracts LLM provider differences
Provider Chain: Gradient → HuggingFace → Local (distilgpt2)
Real Mode Behavior: Returns error, forces ADK agent usage
Test Mode: Returns deterministic JSON responses
Retry Strategy: 3 attempts, exponential backoff (1s-8s)

File: backend/src/ai/gradient_client.py
Purpose: DigitalOcean Gradient AI API client
Key Classes:
  - GradientClient
Methods:
  - chat(messages, stream): Chat completion request
  - health(): Provider health check
Dependencies:
  Internal: config.config
  External: httpx
API Endpoint: {GRADIENT_BASE_URL}/chat/completions
Authentication: Bearer token (GRADIENT_API_KEY)
Timeout: 60 seconds
Models Supported: gpt-4.1-mini, llama3-70b, llama3-8b

File: backend/src/ai/huggingface_client.py
Purpose: HuggingFace Inference API client
Key Classes:
  - HuggingFaceClient
Methods:
  - complete(prompt): Text generation request
  - health(): Provider health check
Dependencies:
  Internal: config.config
  External: httpx
API Endpoint: https://api-inference.huggingface.co/models/{model}
Authentication: Bearer token (HUGGINGFACE_API_KEY)
Timeout: 90 seconds
Default Model: HuggingFaceH4/zephyr-7b-beta

────────────────────────────────────────────────────────────────────────────
DIGITALOCEAN PROVIDER INTEGRATION
────────────────────────────────────────────────────────────────────────────

File: backend/src/providers/digitalocean/digitalocean_agent_router.py
Purpose: Routes requests to deployed ADK agents
Key Classes:
  - DigitalOceanAgentRouter
Methods:
  - route_to_agent(agent_name, prompt, context, trace_id, session_id, stream)
  - route_to_agent_stream(agent_name, prompt, context, trace_id, session_id)
  - health_check_all(): Checks all agent endpoints
Agent Mapping:
  - planning: AXON_PLANNER_AGENT_URL
  - research: AXON_RESEARCH_AGENT_URL
  - reasoning: AXON_REASONING_AGENT_URL
  - builder: AXON_BUILDER_AGENT_URL
Dependencies:
  Internal: config.config, providers.digitalocean.digitalocean_agent_client,
            providers.digitalocean.digitalocean_agent_types
Execution Role: Agent discovery and request routing
Headers: X-Trace-ID, X-AXON-SESSION
Streaming: Supports SSE-style streaming responses

File: backend/src/providers/digitalocean/digitalocean_agent_client.py
Purpose: HTTP client for ADK agent communication
Key Classes:
  - DigitalOceanAgentClient
Methods:
  - call_agent(agent_url, request, trace_id, session_id, stream)
  - call_agent_stream(agent_url, request, trace_id, session_id)
  - health_check(agent_url)
  - _get_headers(trace_id, session_id): Builds request headers
  - _retry_call(func, *args, **kwargs): Retry with exponential backoff
Dependencies:
  Internal: config.config, providers.digitalocean.digitalocean_agent_types
  External: httpx, tenacity
Timeout: Configurable via AXON_AGENT_TIMEOUT (default 120s)
Retry Strategy: 3 attempts, exponential backoff (1s-10s)
Authentication: Bearer token (DIGITALOCEAN_API_TOKEN)
Logging: Latency, payload sizes, trace/session IDs

File: backend/src/providers/digitalocean/digitalocean_agent_types.py
Purpose: Type definitions for agent communication
Key Classes:
  - AgentRequest: Input schema (prompt, context)
  - AgentResponse: Output schema (response, metadata, trace_id)
Dependencies:
  External: pydantic

────────────────────────────────────────────────────────────────────────────
MEMORY AND VECTOR STORE
────────────────────────────────────────────────────────────────────────────

File: backend/src/memory/vector_store.py
Purpose: ChromaDB vector database operations
Key Classes:
  - VectorStore
Methods:
  - add_embedding(content, memory_type, task_id, metadata): Stores vector
  - similarity_search(query, limit, task_id): Semantic search
  - retrieve_context(task_prompt, task_id, limit): Context retrieval
Dependencies:
  Internal: config.config, memory.embeddings
  External: chromadb, asyncio
Storage: Persistent client at VECTOR_DB_PATH (./.chroma)
Fallback: EphemeralClient if persistent state corrupted
Collection: "axon_memory"
Metadata Fields: memory_type, task_id, custom metadata
Distance Metric: Default ChromaDB (L2)

File: backend/src/memory/embeddings.py
Purpose: Text embedding generation
Functions:
  - embed(text): Generates embedding vector
  - _load_model(): Loads sentence transformer (cached)
  - _fallback_embedding(text, dim): Deterministic hash-based fallback
Dependencies:
  Internal: config.config
  External: sentence_transformers, hashlib
Model: sentence-transformers/all-MiniLM-L6-v2 (384 dimensions)
Fallback: SHA256-based deterministic vectors if model unavailable
Normalization: Embeddings are normalized

────────────────────────────────────────────────────────────────────────────
SKILLS SYSTEM
────────────────────────────────────────────────────────────────────────────

File: backend/src/skills/registry.py
Purpose: Skill discovery, registration, and management
Key Classes:
  - SkillRegistry
  - SkillDefinition (dataclass)
Methods:
  - discover_skills(): Scans core_skills and generated_skills packages
  - register_dynamic_skill(module_name): Loads skill from module path
  - register_definition(definition): Registers skill definition
  - all(): Returns all registered skills
  - get(name): Retrieves skill by name
  - generated_skills_path(): Returns path to generated_skills directory
  - _iter_modules(package_name): Discovers modules in package
  - _build_definition(module): Extracts skill metadata
Dependencies:
  Internal: None
  External: importlib, pkgutil, inspect, pathlib
Skill Metadata: name, description, parameters, version, execute function
Discovery Packages: src.skills.core_skills, src.skills.generated_skills
Skill Format: SKILL dict + execute() function

File: backend/src/skills/executor.py
Purpose: Executes registered skills with validation
Key Classes:
  - SkillExecutor
Methods:
  - execute(name, payload): Validates and executes skill
  - _validate(expected, payload): Validates required parameters
Dependencies:
  Internal: skills.registry
  External: inspect
Execution: Supports both sync and async skill functions
Validation: Checks required parameters before execution
Output Format: {"skill": name, "version": version, "output": result}

File: backend/src/skills/core_skills/planning.py
Purpose: Task decomposition skill
Metadata:
  name: planning
  description: Break a task into executable steps
  parameters: task (required), max_steps (optional)
  version: 1.0.0
Algorithm: Returns predefined step templates (max 5 steps)
Output: {"task": str, "steps": [{"step": int, "description": str}]}

File: backend/src/skills/core_skills/coding.py
Purpose: Implementation scaffolding generation
Metadata:
  name: coding
  description: Generate implementation scaffolding
  parameters: task (required), plan (optional)
  version: 1.0.0
Output: {"summary": str, "artifacts": [{"type": str, "name": str}]}

File: backend/src/skills/core_skills/reasoning.py
Purpose: Strategy evaluation skill
Metadata:
  name: reasoning
  description: Evaluate options and tradeoffs
  parameters: plan (required), context (optional)
  version: 1.0.0
Algorithm: Calculates confidence score based on plan complexity
Output: {"confidence": float, "risks": [str], "recommendation": str}

File: backend/src/skills/generated_skills/web_search.py
Purpose: Web search placeholder skill
Metadata:
  name: web_search
  description: Synthesize search intent into research notes
  parameters: query (required)
  version: 1.0.0
Note: Placeholder implementation, no live web access
Output: {"query": str, "notes": [str]}

────────────────────────────────────────────────────────────────────────────
DATABASE LAYER
────────────────────────────────────────────────────────────────────────────

File: backend/src/db/models.py
Purpose: SQLAlchemy ORM models
Key Classes:
  - Base: Declarative base
  - TimestampMixin: id, created_at, updated_at fields
  - Task: Task entity
  - Skill: Skill entity
  - AgentExecution: Agent execution record
  - Artifact: File artifact metadata
  - MemoryRecord: Memory/embedding reference
Dependencies:
  External: sqlalchemy
ID Strategy: UUID4 strings (36 chars)
Timestamps: Timezone-aware, auto-managed
Relationships: Cascade deletes configured

File: backend/src/db/session.py
Purpose: Database session management
Functions:
  - init_db(): Creates all tables
  - close_db(): Disposes engine
  - get_db_session(): Async session generator (FastAPI dependency)
Dependencies:
  Internal: config.config, db.models
  External: sqlalchemy.ext.asyncio
Connection Pool: 10 base, 20 overflow, 30s timeout (PostgreSQL)
Session Config: No autoflush, no expire on commit
Transaction: Auto-commit on success, rollback on exception

────────────────────────────────────────────────────────────────────────────
API LAYER
────────────────────────────────────────────────────────────────────────────

File: backend/src/api/routes/tasks.py
Purpose: Task management endpoints
Endpoints:
  - GET /tasks/: List all tasks
  - GET /tasks/{task_id}: Get task by ID
  - POST /tasks/: Create new task
Dependencies: require_api_key, rate_limit_hook
Response Models: TaskListResponse, TaskResponse
Status Codes: 200 (GET), 201 (POST), 404 (not found)

File: backend/src/api/routes/skills.py
Purpose: Skill listing endpoints
Endpoints:
  - GET /skills/: List all registered skills
Dependencies: require_api_key, rate_limit_hook
Response Models: SkillListResponse

File: backend/src/api/routes/evolution.py
Purpose: Evolution engine endpoints
Endpoints:
  - GET /evolution/: Get evolution status
  - POST /evolution/run: Trigger evolution
Dependencies: require_api_key, rate_limit_hook
Response Models: EvolutionStatus

File: backend/src/api/routes/system.py
Purpose: System health and status endpoints
Endpoints:
  - GET /system/: System status check
Dependencies: require_api_key, rate_limit_hook
Response Models: SystemStatusResponse
Health Checks: Database, vector store, skills, agents, event bus, task queue

File: backend/src/api/websocket/event_stream.py
Purpose: Real-time event streaming via WebSocket
Endpoint: /ws/events
Protocol: WebSocket
Events: All EventBus events streamed to connected clients
Connection: Auto-subscribes on connect, unsubscribes on disconnect
Initial Message: {"event": "connected", "message": "..."}

File: backend/src/api/middleware/logging_middleware.py
Purpose: HTTP request logging middleware
Logs: method, path, status_code, duration_ms
Format: Structured JSON via Loguru

────────────────────────────────────────────────────────────────────────────
CONFIGURATION AND DEPENDENCIES
────────────────────────────────────────────────────────────────────────────

File: backend/src/config/config.py
Purpose: Application settings and environment variables
Key Classes:
  - Settings (Pydantic BaseSettings)
Functions:
  - get_settings(): Cached settings singleton
Configuration Fields:
  - app_name: Application name (default: "AXON")
  - env: Environment (default: "development")
  - test_mode: Enable test mode (default: True)
  - api_key: API authentication key
  - gradient_api_key: DigitalOcean Gradient API key
  - gradient_model: Model name (default: "gpt-4.1-mini")
  - gradient_base_url: Gradient API endpoint
  - huggingface_api_key: HuggingFace API key
  - huggingface_model: HF model name
  - database_url: PostgreSQL connection string
  - embedding_model: Sentence transformer model
  - vector_db_path: ChromaDB storage path
  - cors_origins: CORS allowed origins
  - request_rate_limit_per_minute: Rate limit (default: 120)
  - axon_mode: Operating mode (mock/real)
  - digitalocean_api_token: DO API token
  - gradient_model_access_key: Gradient model access key
  - digitalocean_kb_uuid: Knowledge base UUID
  - axon_agent_timeout: Agent request timeout (default: 120s)
  - axon_planner_agent_url: Planner agent URL
  - axon_research_agent_url: Research agent URL
  - axon_reasoning_agent_url: Reasoning agent URL
  - axon_builder_agent_url: Builder agent URL
Dependencies:
  External: pydantic-settings, python-dotenv
Loading: Reads from .env file via python-dotenv

File: backend/src/config/dependencies.py
Purpose: FastAPI dependency injection and singleton management
Singletons:
  - _event_bus: EventBus instance
  - _skill_registry: SkillRegistry instance
  - _skill_executor: SkillExecutor instance
  - _vector_store: VectorStore instance
  - _llm_service: LLMService instance
  - _orchestrator: AgentOrchestrator instance
  - _task_manager: TaskManager instance
  - _evolution_engine: EvolutionEngine instance
  - _rate_state: Rate limiting state dict
Dependency Functions:
  - get_app_settings(): Returns Settings
  - get_event_bus(): Returns EventBus
  - get_skill_registry(): Returns SkillRegistry
  - get_llm_service(): Returns LLMService
  - get_vector_store(): Returns VectorStore
  - get_task_manager(): Returns TaskManager
  - get_orchestrator(): Returns AgentOrchestrator
  - get_evolution_engine(): Returns EvolutionEngine
  - get_task_service(session): Returns TaskService
  - get_skill_service(session): Returns SkillService
  - get_evolution_service(session): Returns EvolutionService
  - require_api_key(x_api_key, app_settings): API key validation
  - rate_limit_hook(request, app_settings): Rate limiting
Rate Limiting: Per-IP, sliding window (60s), configurable limit
API Key: Optional, validates X-API-Key header

================================================================================
                    4. AGENT SYSTEM ANALYSIS
================================================================================

────────────────────────────────────────────────────────────────────────────
STANDALONE ADK AGENTS (agents/ directory)
────────────────────────────────────────────────────────────────────────────

PLANNER AGENT
Location: agents/planner_agent/main.py
Purpose: Task decomposition and planning
Framework: LangGraph + Gradient SDK
Model: openai-gpt-oss-120b (configurable)
Inference Endpoint: https://inference.do-ai.run

State Schema:
  - prompt: str (task description)
  - context: dict (additional context)
  - plan: dict (generated plan)

Workflow Nodes:
  - analyze_task: Main planning node
    * Constructs system prompt for planning
    * Calls Gradient chat completion
    * Parses JSON response or wraps raw text
    * Updates state with plan

Graph Structure:
  Entry → analyze_task → END

Input Schema:
  {
    "prompt": "Task description",
    "context": {"task_id": "...", ...}
  }

Output Schema:
  {
    "response": "{\"steps\": [...]}",  # JSON string
    "metadata": {
      "agent": "planner",
      "version": "1.0.0"
    }
  }

Dependencies:
  - gradient-adk: Agent deployment framework
  - gradient: Gradient SDK for inference
  - langgraph: State machine framework

Environment Variables:
  - GRADIENT_MODEL_ACCESS_KEY: Model access key (required)
  - GRADIENT_MODEL: Model name (default: openai-gpt-oss-120b)
  - DIGITALOCEAN_API_TOKEN: DO API token
  - DIGITALOCEAN_KB_UUID: Knowledge base UUID

Deployment:
  Command: gradient-adk deploy
  Entrypoint: @entrypoint decorator on main()
  Runtime: DigitalOcean serverless infrastructure

RESEARCH AGENT
Location: agents/research_agent/main.py
Purpose: Information gathering and research
Framework: LangGraph + Gradient SDK
Model: openai-gpt-oss-120b (configurable)
Inference Endpoint: https://inference.do-ai.run

State Schema:
  - prompt: str (research query)
  - context: dict (additional context)
  - research: dict (research results)

Workflow Nodes:
  - conduct_research: Main research node
    * Retrieves documents from Knowledge Base (if configured)
    * Augments prompt with KB context
    * Calls Gradient chat completion
    * Structures research output

Graph Structure:
  Entry → conduct_research → END

Knowledge Base Integration:
  - Uses client.retrieve.documents() for semantic search
  - Retrieves top 5 documents matching query
  - Appends document content to prompt context
  - Gracefully handles KB unavailability

Input Schema:
  {
    "prompt": "Research topic",
    "context": {"task_id": "...", "plan": {...}}
  }

Output Schema:
  {
    "response": "{\"notes\": \"...\", \"sources\": \"...\"}",
    "metadata": {
      "agent": "research",
      "version": "1.0.0"
    }
  }

Special Features:
  - Knowledge Base retrieval (optional)
  - Document context augmentation
  - Fallback to pure LLM if KB unavailable

Dependencies: Same as Planner Agent
Environment Variables: Same as Planner Agent + DIGITALOCEAN_KB_UUID

REASONING AGENT
Location: agents/reasoning_agent/main.py
Purpose: Strategy evaluation and logical analysis
Framework: LangGraph + Gradient SDK
Model: openai-gpt-oss-120b (configurable)
Inference Endpoint: https://inference.do-ai.run

State Schema:
  - prompt: str (evaluation request)
  - context: dict (plan and context)
  - reasoning: dict (analysis results)

Workflow Nodes:
  - evaluate_strategy: Main reasoning node
    * Constructs evaluation prompt
    * Calls Gradient chat completion with lower temperature (0.3)
    * Structures reasoning output with confidence score

Graph Structure:
  Entry → evaluate_strategy → END

Input Schema:
  {
    "prompt": "Strategy to evaluate",
    "context": {"task_id": "...", "plan": {...}}
  }

Output Schema:
  {
    "response": "{\"analysis\": \"...\", \"rationale\": \"...\"}",
    "metadata": {
      "agent": "reasoning",
      "version": "1.0.0"
    }
  }

Temperature: 0.3 (lower for more deterministic reasoning)
Dependencies: Same as Planner Agent
Environment Variables: Same as Planner Agent

BUILDER AGENT
Location: agents/builder_agent/main.py
Purpose: Solution generation and implementation
Framework: LangGraph + Gradient SDK
Model: openai-gpt-oss-120b (configurable)
Inference Endpoint: https://inference.do-ai.run

State Schema:
  - prompt: str (build request)
  - context: dict (plan, reasoning, etc.)
  - build: dict (solution output)

Workflow Nodes:
  - generate_solution: Main building node
    * Constructs solution prompt with all context
    * Calls Gradient chat completion
    * Structures build output

Graph Structure:
  Entry → generate_solution → END

Input Schema:
  {
    "prompt": "Solution requirements",
    "context": {"task_id": "...", "plan": {...}, "reasoning": {...}}
  }

Output Schema:
  {
    "response": "{\"solution\": \"...\", \"implementation\": \"...\"}",
    "metadata": {
      "agent": "builder",
      "version": "1.0.0"
    }
  }

Temperature: 0.7 (balanced creativity and consistency)
Dependencies: Same as Planner Agent
Environment Variables: Same as Planner Agent

────────────────────────────────────────────────────────────────────────────
AGENT DEPLOYMENT WORKFLOW
────────────────────────────────────────────────────────────────────────────

Prerequisites:
  1. DigitalOcean account with Gradient AI access
  2. gradient-adk CLI installed (pip install gradient-adk)
  3. API token with agent deployment permissions
  4. Model access key for Gradient inference

Deployment Steps (per agent):
  1. Navigate to agent directory (e.g., agents/planner_agent)
  2. Copy .env.example to .env
  3. Configure environment variables:
     - GRADIENT_MODEL_ACCESS_KEY
     - GRADIENT_MODEL (optional, defaults to openai-gpt-oss-120b)
     - DIGITALOCEAN_API_TOKEN
     - DIGITALOCEAN_KB_UUID (optional, for research agent)
  4. Run: gradient-adk deploy
  5. Note the deployed agent URL (https://agents.do-ai.run/<agent-id>)
  6. Update backend .env with agent URLs:
     - AXON_PLANNER_AGENT_URL
     - AXON_RESEARCH_AGENT_URL
     - AXON_REASONING_AGENT_URL
     - AXON_BUILDER_AGENT_URL

Local Testing:
  Command: gradient-adk run --input '{"prompt": "test", "context": {}}'
  Purpose: Test agent locally before deployment

Monitoring:
  - View logs: gradient-adk logs <agent-id>
  - Check status: gradient-adk status <agent-id>
  - Health check: GET https://agents.do-ai.run/<agent-id>/health

Agent Endpoints:
  - POST /run: Execute agent with input
  - GET /health: Health check endpoint

Request Format:
  {
    "prompt": "User prompt",
    "context": {"key": "value"},
    "stream": false
  }

Response Format:
  {
    "response": "Agent response (JSON string)",
    "metadata": {"agent": "name", "version": "1.0.0"}
  }

================================================================================
                    5. RUNTIME EXECUTION FLOW
================================================================================

────────────────────────────────────────────────────────────────────────────
COMPLETE REQUEST FLOW (REAL MODE)
────────────────────────────────────────────────────────────────────────────

1. USER REQUEST
   Client → POST /tasks/
   Payload: {"title": "Build API", "description": "Create REST endpoints"}
   Headers: X-API-Key (if configured)

2. API LAYER
   ├─ LoggingMiddleware: Logs request start time
   ├─ require_api_key: Validates API key
   ├─ rate_limit_hook: Checks rate limit (per-IP, 60s window)
   └─ TaskController.create_task()

3. SERVICE LAYER
   TaskService.create_task()
   └─ TaskManager.create_task(session, title, description)

4. TASK MANAGER
   ├─ Creates Task entity (status: "queued")
   ├─ Persists to database
   ├─ Enqueues task ID to asyncio.Queue
   ├─ Publishes event: {"event": "task.created", "task_id": "...", ...}
   └─ Returns Task to client (201 Created)

5. BACKGROUND WORKER
   TaskManager.run() (async loop)
   ├─ Dequeues task ID
   ├─ Loads Task from database
   ├─ Updates status to "running"
   ├─ Publishes event: {"event": "task.progress", "status": "running"}
   └─ Calls AgentOrchestrator.run_pipeline(task, session)

6. AGENT ORCHESTRATION
   AgentOrchestrator.run_pipeline()
   ├─ Generates trace_id (task.id)
   └─ Executes 4-stage pipeline:

   STAGE 1: PLANNING
   ├─ Loads context from VectorStore (semantic search)
   ├─ Builds input: {"id": task_id, "title": "...", "description": "..."}
   ├─ PlanningAgent.execute(input)
   │  ├─ Checks axon_mode == "real"
   │  ├─ Builds prompt with context
   │  ├─ DigitalOceanAgentRouter.route_to_agent("planning", prompt, ...)
   │  │  ├─ Looks up AXON_PLANNER_AGENT_URL
   │  │  ├─ DigitalOceanAgentClient.call_agent()
   │  │  │  ├─ Builds headers: Authorization, X-Trace-ID, X-AXON-SESSION
   │  │  │  ├─ POST to agent URL with retry (3 attempts, exp backoff)
   │  │  │  ├─ Logs: latency, payload sizes, trace_id
   │  │  │  └─ Returns AgentResponse
   │  │  └─ Parses JSON response
   │  ├─ Stores result in VectorStore
   │  ├─ Publishes event: {"event": "agent.step", "agent": "planning", ...}
   │  └─ Returns {"agent": "planning", "plan": {...}, ...}
   ├─ Records AgentExecution in database
   └─ Records MemoryRecord in database

   STAGE 2: RESEARCH
   ├─ Loads context from VectorStore
   ├─ Builds input with plan from Stage 1
   ├─ ResearchAgent.execute(input)
   │  ├─ Routes to AXON_RESEARCH_AGENT_URL
   │  │  ├─ ADK agent retrieves from Knowledge Base (if configured)
   │  │  ├─ Augments prompt with KB documents
   │  │  └─ Calls Gradient inference
   │  ├─ Stores research in VectorStore
   │  └─ Returns {"agent": "research", "research": {...}, ...}
   ├─ Records AgentExecution
   └─ Records MemoryRecord

   STAGE 3: REASONING
   ├─ Loads context from VectorStore
   ├─ Builds input with plan and research
   ├─ ReasoningAgent.execute(input)
   │  ├─ Routes to AXON_REASONING_AGENT_URL
   │  ├─ ADK agent evaluates with temperature=0.3
   │  └─ Returns {"agent": "reasoning", "analysis": {...}, ...}
   ├─ Records AgentExecution
   └─ Records MemoryRecord

   STAGE 4: BUILDER
   ├─ Builds input with plan, research, reasoning
   ├─ BuilderAgent.execute(input)
   │  ├─ Routes to AXON_BUILDER_AGENT_URL
   │  ├─ ADK agent generates solution
   │  └─ Returns {"agent": "builder", "build": {...}, ...}
   ├─ Records AgentExecution
   └─ Records MemoryRecord

7. RESULT AGGREGATION
   ├─ Aggregates all agent outputs
   ├─ Updates Task.status = "completed"
   ├─ Updates Task.result = str(aggregated)
   ├─ Commits to database
   └─ Publishes event: {"event": "task.result", "status": "completed", ...}

8. WEBSOCKET STREAMING
   EventBus → WebSocket clients
   ├─ All events broadcast to connected clients
   └─ Real-time progress updates

9. CLIENT POLLING
   Client → GET /tasks/{task_id}
   └─ Returns Task with status and result

────────────────────────────────────────────────────────────────────────────
MOCK MODE FLOW (TEST/DEVELOPMENT)
────────────────────────────────────────────────────────────────────────────

Differences from Real Mode:
  - Agent.execute() uses local skills instead of ADK agents
  - LLMService returns deterministic mock responses
  - No external API calls
  - Faster execution (no network latency)

Example (Planning Agent):
  1. Execute planning skill (returns predefined steps)
  2. Call LLMService.complete() for refinement
  3. LLMService detects test_mode=True
  4. Returns deterministic JSON response
  5. Stores and returns result

Benefits:
  - No API keys required
  - Deterministic testing
  - Offline development
  - Fast iteration

================================================================================
                       6. DATABASE SCHEMA
================================================================================

────────────────────────────────────────────────────────────────────────────
TABLE: tasks
────────────────────────────────────────────────────────────────────────────
Purpose: Stores user-submitted tasks and execution results

Columns:
  - id: VARCHAR(36), PRIMARY KEY, UUID4
  - title: VARCHAR(255), NOT NULL
  - description: TEXT, NOT NULL, DEFAULT ''
  - status: VARCHAR(50), NOT NULL, DEFAULT 'queued'
    Values: queued, running, completed, failed
  - result: TEXT, NOT NULL, DEFAULT ''
  - created_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()
  - updated_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()

Indexes:
  - PRIMARY KEY on id

Relationships:
  - One-to-Many with agent_executions (CASCADE DELETE)
  - One-to-Many with artifacts (CASCADE DELETE)
  - One-to-Many with memory_records (SET NULL on delete)

Lifecycle:
  1. Created with status='queued'
  2. Updated to status='running' when worker picks up
  3. Updated to status='completed' or 'failed' when done
  4. Result field populated with aggregated agent outputs

────────────────────────────────────────────────────────────────────────────
TABLE: skills
────────────────────────────────────────────────────────────────────────────
Purpose: Persists generated and registered skills

Columns:
  - id: VARCHAR(36), PRIMARY KEY, UUID4
  - name: VARCHAR(255), NOT NULL, UNIQUE
  - description: TEXT, NOT NULL, DEFAULT ''
  - source_code: TEXT, NOT NULL, DEFAULT ''
  - version: VARCHAR(50), NOT NULL, DEFAULT '1.0.0'
  - created_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()
  - updated_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()

Indexes:
  - PRIMARY KEY on id
  - UNIQUE INDEX on name

Relationships: None

Usage:
  - Evolution engine writes generated skills here
  - SkillService reads for listing
  - Source code stored for audit/versioning

────────────────────────────────────────────────────────────────────────────
TABLE: agent_executions
────────────────────────────────────────────────────────────────────────────
Purpose: Records each agent execution in the pipeline

Columns:
  - id: VARCHAR(36), PRIMARY KEY, UUID4
  - task_id: VARCHAR(36), NOT NULL, FOREIGN KEY → tasks.id
  - agent_name: VARCHAR(100), NOT NULL
    Values: planning, research, reasoning, builder
  - status: VARCHAR(50), NOT NULL, DEFAULT 'started'
    Values: started, completed, failed
  - input_payload: JSON, NOT NULL, DEFAULT {}
  - output_payload: JSON, NOT NULL, DEFAULT {}
  - created_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()
  - updated_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()

Indexes:
  - PRIMARY KEY on id
  - INDEX on task_id

Relationships:
  - Many-to-One with tasks (CASCADE DELETE)

Usage:
  - AgentOrchestrator records each stage execution
  - Stores full input/output for debugging
  - Enables pipeline replay and analysis

────────────────────────────────────────────────────────────────────────────
TABLE: artifacts
────────────────────────────────────────────────────────────────────────────
Purpose: Metadata for file artifacts generated by tasks

Columns:
  - id: VARCHAR(36), PRIMARY KEY, UUID4
  - task_id: VARCHAR(36), NOT NULL, FOREIGN KEY → tasks.id
  - name: VARCHAR(255), NOT NULL
  - content_type: VARCHAR(100), NOT NULL, DEFAULT 'text/plain'
  - storage_path: TEXT, NOT NULL, DEFAULT ''
  - metadata_json: JSON, NOT NULL, DEFAULT {}
  - created_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()
  - updated_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()

Indexes:
  - PRIMARY KEY on id
  - INDEX on task_id

Relationships:
  - Many-to-One with tasks (CASCADE DELETE)

Usage:
  - ArtifactStore writes metadata here
  - Files stored on filesystem at storage_path
  - Supports code files, documents, images, etc.

────────────────────────────────────────────────────────────────────────────
TABLE: memory_records
────────────────────────────────────────────────────────────────────────────
Purpose: References to vector embeddings and semantic memory

Columns:
  - id: VARCHAR(36), PRIMARY KEY, UUID4
  - task_id: VARCHAR(36), NULLABLE, FOREIGN KEY → tasks.id
  - memory_type: VARCHAR(50), NOT NULL
    Values: agent_thought, research, reasoning, build_output, agent_output
  - content: TEXT, NOT NULL
  - embedding_ref: VARCHAR(255), NOT NULL, DEFAULT ''
  - metadata_json: JSON, NOT NULL, DEFAULT {}
  - created_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()
  - updated_at: TIMESTAMP WITH TIME ZONE, NOT NULL, DEFAULT now()

Indexes:
  - PRIMARY KEY on id
  - INDEX on task_id

Relationships:
  - Many-to-One with tasks (SET NULL on delete)

Usage:
  - Agents store memories via BaseAgent._remember()
  - Content also stored in ChromaDB vector store
  - embedding_ref links to ChromaDB document ID
  - Enables semantic search and context retrieval

────────────────────────────────────────────────────────────────────────────
MIGRATION HISTORY
────────────────────────────────────────────────────────────────────────────

Version: 0001_initial
Date: 2026-03-12
Description: Initial schema creation
Operations:
  - Created all 5 tables
  - Created indexes
  - Created foreign key constraints
  - Set up cascade behaviors

Migration Tool: Alembic
Configuration: backend/alembic.ini
Versions Directory: backend/alembic/versions/

================================================================================
                    7. CONFIGURATION ANALYSIS
================================================================================

────────────────────────────────────────────────────────────────────────────
BACKEND CONFIGURATION (backend/.env.example)
────────────────────────────────────────────────────────────────────────────

AUTHENTICATION & SECURITY:
  API_KEY=
    Purpose: Optional API key for endpoint authentication
    Usage: Validated by require_api_key dependency
    Default: Empty (authentication disabled)

LLM PROVIDERS:
  GRADIENT_API_KEY=
    Purpose: DigitalOcean Gradient API authentication
    Usage: GradientClient for direct LLM calls (mock mode only)
    Default: Empty

  GRADIENT_MODEL=gpt-4.1-mini
    Purpose: Model name for Gradient inference
    Usage: LLMService model selection
    Default: gpt-4.1-mini
    Options: gpt-4.1-mini, llama3-70b, llama3-8b

  GRADIENT_BASE_URL=https://api.digitalocean.com/v2/ai
    Purpose: Gradient API endpoint
    Usage: GradientClient base URL
    Default: https://api.digitalocean.com/v2/ai

  HUGGINGFACE_API_KEY=
    Purpose: HuggingFace Inference API authentication
    Usage: HuggingFaceClient fallback provider
    Default: Empty

  HUGGINGFACE_MODEL=HuggingFaceH4/zephyr-7b-beta
    Purpose: HuggingFace model name
    Usage: HuggingFaceClient model selection
    Default: HuggingFaceH4/zephyr-7b-beta

DATABASE:
  DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/axon
    Purpose: PostgreSQL connection string
    Usage: SQLAlchemy async engine
    Default: Local PostgreSQL instance
    Format: postgresql+asyncpg://user:pass@host:port/dbname

VECTOR STORE:
  EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
    Purpose: Sentence transformer model for embeddings
    Usage: Embeddings generation
    Default: all-MiniLM-L6-v2 (384 dimensions)

  VECTOR_DB_PATH=./.chroma
    Purpose: ChromaDB persistence directory
    Usage: VectorStore storage location
    Default: ./.chroma (relative to backend/)

RATE LIMITING:
  RATE_LIMIT_PER_MIN=120
    Purpose: Requests per minute per IP
    Usage: rate_limit_hook middleware
    Default: 120 requests/minute
    Value: 0 disables rate limiting

CORS:
  CORS_ORIGINS=["*"]
    Purpose: Allowed CORS origins
    Usage: CORSMiddleware configuration
    Default: ["*"] (all origins)
    Format: JSON array of strings

OPERATING MODE:
  AXON_MODE=mock
    Purpose: Switches between mock and real agent execution
    Usage: Agent routing decision
    Values: mock (local skills), real (ADK agents)
    Default: mock

DIGITALOCEAN INTEGRATION:
  DIGITALOCEAN_API_TOKEN=
    Purpose: DigitalOcean API authentication
    Usage: DigitalOceanAgentClient authorization
    Default: Empty
    Required: For real mode

  GRADIENT_MODEL_ACCESS_KEY=
    Purpose: Gradient model access key
    Usage: ADK agents use for inference
    Default: Empty
    Required: For real mode

  DIGITALOCEAN_KB_UUID=
    Purpose: Knowledge Base UUID
    Usage: Research agent document retrieval
    Default: Empty
    Optional: Enhances research agent

AGENT CONFIGURATION:
  AXON_AGENT_TIMEOUT=120
    Purpose: Agent request timeout in seconds
    Usage: DigitalOceanAgentClient timeout
    Default: 120 seconds

  AXON_PLANNER_AGENT_URL=
    Purpose: Deployed planner agent endpoint
    Usage: DigitalOceanAgentRouter routing
    Default: Empty
    Format: https://agents.do-ai.run/<agent-id>
    Required: For real mode

  AXON_RESEARCH_AGENT_URL=
    Purpose: Deployed research agent endpoint
    Usage: DigitalOceanAgentRouter routing
    Default: Empty
    Required: For real mode

  AXON_REASONING_AGENT_URL=
    Purpose: Deployed reasoning agent endpoint
    Usage: DigitalOceanAgentRouter routing
    Default: Empty
    Required: For real mode

  AXON_BUILDER_AGENT_URL=
    Purpose: Deployed builder agent endpoint
    Usage: DigitalOceanAgentRouter routing
    Default: Empty
    Required: For real mode

────────────────────────────────────────────────────────────────────────────
AGENT CONFIGURATION (agents/*/.env.example)
────────────────────────────────────────────────────────────────────────────

All agents share the same configuration:

  GRADIENT_MODEL_ACCESS_KEY=
    Purpose: Gradient inference authentication
    Usage: AsyncGradient client initialization
    Default: Empty
    Required: Yes

  GRADIENT_MODEL=openai-gpt-oss-120b
    Purpose: Model name for inference
    Usage: chat.completions.create() model parameter
    Default: openai-gpt-oss-120b
    Options: openai-gpt-oss-120b, llama3-70b-instruct, llama3-8b-instruct

  DIGITALOCEAN_API_TOKEN=
    Purpose: DigitalOcean API authentication
    Usage: Agent deployment and management
    Default: Empty
    Required: For deployment

  DIGITALOCEAN_KB_UUID=
    Purpose: Knowledge Base UUID
    Usage: Research agent only (document retrieval)
    Default: Empty
    Optional: Enhances research capabilities

────────────────────────────────────────────────────────────────────────────
DEPENDENCY CONFIGURATION (backend/pyproject.toml)
────────────────────────────────────────────────────────────────────────────

Project Metadata:
  name: axon-backend
  version: 0.1.0
  description: AXON FastAPI backend
  requires-python: >=3.11

Core Dependencies:
  - fastapi: Web framework
  - uvicorn: ASGI server
  - sqlalchemy>=2: ORM and database toolkit
  - asyncpg: PostgreSQL async driver
  - alembic: Database migrations
  - python-dotenv: Environment variable loading
  - pydantic-settings: Settings management
  - httpx: Async HTTP client
  - tenacity: Retry logic
  - orjson: Fast JSON serialization
  - loguru: Structured logging
  - chromadb: Vector database
  - sentence-transformers: Embedding generation

Development Dependencies:
  - pytest: Testing framework
  - pytest-asyncio: Async test support
  - mypy: Static type checking
  - ruff: Linting and formatting

Package Management: uv (fast Python package installer)

────────────────────────────────────────────────────────────────────────────
AGENT DEPENDENCIES (agents/*/requirements.txt)
────────────────────────────────────────────────────────────────────────────

All agents share the same dependencies:
  - gradient-adk: DigitalOcean Agent Development Kit
  - gradient: Gradient SDK for inference
  - langgraph: State machine framework for agents

Installation: pip install -r requirements.txt

================================================================================
                    8. PROVIDER INTEGRATIONS
================================================================================

────────────────────────────────────────────────────────────────────────────
DIGITALOCEAN GRADIENT AI PLATFORM
────────────────────────────────────────────────────────────────────────────

Integration Points:
  1. Gradient Inference API (backend LLM service)
  2. ADK Agent Deployment (standalone agents)
  3. Knowledge Base (research agent)

GRADIENT INFERENCE API:
  Endpoint: https://api.digitalocean.com/v2/ai/chat/completions
  Authentication: Bearer token (GRADIENT_API_KEY)
  Protocol: OpenAI-compatible chat completions
  Models:
    - gpt-4.1-mini: Fast, cost-effective
    - llama3-70b: High capability
    - llama3-8b: Lightweight
  Request Format:
    {
      "model": "gpt-4.1-mini",
      "messages": [{"role": "user", "content": "..."}],
      "stream": false
    }
  Response Format:
    {
      "choices": [{"message": {"content": "..."}}],
      "usage": {"prompt_tokens": 10, "completion_tokens": 20}
    }
  Timeout: 60 seconds
  Retry Strategy: 3 attempts, exponential backoff (1s-8s)
  Error Handling: Falls back to HuggingFace or local model
  Usage Logging: Logs prompt_tokens, completion_tokens, total_tokens

ADK AGENT DEPLOYMENT:
  Platform: DigitalOcean serverless infrastructure
  Deployment Tool: gradient-adk CLI
  Agent Runtime: Python 3.11+
  Inference Endpoint: https://inference.do-ai.run
  Agent Endpoints: https://agents.do-ai.run/<agent-id>
  Authentication: Bearer token (DIGITALOCEAN_API_TOKEN)
  Model Access: GRADIENT_MODEL_ACCESS_KEY
  Request Headers:
    - Authorization: Bearer <token>
    - X-Trace-ID: <trace_id>
    - X-AXON-SESSION: <session_id>
  Request Format:
    {
      "prompt": "User prompt",
      "context": {"key": "value"},
      "stream": false
    }
  Response Format:
    {
      "response": "JSON string",
      "metadata": {"agent": "name", "version": "1.0.0"}
    }
  Timeout: Configurable (default 120s)
  Retry Strategy: 3 attempts, exponential backoff (1s-10s)
  Health Check: GET /health endpoint
  Monitoring: gradient-adk logs/status commands

KNOWLEDGE BASE INTEGRATION:
  Purpose: Document retrieval for research agent
  Configuration: DIGITALOCEAN_KB_UUID
  API: client.retrieve.documents()
  Parameters:
    - knowledge_base_id: KB UUID
    - query: Search query
    - limit: Number of documents (default 5)
  Response: List of documents with content
  Usage: Research agent augments prompts with retrieved documents
  Fallback: Gracefully handles KB unavailability
  Error Handling: Continues without KB if retrieval fails

────────────────────────────────────────────────────────────────────────────
HUGGINGFACE INFERENCE API
────────────────────────────────────────────────────────────────────────────

Integration: Fallback LLM provider (mock mode only)
Endpoint: https://api-inference.huggingface.co/models/{model}
Authentication: Bearer token (HUGGINGFACE_API_KEY)
Default Model: HuggingFaceH4/zephyr-7b-beta
Request Format:
  {
    "inputs": "Prompt text",
    "parameters": {
      "max_new_tokens": 512,
      "return_full_text": false
    }
  }
Response Format:
  [{"generated_text": "..."}]
Timeout: 90 seconds
Retry Strategy: 3 attempts, exponential backoff (1s-8s)
Error Handling: Falls back to local model
Usage: Secondary fallback after Gradient

────────────────────────────────────────────────────────────────────────────
CHROMADB VECTOR DATABASE
────────────────────────────────────────────────────────────────────────────

Integration: Semantic memory and context retrieval
Client Type: PersistentClient (with EphemeralClient fallback)
Storage Path: VECTOR_DB_PATH (./.chroma)
Collection: "axon_memory"
Embedding Model: sentence-transformers/all-MiniLM-L6-v2
Embedding Dimensions: 384
Distance Metric: L2 (default)
Operations:
  - add(): Store document with embedding
  - query(): Semantic similarity search
  - count(): Get document count
Metadata Fields:
  - memory_type: Type of memory (agent_thought, research, etc.)
  - task_id: Associated task ID
  - custom: Additional metadata
Async Support: Uses asyncio.to_thread() for blocking operations
Error Handling: Falls back to EphemeralClient if persistence fails

────────────────────────────────────────────────────────────────────────────
POSTGRESQL DATABASE
────────────────────────────────────────────────────────────────────────────

Integration: Primary data persistence
Driver: asyncpg (async PostgreSQL driver)
Connection String: DATABASE_URL
Connection Pool:
  - pool_size: 10
  - max_overflow: 20
  - pool_timeout: 30 seconds
  - pool_pre_ping: True (connection health check)
ORM: SQLAlchemy 2.0 (async)
Session Management:
  - expire_on_commit: False
  - autoflush: False
  - Auto-commit on success
  - Auto-rollback on exception
Migration Tool: Alembic
Schema: 5 tables (tasks, skills, agent_executions, artifacts, memory_records)
Indexes: Primary keys, foreign keys, unique constraints
Cascade Behaviors: DELETE CASCADE, SET NULL

────────────────────────────────────────────────────────────────────────────
SENTENCE TRANSFORMERS
────────────────────────────────────────────────────────────────────────────

Integration: Text embedding generation
Model: sentence-transformers/all-MiniLM-L6-v2
Model Size: ~80MB
Embedding Dimensions: 384
Normalization: Embeddings are normalized
Loading: Lazy loading with @lru_cache
Fallback: SHA256-based deterministic embeddings if model unavailable
Performance: Fast inference (~10ms per embedding)
Usage: VectorStore.add_embedding() and similarity_search()

================================================================================
                    9. OBSERVABILITY & EVENT SYSTEM
================================================================================

────────────────────────────────────────────────────────────────────────────
LOGGING ARCHITECTURE
────────────────────────────────────────────────────────────────────────────

Framework: Loguru
Format: Structured JSON
Output: stdout
Level: INFO (configurable)
Configuration: utils/logger.py

Log Structure:
  {
    "text": "Log message",
    "record": {
      "time": {"timestamp": "2026-03-14T..."},
      "level": {"name": "INFO"},
      "message": "...",
      "extra": {
        "module": "src.core.agent_orchestrator",
        "task_id": "...",
        "agent_name": "...",
        "execution_time": 1.234567,
        ...
      }
    }
  }

Logger Binding:
  - Each module gets logger with module name
  - Example: logger = get_logger(__name__)
  - Enables filtering by module

Key Log Events:
  - http_request: HTTP request/response
    Fields: method, path, status_code, duration_ms
  
  - task.created: Task created
    Fields: task_id, title, execution_time
  
  - task.started: Task execution started
    Fields: task_id, execution_time
  
  - task.completed: Task execution completed
    Fields: task_id, execution_time
  
  - task_failed: Task execution failed
    Fields: task_id, error
  
  - orchestrator_start: Pipeline started
    Fields: task_id, title
  
  - orchestrator_complete: Pipeline completed
    Fields: task_id
  
  - agent.{name}.started: Agent stage started
    Fields: task_id, agent_name, execution_time
  
  - agent.{name}.completed: Agent stage completed
    Fields: task_id, agent_name, execution_time
  
  - llm_call: LLM API call
    Fields: provider, prompt_tokens, completion_tokens, total_tokens
  
  - routing_to_agent: Routing to ADK agent
    Fields: agent_name, agent_url, trace_id, session_id
  
  - digitalocean_agent_call: ADK agent call completed
    Fields: agent_url, latency, prompt_size, response_size, trace_id
  
  - skill_generated: Evolution engine generated skill
    Fields: name

Performance Tracking:
  - Uses time.perf_counter() for microsecond precision
  - Logs execution_time for all major operations
  - Tracks latency for external API calls

────────────────────────────────────────────────────────────────────────────
EVENT BUS ARCHITECTURE
────────────────────────────────────────────────────────────────────────────

Implementation: Pub/sub pattern with asyncio
Thread Safety: asyncio.Lock for subscriber list
Subscribers: Async callback functions
Event Format: dict with "event" key + payload

Event Types:
  - task.created: New task created
    Payload: {"task_id": str, "title": str, "status": str}
  
  - task.progress: Task status update
    Payload: {"task_id": str, "status": str}
  
  - task.result: Task completed
    Payload: {"task_id": str, "status": str, "result": dict}
  
  - task.error: Task failed
    Payload: {"task_id": str, "error": str}
  
  - agent.step: Agent stage completed
    Payload: {"task_id": str, "agent": str, "data": dict}
  
  - evolution.generated: Skill generated
    Payload: {"skill": str, "description": str}

Subscription:
  - await event_bus.subscribe(callback)
  - Callback signature: async def callback(event: dict) -> None
  - Multiple subscribers supported

Publishing:
  - await event_bus.publish(event)
  - Broadcasts to all subscribers concurrently
  - Uses asyncio.gather() with return_exceptions=True
  - Continues even if subscriber fails

Unsubscription:
  - await event_bus.unsubscribe(callback)
  - Removes callback from subscriber list

Usage:
  - WebSocket streaming (event_stream.py)
  - Real-time UI updates
  - Monitoring and alerting
  - Audit logging

────────────────────────────────────────────────────────────────────────────
WEBSOCKET EVENT STREAMING
────────────────────────────────────────────────────────────────────────────

Endpoint: /ws/events
Protocol: WebSocket
Purpose: Real-time event streaming to clients

Connection Flow:
  1. Client connects to /ws/events
  2. Server accepts WebSocket connection
  3. Server creates asyncio.Queue for client
  4. Server subscribes queue handler to EventBus
  5. Server sends: {"event": "connected", "message": "..."}
  6. Server enters event loop:
     - Waits for events from queue
     - Sends events as JSON to client
  7. On disconnect: Unsubscribes from EventBus

Event Handler:
  async def handler(event: dict) -> None:
      await queue.put(event)

Error Handling:
  - Catches WebSocketDisconnect
  - Ensures unsubscribe on disconnect
  - Graceful cleanup

Client Usage:
  const ws = new WebSocket('ws://localhost:8000/ws/events');
  ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    console.log(data);
  };

────────────────────────────────────────────────────────────────────────────
TRACING AND CORRELATION
────────────────────────────────────────────────────────────────────────────

Trace ID:
  - Generated: task.id (UUID4)
  - Propagated: Through entire pipeline
  - Header: X-Trace-ID
  - Logged: In all agent calls and LLM requests
  - Purpose: Correlate logs across services

Session ID:
  - Optional: User session identifier
  - Header: X-AXON-SESSION
  - Logged: In agent calls
  - Purpose: Track user sessions across tasks

Correlation Strategy:
  1. Task created → trace_id = task.id
  2. AgentOrchestrator passes trace_id to agents
  3. DigitalOceanAgentRouter adds X-Trace-ID header
  4. ADK agents receive and can log trace_id
  5. All logs include task_id/trace_id
  6. Enables end-to-end request tracing

────────────────────────────────────────────────────────────────────────────
METRICS AND MONITORING
────────────────────────────────────────────────────────────────────────────

Performance Metrics (logged):
  - HTTP request duration (ms)
  - Agent execution time (seconds, 6 decimal places)
  - LLM token usage (prompt, completion, total)
  - ADK agent latency (seconds)
  - Payload sizes (bytes)

System Metrics (via /system endpoint):
  - Database status (ok/error)
  - Vector store status (ok/error)
  - Skills loaded count
  - Agents ready (boolean)
  - Event bus status (running/stopped)
  - Task queue status (running/stopped)

Health Checks:
  - GET /health: Basic health check
  - GET /system/: Comprehensive system status
  - Database: SELECT 1 query
  - Vector store: collection.count() call
  - ADK agents: GET /health endpoint

Audit Logging:
  - Runtime audit generation (utils/audit_logger.py)
  - Triggered in test mode on startup
  - Outputs: API routes, agents, skills, tables, config
  - File: backend/audit.log

Monitoring Recommendations:
  - Aggregate logs to centralized logging (e.g., ELK, Datadog)
  - Set up alerts on task_failed events
  - Monitor agent latency and token usage
  - Track queue size and worker status
  - Monitor database connection pool

================================================================================
                         10. SECURITY
================================================================================

────────────────────────────────────────────────────────────────────────────
AUTHENTICATION MECHANISMS
────────────────────────────────────────────────────────────────────────────

API KEY AUTHENTICATION:
  Configuration: API_KEY environment variable
  Header: X-API-Key
  Validation: require_api_key dependency
  Scope: All API endpoints (tasks, skills, evolution, system)
  Behavior:
    - If API_KEY not set: Authentication disabled
    - If API_KEY set: Validates X-API-Key header
    - Invalid key: 401 Unauthorized
  Implementation: config/dependencies.py
  Security Level: Basic (suitable for internal APIs)

DIGITALOCEAN API TOKEN:
  Configuration: DIGITALOCEAN_API_TOKEN
  Usage: ADK agent deployment and communication
  Header: Authorization: Bearer <token>
  Scope: Agent deployment, health checks, agent calls
  Validation: DigitalOcean platform
  Security Level: OAuth2 Bearer token

GRADIENT MODEL ACCESS KEY:
  Configuration: GRADIENT_MODEL_ACCESS_KEY
  Usage: ADK agents for inference
  Scope: Gradient inference API calls
  Validation: DigitalOcean Gradient platform
  Security Level: API key

────────────────────────────────────────────────────────────────────────────
AUTHORIZATION
────────────────────────────────────────────────────────────────────────────

Current Implementation:
  - No role-based access control (RBAC)
  - No user management
  - Single API key for all operations
  - No per-task or per-resource authorization

Recommendations:
  - Implement user authentication (JWT, OAuth2)
  - Add role-based access control
  - Implement per-resource authorization
  - Add task ownership and permissions

────────────────────────────────────────────────────────────────────────────
RATE LIMITING
────────────────────────────────────────────────────────────────────────────

Implementation: rate_limit_hook dependency
Strategy: Per-IP sliding window
Window: 60 seconds
Limit: Configurable (RATE_LIMIT_PER_MIN, default 120)
Storage: In-memory dict (_rate_state)
Behavior:
  - Tracks request count per IP
  - Resets window after 60 seconds
  - Returns 429 Too Many Requests if exceeded
  - Limit=0 disables rate limiting

Limitations:
  - In-memory storage (not distributed)
  - Lost on restart
  - No persistence
  - IP-based (can be bypassed with proxies)

Recommendations:
  - Use Redis for distributed rate limiting
  - Implement token bucket algorithm
  - Add per-user rate limiting
  - Add burst allowance

────────────────────────────────────────────────────────────────────────────
CORS CONFIGURATION
────────────────────────────────────────────────────────────────────────────

Configuration: CORS_ORIGINS environment variable
Default: ["*"] (all origins allowed)
Middleware: CORSMiddleware
Settings:
  - allow_origins: Configurable list
  - allow_credentials: True
  - allow_methods: ["*"] (all methods)
  - allow_headers: ["*"] (all headers)

Security Implications:
  - Default allows all origins (insecure for production)
  - Credentials enabled (cookies, auth headers)
  - All methods and headers allowed

Recommendations:
  - Restrict origins to known domains in production
  - Limit allowed methods to required ones
  - Limit allowed headers
  - Consider removing allow_credentials if not needed

────────────────────────────────────────────────────────────────────────────
INPUT VALIDATION
────────────────────────────────────────────────────────────────────────────

Framework: Pydantic models
Validation Points:
  - API request bodies (TaskCreate, etc.)
  - Configuration settings (Settings)
  - Agent requests/responses (AgentRequest, AgentResponse)

TaskCreate Validation:
  - title: min_length=1, max_length=255
  - description: Optional string

Skill Execution Validation:
  - Checks required parameters
  - Validates parameter presence
  - Raises ValueError if missing

Database Validation:
  - SQLAlchemy type constraints
  - NOT NULL constraints
  - String length limits
  - Foreign key constraints

Recommendations:
  - Add input sanitization for XSS prevention
  - Validate file uploads (if added)
  - Add SQL injection prevention (SQLAlchemy handles this)
  - Validate JSON payloads more strictly

────────────────────────────────────────────────────────────────────────────
SECRETS MANAGEMENT
────────────────────────────────────────────────────────────────────────────

Current Implementation:
  - Environment variables via .env files
  - python-dotenv for loading
  - No encryption at rest
  - No secrets rotation

Secrets:
  - API_KEY
  - GRADIENT_API_KEY
  - HUGGINGFACE_API_KEY
  - DIGITALOCEAN_API_TOKEN
  - GRADIENT_MODEL_ACCESS_KEY
  - DATABASE_URL (contains password)

Security Concerns:
  - .env files in version control (.gitignore required)
  - No encryption
  - No audit trail
  - No rotation mechanism

Recommendations:
  - Use secrets management service (AWS Secrets Manager, HashiCorp Vault)
  - Implement secrets rotation
  - Encrypt secrets at rest
  - Add audit logging for secret access
  - Use IAM roles instead of API keys where possible

────────────────────────────────────────────────────────────────────────────
DATA PROTECTION
────────────────────────────────────────────────────────────────────────────

Database:
  - PostgreSQL with password authentication
  - Connection string in environment variable
  - No encryption at rest (depends on PostgreSQL config)
  - No encryption in transit (depends on connection string)

Vector Store:
  - ChromaDB with local file storage
  - No encryption at rest
  - No access control

Recommendations:
  - Enable PostgreSQL SSL/TLS
  - Encrypt database at rest
  - Implement data retention policies
  - Add PII detection and masking
  - Encrypt vector store data

────────────────────────────────────────────────────────────────────────────
NETWORK SECURITY
────────────────────────────────────────────────────────────────────────────

HTTPS:
  - Not enforced at application level
  - Depends on deployment (reverse proxy, load balancer)

Recommendations:
  - Enforce HTTPS in production
  - Use TLS 1.2+ only
  - Implement HSTS headers
  - Add security headers (CSP, X-Frame-Options, etc.)

External API Calls:
  - All use HTTPS (Gradient, HuggingFace, ADK agents)
  - Certificate validation enabled (httpx default)
  - Timeout configured (prevents hanging)

────────────────────────────────────────────────────────────────────────────
SECURITY BEST PRACTICES IMPLEMENTED
────────────────────────────────────────────────────────────────────────────

✓ HTTPS for external API calls
✓ Timeout configuration (prevents DoS)
✓ Retry with exponential backoff (prevents thundering herd)
✓ Input validation with Pydantic
✓ SQL injection prevention (SQLAlchemy ORM)
✓ Rate limiting (basic)
✓ API key authentication (optional)
✓ Structured logging (audit trail)
✓ Error handling (no sensitive data in errors)

────────────────────────────────────────────────────────────────────────────
SECURITY RECOMMENDATIONS
────────────────────────────────────────────────────────────────────────────

HIGH PRIORITY:
  1. Restrict CORS origins in production
  2. Implement secrets management
  3. Enable database encryption (at rest and in transit)
  4. Add user authentication and authorization
  5. Implement distributed rate limiting

MEDIUM PRIORITY:
  6. Add security headers (HSTS, CSP, etc.)
  7. Implement audit logging for sensitive operations
  8. Add input sanitization for XSS prevention
  9. Implement data retention and deletion policies
  10. Add monitoring and alerting for security events

LOW PRIORITY:
  11. Implement API versioning
  12. Add request signing for agent communication
  13. Implement IP whitelisting
  14. Add honeypot endpoints for intrusion detection
  15. Conduct security audit and penetration testing

================================================================================
                    11. PERFORMANCE & CONCURRENCY
================================================================================

────────────────────────────────────────────────────────────────────────────
ASYNC EXECUTION MODEL
────────────────────────────────────────────────────────────────────────────

Framework: asyncio (Python 3.11+)
ASGI Server: Uvicorn
Concurrency Model: Single-threaded event loop with async/await

Async Components:
  - FastAPI endpoints (async def)
  - Database operations (AsyncSession, asyncpg)
  - HTTP clients (httpx.AsyncClient)
  - LLM API calls (async)
  - Agent calls (async)
  - Vector store operations (asyncio.to_thread for blocking ops)
  - Event bus (async pub/sub)
  - Task manager (async queue)

Benefits:
  - High concurrency with low memory overhead
  - Non-blocking I/O operations
  - Efficient handling of multiple requests
  - Scales well for I/O-bound workloads

Limitations:
  - Single-threaded (CPU-bound tasks block event loop)
  - Requires async-compatible libraries
  - Debugging can be complex

────────────────────────────────────────────────────────────────────────────
BACKGROUND WORKERS
────────────────────────────────────────────────────────────────────────────

Task Manager Worker:
  - Single background task (asyncio.create_task)
  - Processes task queue sequentially
  - Runs continuously until stopped
  - Graceful shutdown on stop()

Worker Characteristics:
  - Queue: asyncio.Queue (unbounded)
  - Concurrency: 1 (sequential processing)
  - Error Handling: Catches exceptions, continues processing
  - Lifecycle: Started on app startup, stopped on shutdown

Performance Implications:
  - Tasks processed one at a time
  - Long-running tasks block queue
  - No parallelism for task execution
  - Simple and predictable

Scaling Recommendations:
  - Implement worker pool for parallel processing
  - Use distributed task queue (Celery, RQ)
  - Add task prioritization
  - Implement task timeout and cancellation

────────────────────────────────────────────────────────────────────────────
DATABASE PERFORMANCE
────────────────────────────────────────────────────────────────────────────

Connection Pooling:
  - Driver: asyncpg (high-performance async PostgreSQL)
  - Pool size: 10 connections
  - Max overflow: 20 connections
  - Pool timeout: 30 seconds
  - Pre-ping: Enabled (connection health check)

Session Management:
  - Async sessions (AsyncSession)
  - No autoflush (manual control)
  - No expire on commit (reduces queries)
  - Auto-commit on success
  - Auto-rollback on exception

Query Optimization:
  - Indexes on foreign keys (task_id)
  - Unique index on skills.name
  - Primary keys on all tables
  - Cascade deletes (reduces queries)

Performance Characteristics:
  - Connection reuse (pooling)
  - Minimal round trips (no autoflush)
  - Efficient bulk operations (SQLAlchemy 2.0)
  - Async I/O (non-blocking)

Recommendations:
  - Add query result caching
  - Implement read replicas for scaling
  - Add database query logging for slow queries
  - Optimize N+1 queries with eager loading
  - Add database monitoring

────────────────────────────────────────────────────────────────────────────
VECTOR STORE PERFORMANCE
────────────────────────────────────────────────────────────────────────────

Implementation: ChromaDB
Storage: Persistent (file-based)
Embedding Model: sentence-transformers/all-MiniLM-L6-v2
Embedding Dimensions: 384

Performance Characteristics:
  - Embedding generation: ~10ms per text
  - Similarity search: ~50-100ms (depends on collection size)
  - Add operation: ~20-50ms
  - Blocking operations wrapped in asyncio.to_thread()

Optimization:
  - Model loaded once and cached (@lru_cache)
  - Embeddings normalized for faster search
  - Fallback to deterministic embeddings if model unavailable

Scaling Limitations:
  - Single-node (no distributed search)
  - File-based storage (limited by disk I/O)
  - No sharding or replication

Recommendations:
  - Migrate to distributed vector database (Pinecone, Weaviate, Qdrant)
  - Implement embedding caching
  - Add batch embedding generation
  - Optimize collection size (pruning old memories)
  - Add vector store monitoring

────────────────────────────────────────────────────────────────────────────
CACHING LAYERS
────────────────────────────────────────────────────────────────────────────

Current Caching:
  - Settings: @lru_cache(maxsize=1) on get_settings()
  - Embedding model: @lru_cache(maxsize=1) on _load_model()
  - Singletons: Module-level instances in dependencies.py

No Caching:
  - API responses
  - Database queries
  - LLM responses
  - Agent responses
  - Vector search results

Recommendations:
  - Add Redis for distributed caching
  - Cache frequent database queries (skills, system status)
  - Cache LLM responses (with TTL)
  - Implement HTTP response caching (ETag, Cache-Control)
  - Add cache invalidation strategy

────────────────────────────────────────────────────────────────────────────
HTTP CLIENT PERFORMANCE
────────────────────────────────────────────────────────────────────────────

Library: httpx (async HTTP client)
Connection Pooling: Enabled (default)
Timeout Configuration:
  - Gradient: 60 seconds
  - HuggingFace: 90 seconds
  - ADK agents: 120 seconds (configurable)

Retry Strategy:
  - Attempts: 3
  - Backoff: Exponential (1s-8s for LLM, 1s-10s for agents)
  - Retry on: HTTPError, TimeoutException
  - Library: tenacity

Performance Characteristics:
  - Connection reuse (HTTP/1.1 keep-alive)
  - Async I/O (non-blocking)
  - Automatic retry (resilience)
  - Timeout protection (prevents hanging)

Recommendations:
  - Implement circuit breaker pattern
  - Add request/response compression
  - Monitor connection pool usage
  - Add request deduplication
  - Implement adaptive timeout

────────────────────────────────────────────────────────────────────────────
MEMORY MANAGEMENT
────────────────────────────────────────────────────────────────────────────

Memory Usage:
  - Embedding model: ~80MB (loaded once)
  - ChromaDB: Depends on collection size
  - Database connection pool: ~10-30 connections
  - HTTP connection pools: Minimal
  - Task queue: Unbounded (potential memory leak)

Memory Optimization:
  - Lazy loading (embedding model, local LLM)
  - Singleton pattern (shared instances)
  - Connection pooling (reuse)
  - No large in-memory caches

Memory Concerns:
  - Unbounded task queue (can grow indefinitely)
  - No memory limits on vector store
  - No limits on task result size
  - No limits on agent response size

Recommendations:
  - Add queue size limits
  - Implement memory monitoring
  - Add result size limits
  - Implement memory-based backpressure
  - Add garbage collection tuning

────────────────────────────────────────────────────────────────────────────
LATENCY CHARACTERISTICS
────────────────────────────────────────────────────────────────────────────

Typical Latencies (Real Mode):
  - API request handling: <10ms
  - Database query: 5-20ms
  - Vector search: 50-100ms
  - Embedding generation: ~10ms
  - ADK agent call: 2-5 seconds (model dependent)
  - Full pipeline (4 agents): 8-20 seconds

Latency Breakdown (per task):
  1. Task creation: ~20ms (DB + queue)
  2. Planning agent: 2-5s
  3. Research agent: 2-5s (+ KB retrieval ~500ms)
  4. Reasoning agent: 2-5s
  5. Builder agent: 2-5s
  6. Result aggregation: ~50ms (DB + vector store)
  Total: 8-20 seconds

Optimization Opportunities:
  - Parallel agent execution (where possible)
  - Reduce context loading overhead
  - Optimize prompt construction
  - Cache agent responses
  - Implement streaming for faster perceived latency

────────────────────────────────────────────────────────────────────────────
THROUGHPUT CHARACTERISTICS
────────────────────────────────────────────────────────────────────────────

Current Throughput:
  - API requests: ~1000 req/s (limited by rate limiting)
  - Task processing: ~3-7 tasks/minute (sequential worker)
  - Database operations: ~100-500 ops/s (connection pool)
  - Vector operations: ~20-50 ops/s (embedding generation)

Bottlenecks:
  1. Sequential task processing (single worker)
  2. Agent latency (2-5s per agent)
  3. Vector store (single-node)
  4. Database (single instance)

Scaling Strategies:
  - Horizontal: Multiple backend instances + load balancer
  - Vertical: Larger database, more workers
  - Distributed: Task queue, vector database, read replicas

Recommendations:
  - Implement worker pool (parallel task processing)
  - Add load balancing
  - Implement task sharding
  - Add database read replicas
  - Migrate to distributed vector database

────────────────────────────────────────────────────────────────────────────
PERFORMANCE MONITORING
────────────────────────────────────────────────────────────────────────────

Current Monitoring:
  - Execution time logging (all operations)
  - HTTP request duration
  - Agent latency
  - Token usage
  - Payload sizes

Missing Monitoring:
  - CPU usage
  - Memory usage
  - Database connection pool metrics
  - Queue depth over time
  - Error rates
  - P50/P95/P99 latencies

Recommendations:
  - Add APM (Application Performance Monitoring)
  - Implement metrics collection (Prometheus)
  - Add dashboards (Grafana)
  - Set up alerting (PagerDuty, Opsgenie)
  - Implement distributed tracing (Jaeger, Zipkin)

================================================================================
                    12. TESTING INFRASTRUCTURE
================================================================================

────────────────────────────────────────────────────────────────────────────
TEST FRAMEWORK
────────────────────────────────────────────────────────────────────────────

Framework: pytest
Async Support: pytest-asyncio
Configuration: backend/tests/conftest.py
Test Directory: backend/tests/

Test Files:
  - conftest.py: Test configuration and fixtures
  - test_agent_pipeline.py: Agent orchestration tests
  - test_api_tasks.py: API endpoint tests
  - test_task_service.py: Service layer tests

Test Execution:
  Command: pytest
  Working Directory: backend/
  Python Path: Configured in conftest.py

────────────────────────────────────────────────────────────────────────────
TEST COVERAGE AREAS
────────────────────────────────────────────────────────────────────────────

AGENT PIPELINE TESTS (test_agent_pipeline.py):
  Test: test_agent_orchestrator_pipeline_runs
  Purpose: Validates full 4-stage agent pipeline
  Approach: Uses fake implementations (mocks)
  Coverage:
    - AgentOrchestrator.run_pipeline()
    - All 4 agents (planning, research, reasoning, builder)
    - Event bus integration
    - Memory storage
    - Database recording

  Fake Implementations:
    - FakeLLM: Returns mock LLM responses
    - FakeSkills: Returns mock skill outputs
    - FakeMemory: Simulates vector store
    - FakeSession: Simulates database session

  Assertions:
    - Result contains all agent outputs
    - Database records created (8+ items)
    - Pipeline completes successfully

TASK SERVICE TESTS (test_task_service.py):
  Test: test_task_service_create_task
  Purpose: Validates task creation flow
  Approach: Uses fake task manager
  Coverage:
    - TaskService.create_task()
    - TaskCreate schema validation
    - Task manager integration

  Fake Implementations:
    - FakeTaskManager: Simulates task manager operations

  Assertions:
    - Task created with correct title
    - Task status is 'queued'

API ENDPOINT TESTS (test_api_tasks.py):
  Status: File exists but content not analyzed
  Expected Coverage:
    - POST /tasks/
    - GET /tasks/
    - GET /tasks/{task_id}
    - Authentication
    - Rate limiting
    - Error handling

────────────────────────────────────────────────────────────────────────────
FIXTURES AND MOCKING STRATEGY
────────────────────────────────────────────────────────────────────────────

Mocking Approach:
  - Fake classes instead of mock libraries
  - Implements same interface as real components
  - Returns deterministic test data
  - Simpler than complex mock configurations

Fake Implementations:
  - FakeLLM: Simulates LLM service
    * Returns formatted responses based on prompt keywords
    * No external API calls
  
  - FakeSkills: Simulates skill executor
    * Returns predefined outputs per skill
    * Validates skill names
  
  - FakeMemory: Simulates vector store
    * Returns mock context
    * Tracks stored memories
  
  - FakeSession: Simulates database session
    * Tracks added items
    * No actual database operations
  
  - FakeTaskManager: Simulates task manager
    * Returns mock tasks
    * No queue operations

Benefits:
  - Fast test execution (no I/O)
  - Deterministic results
  - No external dependencies
  - Easy to understand and maintain

Limitations:
  - Doesn't test real integrations
  - May miss edge cases in real implementations
  - Requires maintenance when interfaces change

────────────────────────────────────────────────────────────────────────────
TEST MODES
────────────────────────────────────────────────────────────────────────────

TEST MODE (test_mode=True):
  Configuration: TEST_MODE=True in .env
  Behavior:
    - LLMService returns deterministic mock responses
    - No external API calls
    - Audit log generated on startup
    - Fast execution
  
  Use Cases:
    - Unit tests
    - CI/CD pipelines
    - Local development
    - Offline testing

MOCK MODE (axon_mode=mock):
  Configuration: AXON_MODE=mock in .env
  Behavior:
    - Agents use local skills instead of ADK agents
    - LLMService makes real API calls (if configured)
    - Falls back to local model if no API keys
    - Slower than test mode
  
  Use Cases:
    - Integration testing
    - Development without ADK agents
    - Testing skill implementations
    - API provider testing

REAL MODE (axon_mode=real):
  Configuration: AXON_MODE=real in .env
  Behavior:
    - Agents route to deployed ADK agents
    - LLMService disabled (returns error)
    - Full production behavior
    - Requires deployed agents
  
  Use Cases:
    - Production deployment
    - End-to-end testing
    - Performance testing
    - User acceptance testing

────────────────────────────────────────────────────────────────────────────
EVALUATION INFRASTRUCTURE
────────────────────────────────────────────────────────────────────────────

Evaluation Script: backend/scripts/run_agent_evaluation.py
Dataset: backend/scripts/evaluation_dataset_example.csv
Purpose: Automated agent quality assessment

Dataset Format (CSV):
  - prompt: Input task description
  - expected_output: Expected result
  - category: Test category
  - Additional metadata columns

Evaluation Process:
  1. Load dataset from CSV
  2. For each test case:
     - Submit task via API
     - Wait for completion
     - Compare result with expected output
     - Calculate accuracy metrics
  3. Generate evaluation report
  4. Export results

Metrics:
  - Accuracy: Percentage of correct outputs
  - Latency: Average execution time
  - Success rate: Percentage of completed tasks
  - Error rate: Percentage of failed tasks

Usage:
  python backend/scripts/run_agent_evaluation.py \
    --dataset evaluation_dataset_example.csv \
    --output results.json

Benefits:
  - Automated quality assessment
  - Regression detection
  - Performance benchmarking
  - A/B testing support

────────────────────────────────────────────────────────────────────────────
TESTING GAPS AND RECOMMENDATIONS
────────────────────────────────────────────────────────────────────────────

Current Coverage:
  ✓ Agent pipeline orchestration
  ✓ Task service layer
  ✓ Basic API endpoints
  ✓ Fake implementations for unit tests

Missing Coverage:
  ✗ Database integration tests
  ✗ Vector store integration tests
  ✗ LLM provider integration tests
  ✗ ADK agent integration tests
  ✗ WebSocket event streaming tests
  ✗ Rate limiting tests
  ✗ Authentication tests
  ✗ Error handling tests
  ✗ Concurrent request tests
  ✗ Evolution engine tests
  ✗ Skill registry tests
  ✗ Skill executor tests

Recommendations:

HIGH PRIORITY:
  1. Add integration tests with real database
  2. Add API endpoint tests (authentication, rate limiting)
  3. Add error handling tests
  4. Add concurrent request tests
  5. Increase test coverage to >80%

MEDIUM PRIORITY:
  6. Add vector store integration tests
  7. Add evolution engine tests
  8. Add skill system tests
  9. Add WebSocket tests
  10. Add performance tests

LOW PRIORITY:
  11. Add load tests
  12. Add chaos engineering tests
  13. Add security tests
  14. Add contract tests for ADK agents
  15. Add mutation testing

Test Infrastructure Improvements:
  - Add test coverage reporting (pytest-cov)
  - Add CI/CD integration (GitHub Actions, GitLab CI)
  - Add test data factories (factory_boy)
  - Add database fixtures (pytest-postgresql)
  - Add API test client (httpx.AsyncClient)
  - Add test parallelization (pytest-xdist)
  - Add test reporting (pytest-html)

================================================================================
                    13. DEPLOYMENT WORKFLOW
================================================================================

────────────────────────────────────────────────────────────────────────────
BACKEND DEPLOYMENT
────────────────────────────────────────────────────────────────────────────

Prerequisites:
  - Python 3.11+
  - PostgreSQL database
  - uv package manager (optional, recommended)

Installation Steps:
  1. Clone repository
  2. Navigate to backend/ directory
  3. Create virtual environment:
     python -m venv .venv
     source .venv/bin/activate  # Linux/Mac
     .venv\Scripts\activate     # Windows
  
  4. Install dependencies:
     pip install -e .
     # OR with uv:
     uv pip install -e .
  
  5. Configure environment:
     cp .env.example .env
     # Edit .env with your configuration
  
  6. Run database migrations:
     alembic upgrade head
  
  7. Start server:
     uvicorn src.main:app --host 0.0.0.0 --port 8000
     # OR with reload for development:
     uvicorn src.main:app --reload

Environment Configuration:
  - Set DATABASE_URL to PostgreSQL connection string
  - Set AXON_MODE=mock for development, real for production
  - Configure API keys if using real LLM providers
  - Set CORS_ORIGINS for production
  - Configure rate limiting
  - Set API_KEY for authentication (optional)

Production Deployment:
  - Use production ASGI server (uvicorn with workers)
  - Set up reverse proxy (nginx, Caddy)
  - Enable HTTPS
  - Configure logging to file or centralized service
  - Set up monitoring and alerting
  - Configure database connection pooling
  - Set up database backups
  - Configure vector store persistence

Example Production Command:
  uvicorn src.main:app \
    --host 0.0.0.0 \
    --port 8000 \
    --workers 4 \
    --log-level info \
    --access-log

Docker Deployment (Recommended):
  # Dockerfile example
  FROM python:3.11-slim
  WORKDIR /app
  COPY pyproject.toml .
  RUN pip install -e .
  COPY . .
  CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

  # docker-compose.yml example
  version: '3.8'
  services:
    backend:
      build: .
      ports:
        - "8000:8000"
      environment:
        - DATABASE_URL=postgresql+asyncpg://postgres:postgres@db:5432/axon
      depends_on:
        - db
    db:
      image: postgres:15
      environment:
        - POSTGRES_PASSWORD=postgres
        - POSTGRES_DB=axon
      volumes:
        - postgres_data:/var/lib/postgresql/data
  volumes:
    postgres_data:

────────────────────────────────────────────────────────────────────────────
ADK AGENT DEPLOYMENT
────────────────────────────────────────────────────────────────────────────

Prerequisites:
  - DigitalOcean account with Gradient AI access
  - gradient-adk CLI installed
  - API token with agent deployment permissions
  - Model access key for Gradient inference

Installation:
  pip install gradient-adk

Deployment Process (per agent):

1. PLANNER AGENT:
   cd agents/planner_agent
   cp .env.example .env
   # Edit .env:
   # GRADIENT_MODEL_ACCESS_KEY=gm-...
   # GRADIENT_MODEL=openai-gpt-oss-120b
   # DIGITALOCEAN_API_TOKEN=dop_v1_...
   gradient-adk deploy
   # Note the agent URL: https://agents.do-ai.run/<planner-id>

2. RESEARCH AGENT:
   cd agents/research_agent
   cp .env.example .env
   # Edit .env (same as planner + optional KB UUID)
   # DIGITALOCEAN_KB_UUID=kb-...
   gradient-adk deploy
   # Note the agent URL

3. REASONING AGENT:
   cd agents/reasoning_agent
   cp .env.example .env
   # Edit .env (same as planner)
   gradient-adk deploy
   # Note the agent URL

4. BUILDER AGENT:
   cd agents/builder_agent
   cp .env.example .env
   # Edit .env (same as planner)
   gradient-adk deploy
   # Note the agent URL

5. UPDATE BACKEND CONFIGURATION:
   # Edit backend/.env
   AXON_MODE=real
   DIGITALOCEAN_API_TOKEN=dop_v1_...
   GRADIENT_MODEL_ACCESS_KEY=gm-...
   AXON_PLANNER_AGENT_URL=https://agents.do-ai.run/<planner-id>
   AXON_RESEARCH_AGENT_URL=https://agents.do-ai.run/<research-id>
   AXON_REASONING_AGENT_URL=https://agents.do-ai.run/<reasoning-id>
   AXON_BUILDER_AGENT_URL=https://agents.do-ai.run/<builder-id>

Local Testing (before deployment):
  cd agents/planner_agent
  gradient-adk run --input '{"prompt": "Create a plan", "context": {}}'

Agent Management:
  # View logs
  gradient-adk logs <agent-id>
  
  # Check status
  gradient-adk status <agent-id>
  
  # Update agent
  gradient-adk deploy  # In agent directory
  
  # Delete agent
  gradient-adk delete <agent-id>

Health Checks:
  # From backend
  curl https://agents.do-ai.run/<agent-id>/health
  
  # From code
  GET /system/  # Checks all agent health

Monitoring:
  - View logs via gradient-adk logs
  - Monitor latency in backend logs
  - Check health endpoints regularly
  - Set up alerts for agent failures

────────────────────────────────────────────────────────────────────────────
DATABASE SETUP
────────────────────────────────────────────────────────────────────────────

PostgreSQL Installation:
  # Ubuntu/Debian
  sudo apt-get install postgresql postgresql-contrib
  
  # macOS
  brew install postgresql
  
  # Windows
  # Download installer from postgresql.org

Database Creation:
  sudo -u postgres psql
  CREATE DATABASE axon;
  CREATE USER axon_user WITH PASSWORD 'secure_password';
  GRANT ALL PRIVILEGES ON DATABASE axon TO axon_user;
  \q

Connection String:
  postgresql+asyncpg://axon_user:secure_password@localhost:5432/axon

Migration:
  # Run migrations
  cd backend
  alembic upgrade head
  
  # Create new migration
  alembic revision --autogenerate -m "description"
  
  # Rollback
  alembic downgrade -1

Production Recommendations:
  - Use managed PostgreSQL (DigitalOcean, AWS RDS, etc.)
  - Enable SSL/TLS
  - Configure backups
  - Set up replication
  - Monitor performance
  - Tune connection pool

────────────────────────────────────────────────────────────────────────────
VECTOR STORE SETUP
────────────────────────────────────────────────────────────────────────────

ChromaDB Setup:
  - Automatically initialized on first run
  - Persistent storage at VECTOR_DB_PATH
  - No separate installation required

Configuration:
  VECTOR_DB_PATH=./.chroma  # Relative to backend/

Data Persistence:
  - Data stored in .chroma/ directory
  - Includes SQLite database and vector data
  - Backup by copying entire directory

Migration to Production Vector DB:
  # For production, consider:
  - Pinecone (managed, scalable)
  - Weaviate (open-source, distributed)
  - Qdrant (open-source, fast)
  - Milvus (open-source, enterprise)

────────────────────────────────────────────────────────────────────────────
MONITORING AND OBSERVABILITY SETUP
────────────────────────────────────────────────────────────────────────────

Logging:
  - Logs to stdout (JSON format)
  - Capture with logging aggregator:
    * ELK Stack (Elasticsearch, Logstash, Kibana)
    * Datadog
    * Splunk
    * CloudWatch Logs

Metrics:
  - Implement Prometheus metrics
  - Set up Grafana dashboards
  - Monitor:
    * Request rate
    * Error rate
    * Latency (P50, P95, P99)
    * Queue depth
    * Database connections
    * Memory usage
    * CPU usage

Alerting:
  - Set up alerts for:
    * High error rate
    * High latency
    * Queue backup
    * Database connection exhaustion
    * Agent failures
    * Disk space

Health Checks:
  - GET /health: Basic health
  - GET /system/: Comprehensive status
  - Monitor from load balancer or uptime service

────────────────────────────────────────────────────────────────────────────
SCALING STRATEGY
────────────────────────────────────────────────────────────────────────────

Horizontal Scaling:
  1. Deploy multiple backend instances
  2. Add load balancer (nginx, HAProxy, AWS ALB)
  3. Use shared database (managed PostgreSQL)
  4. Use distributed vector store
  5. Implement distributed task queue (Redis, RabbitMQ)
  6. Share session state (Redis)

Vertical Scaling:
  1. Increase server resources (CPU, RAM)
  2. Increase database resources
  3. Increase worker count (uvicorn --workers)
  4. Optimize connection pools

Database Scaling:
  1. Add read replicas
  2. Implement connection pooling (PgBouncer)
  3. Partition large tables
  4. Optimize indexes
  5. Cache frequent queries

Agent Scaling:
  - ADK agents auto-scale on DigitalOcean
  - No manual scaling required
  - Monitor quotas and limits

────────────────────────────────────────────────────────────────────────────
ROLLBACK STRATEGY
────────────────────────────────────────────────────────────────────────────

Backend Rollback:
  1. Keep previous version deployed
  2. Switch load balancer to previous version
  3. Rollback database migrations if needed:
     alembic downgrade -1
  4. Monitor for issues

Agent Rollback:
  1. Set AXON_MODE=mock in backend
  2. Backend uses local skills
  3. No code changes required
  4. Agents remain deployed but unused

Database Rollback:
  1. Restore from backup
  2. Replay transaction log if needed
  3. Run migration rollback
  4. Verify data integrity

Vector Store Rollback:
  1. Restore .chroma/ directory from backup
  2. Restart backend
  3. Verify embeddings

────────────────────────────────────────────────────────────────────────────
DISASTER RECOVERY
────────────────────────────────────────────────────────────────────────────

Backup Strategy:
  - Database: Daily full backup, hourly incremental
  - Vector store: Daily backup of .chroma/ directory
  - Configuration: Version control (.env templates)
  - Code: Git repository

Recovery Procedures:
  1. Restore database from backup
  2. Restore vector store from backup
  3. Deploy backend from Git
  4. Redeploy agents if needed
  5. Verify system health
  6. Resume operations

RTO (Recovery Time Objective): <1 hour
RPO (Recovery Point Objective): <1 hour (depends on backup frequency)

================================================================================
                    14. ARCHITECTURAL SUMMARY
================================================================================

────────────────────────────────────────────────────────────────────────────
SYSTEM PURPOSE
────────────────────────────────────────────────────────────────────────────

AXON is a multi-agent AI orchestration platform designed to decompose complex
tasks into specialized stages and execute them through a coordinated pipeline
of AI agents. The system provides:

1. Task Management: Queue-based async task processing
2. Agent Orchestration: 4-stage pipeline (Plan → Research → Reason → Build)
3. Semantic Memory: Vector-based context retrieval and storage
4. Self-Evolution: Automatic skill generation from failure patterns
5. Real-time Streaming: WebSocket-based event broadcasting
6. Dual Execution Modes: Mock (local) and Real (cloud-deployed agents)

────────────────────────────────────────────────────────────────────────────
MAJOR COMPONENTS
────────────────────────────────────────────────────────────────────────────

1. BACKEND (FastAPI)
   Purpose: Orchestration layer and API gateway
   Responsibilities:
     - HTTP API endpoints (tasks, skills, evolution, system)
     - Task queue management
     - Agent pipeline coordination
     - Database persistence
     - Vector memory management
     - Event broadcasting
     - Authentication and rate limiting
   
   Key Modules:
     - main.py: Application entrypoint
     - core/agent_orchestrator.py: Pipeline coordinator
     - core/task_manager.py: Async task queue
     - core/event_bus.py: Pub/sub event system
     - core/evolution_engine.py: Skill generation
     - api/: HTTP routes and controllers
     - agents/: Agent implementations
     - skills/: Skill registry and executor

2. ADK AGENTS (DigitalOcean Gradient)
   Purpose: Specialized AI agents for task stages
   Agents:
     - Planner: Task decomposition and planning
     - Research: Information gathering and synthesis
     - Reasoning: Strategy evaluation and analysis
     - Builder: Solution generation and implementation
   
   Framework: LangGraph + Gradient SDK
   Deployment: DigitalOcean serverless infrastructure
   Inference: Gradient AI Platform (openai-gpt-oss-120b, llama3-*)
   
   Features:
     - State machine workflows (LangGraph)
     - Knowledge Base integration (Research agent)
     - Configurable temperature per agent
     - Streaming support
     - Health monitoring

3. DATABASE (PostgreSQL)
   Purpose: Persistent data storage
   Tables:
     - tasks: User tasks and results
     - skills: Generated and registered skills
     - agent_executions: Pipeline execution records
     - artifacts: File artifact metadata
     - memory_records: Vector embedding references
   
   Features:
     - Async operations (asyncpg)
     - Connection pooling
     - Migrations (Alembic)
     - Cascade deletes
     - Indexed foreign keys

4. VECTOR STORE (ChromaDB)
   Purpose: Semantic memory and context retrieval
   Features:
     - Persistent storage
     - Sentence transformer embeddings (384 dim)
     - Similarity search
     - Task-scoped retrieval
     - Metadata filtering
   
   Usage:
     - Context loading for agents
     - Memory storage after agent execution
     - Semantic search for related information

5. SKILLS SYSTEM
   Purpose: Pluggable execution modules
   Types:
     - Core skills: Built-in (planning, coding, reasoning)
     - Generated skills: Created by evolution engine
   
   Features:
     - Dynamic discovery
     - Parameter validation
     - Async execution
     - Version management
     - Source code persistence

6. EVENT BUS
   Purpose: Real-time event distribution
   Features:
     - Pub/sub pattern
     - Async subscribers
     - WebSocket streaming
     - Event types: task lifecycle, agent steps, evolution
   
   Usage:
     - Real-time UI updates
     - Monitoring and alerting
     - Audit logging

────────────────────────────────────────────────────────────────────────────
AGENT ARCHITECTURE
────────────────────────────────────────────────────────────────────────────

Pipeline Stages:
  1. PLANNING
     Input: Task title and description
     Process: Decompose into actionable steps
     Output: Structured plan with steps
     Context: Semantic memory retrieval
  
  2. RESEARCH
     Input: Task + Plan
     Process: Gather information, retrieve from KB
     Output: Research notes and synthesis
     Context: Semantic memory + Knowledge Base
  
  3. REASONING
     Input: Task + Plan + Research
     Process: Evaluate strategy, analyze tradeoffs
     Output: Analysis and rationale
     Context: Semantic memory
  
  4. BUILDER
     Input: Task + Plan + Research + Reasoning
     Process: Generate final solution
     Output: Implementation and artifacts
     Context: All previous stages

Execution Flow:
  Task Created → Queue → Worker Picks Up → Orchestrator Executes Pipeline
  → Each Stage: Load Context → Route to Agent → Store Result → Emit Event
  → Aggregate Results → Update Task → Broadcast Completion

Agent Communication:
  Backend → DigitalOceanAgentRouter → DigitalOceanAgentClient
  → HTTP POST to Agent URL → ADK Agent → Gradient Inference
  → Response → Parse → Store → Return

────────────────────────────────────────────────────────────────────────────
DATA FLOW
────────────────────────────────────────────────────────────────────────────

1. TASK SUBMISSION
   Client → POST /tasks/ → TaskController → TaskService → TaskManager
   → Create Task (DB) → Enqueue → Emit task.created → Return Task

2. TASK PROCESSING
   Worker → Dequeue → Load Task → Update Status → Orchestrator
   → For Each Stage:
      - Load Context (Vector Store)
      - Execute Agent (ADK or Local)
      - Store Result (Vector Store + DB)
      - Emit Event (EventBus)
   → Aggregate Results → Update Task → Emit task.result

3. CONTEXT RETRIEVAL
   Agent → VectorStore.retrieve_context() → Similarity Search
   → Top K Documents → Concatenate → Return Context String

4. MEMORY STORAGE
   Agent → VectorStore.add_embedding() → Generate Embedding
   → Store in ChromaDB → Create MemoryRecord (DB) → Return ID

5. EVENT BROADCASTING
   Component → EventBus.publish() → Notify Subscribers
   → WebSocket Clients → Send JSON → Client Receives

6. SKILL EVOLUTION
   Evolution Trigger → Count Failed Tasks → Generate Skill (LLM)
   → Write Python Module → Reload Registry → Persist to DB
   → Emit evolution.generated

────────────────────────────────────────────────────────────────────────────
CONTROL FLOW
────────────────────────────────────────────────────────────────────────────

Application Lifecycle:
  1. Startup (lifespan)
     - Configure logging
     - Initialize database (create tables)
     - Start task manager worker
     - Generate audit log (test mode)
  
  2. Request Handling
     - Middleware: Logging, CORS
     - Dependencies: API key, rate limiting
     - Route handler
     - Service layer
     - Database/external APIs
     - Response
  
  3. Background Processing
     - Task manager worker loop
     - Dequeue task
     - Execute pipeline
     - Handle errors
     - Continue
  
  4. Shutdown (lifespan)
     - Stop task manager
     - Close database connections
     - Cleanup resources

Error Handling:
  - HTTP errors: FastAPI exception handlers
  - Task errors: Caught, logged, task marked failed
  - Agent errors: Retry with exponential backoff
  - Database errors: Rollback transaction
  - Vector store errors: Fallback to ephemeral client

────────────────────────────────────────────────────────────────────────────
KEY DESIGN PATTERNS
────────────────────────────────────────────────────────────────────────────

1. PIPELINE PATTERN
   - Sequential stages with data flow
   - Each stage enriches the context
   - Final stage produces output

2. REPOSITORY PATTERN
   - Service layer abstracts data access
   - Database operations encapsulated
   - Testable with fake implementations

3. DEPENDENCY INJECTION
   - FastAPI Depends() for services
   - Singleton instances in dependencies.py
   - Loose coupling between components

4. PUB/SUB PATTERN
   - EventBus for decoupled communication
   - Multiple subscribers per event
   - Async event handling

5. STRATEGY PATTERN
   - LLMService with provider fallback
   - Agent execution (mock vs real mode)
   - Skill execution (sync vs async)

6. FACTORY PATTERN
   - SkillRegistry builds SkillDefinitions
   - Task creation in TaskManager
   - Agent instantiation in Orchestrator

7. SINGLETON PATTERN
   - Settings (cached)
   - Embedding model (cached)
   - Core services (module-level)

8. RETRY PATTERN
   - Exponential backoff for API calls
   - Configurable attempts
   - Graceful degradation

────────────────────────────────────────────────────────────────────────────
ARCHITECTURAL STRENGTHS
────────────────────────────────────────────────────────────────────────────

✓ Modular design with clear separation of concerns
✓ Async/await for high concurrency
✓ Dual execution modes (mock/real) for flexibility
✓ Semantic memory for context-aware agents
✓ Self-evolution capability
✓ Real-time event streaming
✓ Comprehensive logging and tracing
✓ Retry and fallback mechanisms
✓ Type safety with Pydantic
✓ Database migrations with Alembic
✓ Testable with fake implementations
✓ Extensible skill system
✓ Cloud-native agent deployment

────────────────────────────────────────────────────────────────────────────
ARCHITECTURAL LIMITATIONS
────────────────────────────────────────────────────────────────────────────

✗ Single worker (sequential task processing)
✗ In-memory rate limiting (not distributed)
✗ No user management or RBAC
✗ Single-node vector store (not scalable)
✗ No caching layer (Redis)
✗ No circuit breaker for external APIs
✗ No distributed tracing (Jaeger, Zipkin)
✗ No metrics collection (Prometheus)
✗ Limited error recovery (no retry for tasks)
✗ No task prioritization
✗ No task cancellation
✗ No streaming for long-running tasks
✗ No agent result caching

────────────────────────────────────────────────────────────────────────────
FUTURE ENHANCEMENTS
────────────────────────────────────────────────────────────────────────────

SHORT TERM:
  1. Implement worker pool for parallel task processing
  2. Add Redis for distributed caching and rate limiting
  3. Implement user authentication and authorization
  4. Add task prioritization and cancellation
  5. Improve error recovery with task retry

MEDIUM TERM:
  6. Migrate to distributed vector database (Pinecone, Weaviate)
  7. Add circuit breaker for external APIs
  8. Implement distributed tracing
  9. Add metrics collection and dashboards
  10. Implement agent result caching

LONG TERM:
  11. Add multi-tenancy support
  12. Implement agent marketplace
  13. Add custom agent creation UI
  14. Implement agent versioning and A/B testing
  15. Add federated learning for agent improvement

────────────────────────────────────────────────────────────────────────────
CONCLUSION
────────────────────────────────────────────────────────────────────────────

AXON is a well-architected multi-agent orchestration platform with a clear
separation of concerns, modern async architecture, and flexible execution
modes. The system successfully integrates DigitalOcean's Gradient AI Platform
for scalable agent deployment while maintaining local development capabilities.

Key strengths include the modular design, semantic memory system, self-evolution
capability, and comprehensive observability. The dual execution modes (mock/real)
enable rapid development and testing without external dependencies.

Primary areas for improvement include scaling the task processing (worker pool),
implementing distributed caching and rate limiting, adding user management, and
migrating to a distributed vector database for production scalability.

The architecture is production-ready for moderate workloads and provides a solid
foundation for future enhancements and scaling.

================================================================================
                         END OF AUDIT REPORT
================================================================================