Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
442 lines (326 loc) · 13.7 KB

DEPLOYMENT.md

File metadata and controls

442 lines (326 loc) · 13.7 KB
Raw

AuraRouter Deployment Guide

Deployment Options

AuraRouter supports four deployment contexts:

  1. Standalone Python - Install via PyPI or from source. Feature-complete with durable configuration.
  2. Standalone Executable (Windows) - Single-file .exe with all dependencies bundled.
  3. AuraGrid MAS - Deploy as a managed application service on AuraGrid. See AURAGRID.md.
  4. Conda - Isolated environment via conda/mamba. Uses the same package.

Standalone Executable (Windows)

AuraRouter can be compiled into a single-file executable that bundles Python, PySide6, and the llama-server binaries. This is ideal for distribution without requiring a local Python installation.

To build the executable:

  1. Install build dependencies: pip install PyInstaller
  2. Run the build script from the project root: python build.py
  3. The executable will be available at aurarouter/dist/aurarouter.exe.

Prerequisites

  • Python 3.12+
  • For local inference: Ollama and/or llama.cpp
  • For cloud providers via openapi: API keys for your provider

Installation

Option A: PyPI Install

# Core (MCP server + GUI + llamacpp-server HTTP provider)
pip install aurarouter

# Add HuggingFace model downloads
pip install aurarouter[local]

# Install everything (local + AuraGrid + dev tools)
pip install aurarouter[all]

Option B: Source Install

git clone https://github.com/auracoredynamics/aurarouter.git
cd aurarouter

# Install core dependencies
pip install -r requirements.txt

# (Optional) Install local inference dependencies
pip install -r requirements-local.txt

# Install the package in editable mode
pip install -e .

Option C: Conda

conda env create -f environment.yaml
conda activate aurarouter

Configuration

Config File Location

AuraRouter searches for auraconfig.yaml in this order:

  1. --config CLI argument
  2. AURACORE_ROUTER_CONFIG environment variable
  3. ~/.auracore/aurarouter/auraconfig.yaml

Creating Initial Configuration

Interactive:

aurarouter --install

This creates ~/.auracore/aurarouter/auraconfig.yaml (if it doesn't exist) and offers to register AuraRouter with MCP clients.

Manual: Create ~/.auracore/aurarouter/auraconfig.yaml:

system:
  log_level: INFO
  default_timeout: 120.0

models:
  # Local models (via Ollama)
  local_qwen:
    provider: ollama
    endpoint: http://localhost:11434/api/generate
    model_name: qwen2.5-coder:7b
    parameters:
      temperature: 0.1
      num_ctx: 4096

  # Local models (via llama-server HTTP - no native deps)
  # local_llama_server:
  #   provider: llamacpp-server
  #   endpoint: http://localhost:8080
  #   parameters:
  #     temperature: 0.1
  #     n_predict: 2048

  # Local models (managed llama-server - uses bundled llama-server binary)
  # local_llama_embedded:
  #   provider: llamacpp
  #   model_path: "/path/to/model.gguf"
  #   parameters:
  #     n_ctx: 4096
  #     n_gpu_layers: -1
  #     temperature: 0.1

  # OpenAPI-compatible endpoint (vLLM, LocalAI, LM Studio, etc.)
  # my_vllm:
  #   provider: openapi
  #   endpoint: http://localhost:8000/v1
  #   model_name: meta-llama/Llama-3-8B
  #   tags: [private, coding]

roles:
  router:    [local_qwen]
  reasoning: [local_qwen]
  coding:    [local_qwen]

# Fault tolerance — circuit breaker per provider
# resilience:
#   failure_threshold: 5    # consecutive failures before circuit opens (default: 5)
#   reset_timeout: 60.0     # seconds before open circuit probes (default: 60.0)

# Model telemetry polling
# telemetry:
#   poll_interval: 15.0     # background provider poll interval in seconds (default: 15.0)

# Optional: map synonyms to canonical role names for intent classification
# semantic_verbs:
#   coding:
#     synonyms: [programming, code generation, developer]
#   reasoning:
#     synonyms: [planner, architect, planning]

Semantic Verbs, Roles, and Custom Intents

AuraRouter uses a layered system to connect user prompts to the right model:

Semantic verbs map synonym tags to canonical role names. They handle tag-to-role resolution during auto-join. For example, a model tagged [programming] is automatically associated with the coding role because programming is a synonym:

semantic_verbs:
  coding: [code, program, develop, programming]
  reasoning: [planner, architect, planning, analysis]

Role bindings in analyzers declare custom intents that map to roles. This is the mechanism for domain-specific intent classification. Each key in role_bindings becomes an intent that the router model can classify tasks into:

catalog:
  my-analyzer:
    kind: analyzer
    analyzer_kind: intent_triage
    role_bindings:
      generate_code: coding       # "generate_code" intent -> coding role
      review: reasoning           # "review" intent -> reasoning role
      sar_detection: coding       # Custom domain intent -> coding role

Supported intents on model artifacts declare which intents a specific model is suited for. When ComputeFabric.filter_chain_by_intent() is called, models that declare supported_intents are preferred for matching intents:

catalog:
  sar-model:
    kind: model
    display_name: SAR Specialist
    provider: ollama
    supported_intents: [sar_detection, sar_geolocation]

The relationship flows as: semantic_verbs resolve tags to roles, role_bindings declare intents that map to roles, and supported_intents on models declare per-intent eligibility within a role's chain.

Configuration Persistence

Configuration changes made through the GUI or the ConfigLoader API are saved atomically back to the YAML file. Changes survive restarts.

Environment Variable Overrides

Any config key can be overridden via AURAROUTER_* environment variables using __ for nesting:

# Override a model endpoint
export AURAROUTER_MODELS__LOCAL_QWEN__ENDPOINT=http://192.168.1.100:11434/api/generate

# Override API keys (for openapi provider)
# export MY_API_KEY=your-key-here

Provider Setup

Ollama (Local HTTP)

  1. Install Ollama
  2. Pull a model: ollama pull qwen2.5-coder:7b
  3. Configure in YAML: 
    local_model:
  provider: ollama
  endpoint: http://localhost:11434/api/generate
  model_name: qwen2.5-coder:7b

llama.cpp Server (Local HTTP, Zero Native Deps)

  1. Download llama-server from llama.cpp releases
  2. Download a GGUF model (e.g., via aurarouter download-model)
  3. Start the server: llama-server -m model.gguf --port 8080
  4. Configure in YAML: 
    local_llama:
  provider: llamacpp-server
  endpoint: http://localhost:8080

llama.cpp Managed (Local Native)

AuraRouter bundles pre-built llama-server binaries for Windows x64, Linux x64, and macOS x64. No compilation or external installation is required.

  1. Download a GGUF model: 
    aurarouter download-model \
  --repo Qwen/Qwen2.5-Coder-7B-Instruct-GGUF \
  --file qwen2.5-coder-7b-instruct-q4_k_m.gguf
  2. Configure in YAML: 
    local_embedded:
  provider: llamacpp
  model_path: ~/.auracore/models/qwen2.5-coder-7b-instruct-q4_k_m.gguf
  parameters:
    n_ctx: 4096
    n_gpu_layers: -1

AuraRouter automatically detects your OS and uses the correct binary. The llama-server subprocess starts on demand and stops when AuraRouter exits.

Custom binary path: To use your own llama-server build (e.g., a CUDA-enabled build for GPU acceleration), set the environment variable:

export AURAROUTER_LLAMACPP_BIN=/path/to/your/llama-server

Or add to auraconfig.yaml:

system:
  llamacpp_binary: /path/to/your/llama-server

OpenAPI-Compatible (Local/Cloud HTTP)

Works with any endpoint implementing the OpenAI chat completions API: vLLM, text-generation-inference, LocalAI, LM Studio, etc.

  1. Start your OpenAI-compatible server (e.g., vllm serve meta-llama/Llama-3-8B)
  2. Configure in YAML: 
    my_vllm:
  provider: openapi
  endpoint: http://localhost:8000/v1
  model_name: meta-llama/Llama-3-8B
  # api_key: optional-key  # or env_key: VLLM_API_KEY
  parameters:
    temperature: 0.7
    max_tokens: 2048

The provider sends requests to {endpoint}/chat/completions using the standard OpenAI request/response format. Locality (local vs cloud) is inferred from the endpoint address -- localhost/127.0.0.1 endpoints are treated as local; others as cloud. You can override this with an explicit locality: local or locality: cloud field.

Running

Standalone Executable (Windows)

# MCP server (default)
.\aurarouter\dist\aurarouter.exe

# Desktop GUI
.\aurarouter\dist\aurarouter.exe gui

# With explicit config
.\aurarouter\dist\aurarouter.exe --config /path/to/auraconfig.yaml

Python Module (Installed)

aurarouter                                    # Uses default config location
aurarouter gui                                # Launch desktop GUI
aurarouter --config /path/to/auraconfig.yaml  # Explicit config
python -m aurarouter                          # Module invocation

GUI Administration

The GUI provides full service lifecycle management:

  • Environment selector: Switch between Local and AuraGrid at runtime (toolbar dropdown).
  • Service controls: Start/Stop/Pause buttons manage the MCP server subprocess (Local) or MAS lifecycle (AuraGrid).
  • Health dashboard: Click the health indicator to see per-model status. Use "Check" to run diagnostics.
  • Document upload: Attach files as context for tasks via the Execute tab.
  • DAG visualization: Expandable execution trace showing the classify/plan/execute pipeline with per-node telemetry.
  • Privacy-aware routing: Automatically re-routes PII-containing prompts to local/private-tagged models.
  • Prompt history: Recent tasks are saved and restorable from a dropdown.
  • Keyboard shortcuts: Ctrl+Enter (execute), Ctrl+N (new prompt), Escape (cancel).

See GUI_GUIDE.md for the complete GUI reference.

Model Management

# Download a GGUF model from HuggingFace
aurarouter download-model \
  --repo Qwen/Qwen2.5-Coder-7B-Instruct-GGUF \
  --file qwen2.5-coder-7b-instruct-q4_k_m.gguf

# List locally downloaded models
aurarouter list-models

# Remove a downloaded model (deletes file)
aurarouter remove-model --file qwen2.5-coder-7b-instruct-q4_k_m.gguf

# Remove from registry only, keep the file
aurarouter remove-model --file model.gguf --keep-file

Downloaded models are stored in ~/.auracore/models/ by default, with a models.json registry tracking metadata (repo, filename, size, download date).

MCP Asset Management

AuraRouter exposes two MCP tools for programmatic asset management. These enable external services (e.g., AuraGrid MAS nodes running fine-tuning jobs) to discover and register GGUF models without manual config editing.

Listing Assets

The list_assets tool queries the physical model registry (~/.auracore/models/models.json):

# Returns JSON array of all downloaded GGUF models
result = mcp_client.call_tool("list_assets", {})

Each entry includes repo, filename, path, size_bytes, downloaded_at, and optional gguf_metadata.

Registering Assets

The register_asset tool registers a new GGUF file and adds it to the routing config:

result = mcp_client.call_tool("register_asset", {
    "model_id": "my-finetuned-qwen",       # Unique routing identifier
    "file_path": "/path/to/model.gguf",     # Absolute path to .gguf file
    "repo": "myorg/my-finetuned-model",     # HuggingFace repo or "local"
    "tags": "coding,local,fine-tuned"       # Comma-separated capability tags
})

Registration performs three operations atomically:

  1. Validates the file exists and has a .gguf extension
  2. Adds the model to auraconfig.yaml with the llamacpp provider
  3. Registers the file in the FileModelStorage registry

The model becomes routable after the next server restart or config reload. Registration does not auto-add the model to any role chain — the caller must update role chains separately.

Both tools can be disabled via auraconfig.yaml:

mcp:
  tools:
    assets.list:
      enabled: false
    assets.register:
      enabled: false

Data Directories

Path Purpose
~/.auracore/aurarouter/auraconfig.yaml Runtime configuration
~/.auracore/aurarouter/aurarouter.pid Singleton PID lock file
~/.auracore/aurarouter/history.json GUI prompt history (last 20 tasks + results)
~/.auracore/models/ Downloaded GGUF model files
~/.auracore/models/models.json Model registry (auto-managed)

Programmatic Usage

from aurarouter.config import ConfigLoader
from aurarouter.fabric import ComputeFabric

# Load config
config = ConfigLoader(config_path="auraconfig.yaml")

# Execute tasks (AuraRouter is content-agnostic)
fabric = ComputeFabric(config)
result = fabric.execute("coding", "Write a hello world function in Python")
result = fabric.execute("coding", "Summarize the key findings from the attached report")

# Modify and save config
config.set_model("new_model", {"provider": "ollama", "model_name": "llama3"})
config.set_role_chain("coding", ["new_model", "cloud_gemini"])
config.save()

# List physical GGUF assets (independent of config)
from aurarouter.mcp_tools import list_assets, register_asset
import json

assets = json.loads(list_assets())
print(f"{len(assets)} GGUF model(s) in local storage")

# Register a new GGUF model for routing
result = json.loads(register_asset(
    model_id="my-model",
    file_path="/path/to/model.gguf",
    tags="coding,private",
))