Skip to content

rodneymbrown1/LlamaVault

BranchesTags

Repository files navigation

LlamaVault

LlamaVault

Offline-first AI agent system powered by llama.cpp, secured by AI SAFE²

Self-hosted · MCP-native · Governance-ready · Single-command deploy

What is LlamaVault?

LlamaVault is a ready-to-deploy, offline-capable AI agent that runs quantized LLMs locally on your own infrastructure. It combines three battle-tested components:

No data leaves your machine. No cloud API keys required. Extend capabilities via MCP servers.

Architecture

┌─────────────────────────────────────────────────────────┐
│                   AWS Lightsail Instance                 │
│                 (2xlarge_3_0 / 16GB RAM)                 │
│                                                         │
│  ┌──────────────┐    ┌──────────────┐   ┌────────────┐  │
│  │   nanobot     │───▶│  SAFE² GW    │──▶│ llama.cpp  │  │
│  │  agent        │    │  (proxy)     │   │  server    │  │
│  │  :18790       │    │  :8000       │   │  :8080     │  │
│  └──────┬───────┘    └──────────────┘   └────────────┘  │
│         │                                      │         │
│  ┌──────┴───────┐                       ┌─────┴──────┐  │
│  │ MCP servers  │                       │  /data/     │  │
│  │ (stdio/HTTP) │                       │   models/   │  │
│  │ sessions/    │                       │   *.gguf    │  │
│  │ memory/      │                       │             │  │
│  └──────────────┘                       └────────────┘  │
│                                                         │
│  ┌──────────────┐                                       │
│  │   nginx      │ ◀── public :80/443                    │
│  │  reverse     │──▶ nanobot :18790                     │
│  │  proxy       │                                       │
│  └──────────────┘                                       │
└─────────────────────────────────────────────────────────┘
         ▲
         │
   MCP Clients / HTTP API / CLI

Request flow: API request or MCP call → nanobot agent → AI SAFE² gateway (input sanitization, rate limiting, audit logging) → llama.cpp server → response back through the stack. The agent can also call out to configured MCP servers for external capabilities (filesystem, GitHub, databases, etc.).

Quick Start

Prerequisites

  • Docker & Docker Compose
  • Python 3.10+ (for model download script)
  • ~4GB disk space for the smallest model

1. Clone and configure

git clone https://github.com/rodneymbrown1/LlamaVault.git
cd LlamaVault

Edit llamavault.yaml — this is the single config file for the entire system.

2. Download a model

make model-download

Downloads the model specified in llamavault.yaml. Or override: make model-download MODEL=llama-3.1-8b-instruct

3. Start locally

make dev

Three services will start:

  • llama-server at http://localhost:8080 — LLM inference
  • safe2-gateway at http://localhost:8000 — governance proxy
  • nanobot at http://localhost:18790 — agent gateway

4. Test it

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3.2-3b-instruct", "messages": [{"role": "user", "content": "Hello!"}]}'

MCP Integration

LlamaVault uses the Model Context Protocol as its primary extension mechanism. Instead of built-in channel connectors, capabilities are added via MCP servers.

Configuring MCP Servers

Add MCP servers in llamavault.yaml:

mcp_servers:
  # Stdio transport — runs MCP server as a subprocess
  filesystem:
    command: npx
    args: ["-y", "@anthropic/mcp-filesystem-server", "/data"]

  # With environment variables
  github:
    command: npx
    args: ["-y", "@anthropic/mcp-github-server"]
    env:
      GITHUB_TOKEN: ghp_xxx

  # HTTP transport — connects to a remote MCP server
  custom:
    url: http://localhost:3000/mcp
    headers:
      Authorization: "Bearer xxx"

mcp_allowed_commands: [npx, uvx]

### MCP Server Config Options

| Field | Type | Description |
|-------|------|-------------|
| `command` | string | Stdio: command to run (e.g. `npx`, `uvx`) |
| `args` | string[] | Stdio: command arguments |
| `env` | object | Stdio: extra environment variables |
| `url` | string | HTTP: streamable HTTP endpoint URL |
| `headers` | object | HTTP: custom HTTP headers |
| `tool_timeout` | int | Seconds before a tool call is cancelled (default: 30) |

### Popular MCP Servers

| Server | Install | Description |
|--------|---------|-------------|
| Filesystem | `npx -y @anthropic/mcp-filesystem-server /path` | Read/write local files |
| GitHub | `npx -y @anthropic/mcp-github-server` | Repos, issues, PRs |
| PostgreSQL | `npx -y @anthropic/mcp-postgres-server` | Query databases |
| Brave Search | `npx -y @anthropic/mcp-brave-search` | Web search |
| Memory | `npx -y @anthropic/mcp-memory-server` | Persistent key-value store |

See [MCP Server Registry](https://github.com/modelcontextprotocol/servers) for the full list.

---

## Available Models

| Model | Size | Min RAM | Tool Calling | Tier |
|-------|------|---------|--------------|------|
| Llama 3.2 3B Instruct (Q4) | 2.0 GB | 4 GB | Yes | Minimum |
| Phi-3 Mini (Q4) | 2.3 GB | 4 GB | No | Minimum |
| Llama 3.1 8B Instruct (Q4) | 4.9 GB | 8 GB | Yes | Recommended |
| Mistral 7B Instruct v0.2 (Q4) | 4.4 GB | 8 GB | No | Recommended |
| Qwen 2.5 14B Instruct (Q4) | 8.9 GB | 16 GB | Yes | Full |

```bash
# List all available models
make model-list

# Download a specific model
make model-download MODEL=llama-3.1-8b-instruct

# Hot-swap without restarting the agent
make model-switch MODEL=llama-3.1-8b-instruct

AI SAFE² Governance

LlamaVault integrates the AI SAFE² Framework across all five pillars:

Pillar Implementation
S — Sanitize & Isolate Gateway blocks prompt injection via pattern matching; MCP server outputs treated as untrusted
A — Audit & Inventory Every inference request logged with Prometheus metrics; immutable audit trail
F — Fail-Safe & Recovery Global kill switch halts all inference in <100ms; rate limiting; circuit breakers
E — Engage & Monitor Heartbeat protocol checks model health, alignment drift, and system resources
E² — Evolve & Educate Governance templates (SOUL.md, AGENTS.md) versioned and PR-gated

Kill Switch

# Emergency halt — blocks all inference
make kill-switch-on

# Resume
make kill-switch-off

# Check status
make health

Governance Files

Located in governance/, these are mounted into the nanobot workspace:

  • SOUL.md — Agent identity, ethical constraints, Love Equation alignment
  • AGENTS.md — Operating manual, MCP security, tool priority
  • HEARTBEAT.md — Periodic health checks and drift detection
  • SUBAGENT-POLICY.md — Delegation rules and trust levels
  • KILL-SWITCH.md — Emergency halt protocol (P3.T5.7)

Deploy to AWS Lightsail

Hardware Tiers

Tier Bundle RAM Models Cost/mo
Minimum xlarge_3_0 8 GB 3B Q4 ~$40
Recommended 2xlarge_3_0 16 GB 7-8B Q4 ~$80
Full 4xlarge_3_0 32 GB 13B+ Q4 ~$160

Deploy

# Full pipeline: scan → CDK → build → push
make deploy

# Or step by step:
make scan                    # AI SAFE² security scan
make infra-deploy            # CDK provisions Lightsail + ECR + Secrets
make deploy --skip-infra     # Build and push images only

The CDK stack provisions:

  • Lightsail instance (Ubuntu 22.04, 16GB RAM default)
  • 60GB persistent disk for models and agent state
  • Static IP
  • ECR repository for Docker images
  • Secrets Manager for configuration
  • nginx reverse proxy with TLS-ready config
  • systemd auto-start on boot
  • Auto-downloads the default model on first boot

Configuration

All deploy settings are in llamavault.yaml:

deploy:
  region: us-east-1
  bundle: 2xlarge_3_0       # Instance size
  disk_size_gb: 60
  instance_name: llamavault
  ssh_cidrs: ["YOUR_IP/32"] # Required — restrict SSH

Project Structure

LlamaVault/
├── llamavault.yaml             # Single config file — edit this
├── docker-compose.yml          # Production: 3-service stack
├── docker-compose.dev.yml      # Dev mode with port exposure
├── Makefile                    # All operations: dev, deploy, model, health
│
├── llama-server/               # llama.cpp inference container
│   ├── Dockerfile
│   └── entrypoint.sh
│
├── gateway/                    # AI SAFE² governance proxy
│   ├── Dockerfile
│   ├── main.py                 # Input sanitization, rate limiting, kill switch
│   └── config/default.yaml     # Security policy
│
├── governance/                 # OpenClaw governance templates
│   ├── SOUL.md                 # Agent identity & ethics
│   ├── AGENTS.md               # Operating manual
│   ├── HEARTBEAT.md            # Health monitoring
│   ├── SUBAGENT-POLICY.md      # Delegation rules
│   └── KILL-SWITCH.md          # Emergency halt protocol
│
├── models/                     # Model registry & downloader
│   ├── models.yaml
│   └── download.sh
│
├── infra/                      # AWS CDK (TypeScript)
│   ├── lib/
│   │   ├── llamavault-shared-stack.ts
│   │   ├── llamavault-instance-stack.ts
│   │   └── scripts/user-data.sh
│   └── bin/infra.ts
│
└── scripts/                    # Operations
    ├── deploy.sh
    ├── model-switch.sh
    ├── health-check.sh
    └── safe2-scan.sh

Makefile Reference

make help              # Show all targets
make show-config       # Show current config from llamavault.yaml
make dev               # Start in dev mode
make dev-down          # Stop dev services
make up                # Start production
make down              # Stop production
make build             # Build all images
make health            # Check all services
make model-download    # Download model from config (or MODEL=name)
make model-list        # List available/downloaded models
make model-switch      # Switch model + update config: MODEL=name
make scan              # Run AI SAFE² scanner
make kill-switch-on    # Halt all inference
make kill-switch-off   # Resume inference
make deploy            # Full AWS deploy
make infra-deploy      # CDK only

License

MIT

Built with llama.cpp · Secured by AI SAFE² · Powered by CloudAgentKit

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages