Skip to content

terraform.md

Latest commit

 

History

History
701 lines (523 loc) · 17.4 KB

terraform.md

File metadata and controls

701 lines (523 loc) · 17.4 KB
Raw

OpenClaw Cloud Native - Terraform Deployment

This directory contains Terraform configuration for deploying OpenClaw to Kubernetes.

Prerequisites

  • Terraform >= 1.0
  • kubectl configured with cluster access
  • Kubernetes cluster with sufficient resources

Quick Start

cd terraform
terraform init
terraform apply

Quick Start

For development (with HostPath - simpler):

cd terraform
terraform init
terraform apply -var="use_hostpath=true"

This uses host directories for storage - no PVCs required.

For production (with PVCs - recommended):

cd terraform
terraform init
terraform apply

This will:

  1. Create the openclaw namespace
  2. Generate a gateway token
  3. Create PVCs for config and workspace (or use hostPath if enabled)
  4. Deploy the gateway
  5. Create a LoadBalancer service

Important: The onboarding job must run first to initialize config. Use one of these approaches:

  • Use setup.sh for automated setup with proper ordering
  • Or manually run onboarding before the first gateway deployment (see below)

Configuration

Basic Configuration

Create a terraform.tfvars file:

namespace = "openclaw"
container_image = "ghcr.io/openclaw/openclaw:latest"
busybox_image = "busybox:latest"
gateway_replicas = 1
service_type = "LoadBalancer"

Node Scheduling

node_selector = {
  "openclaw-enabled" = "true"
  "node.kubernetes.io/instance-type" = "t3.large"
}

Storage

PVC (default for production):

use_hostpath = false
config_storage_size = "5Gi"
workspace_storage_size = "20Gi"

HostPath (for development):

use_hostpath = true
fix_hostpath_permissions = true
config_hostpath = "/var/lib/openclaw/config"
workspace_hostpath = "/var/lib/openclaw/workspace"

When using hostPath, a fix-permissions init container will automatically:

  • Set ownership to uid 1000 (the "node" user in the container)
  • Set permissions to 700 (owner-only access)

This eliminates the need to manually configure permissions on host nodes.

Service Type

service_type = "NodePort"  # Options: LoadBalancer, NodePort, ClusterIP

Gateway Ports

gateway_port = 18789  # Gateway service port
bridge_port  = 18790  # Bridge service port
gateway_host_port = 0  # Host port to expose on node (0 = disabled)

Additional hostPath Mounts

For mounting additional host directories into the gateway container:

gateway_additional_hostpath_mounts = [
  {
    name       = "custom-tools"
    host_path  = "/opt/tools"
    mount_path = "/tools"
    read_only  = true
    type       = "DirectoryOrCreate"  # Optional, default: "DirectoryOrCreate"
  }
]

Important Notes:

  • Additional mounts work with both use_hostpath = true and use_hostpath = false (PVC) storage modes
  • Additional mounts are only mounted on the main gateway container, not on init containers
  • The type field accepts any valid Kubernetes hostPath type (e.g., Directory, DirectoryOrCreate, File, FileOrCreate)

Claude AI Integration (Optional)

claude_ai_session_key   = "your-session-key"
claude_web_session_key  = "your-web-session-key"
claude_web_cookie       = "your-cookie"

Browserless Chrome (Optional)

For browser automation capabilities, deploy the browserless Chrome service:

create_browserless = true
browserless_image  = "ghcr.io/browserless/chromium:latest"
browserless_replicas = 1
browserless_port   = 3000

Shared Memory (shm) Configuration:

The browserless container mounts a shared memory volume (/dev/shm) in memory for optimal Chrome performance. Configure the size based on your workload:

browserless_shm_size = "2Gi"  # Default: "1Gi"

Larger values (2-4Gi) are recommended for running multiple concurrent browser tabs or processing large pages.

The browserless service uses a token-based authentication. The token is automatically generated by Terraform and stored in a Kubernetes Secret.

The browserless service will be available within the cluster at:

  • Service name: openclaw-browserless
  • Port: 3000
  • URL: http://openclaw-browserless.openclaw:3000

To retrieve the token:

terraform output browserless_token

SearXNG Local Search Engine (Optional)

SearXNG is a privacy-respecting, self-hosted metasearch engine that aggregates results from multiple search engines.

Enable SearXNG:

create_searxng = true

Storage Options:

PVC (default for production):

create_searxng = true
searxng_config_storage_size = "100Mi"
searxng_data_storage_size   = "500Mi"

HostPath (for development):

create_searxng = true
use_hostpath = true
searxng_config_hostpath = "/var/lib/openclaw/searxng/config"
searxng_data_hostpath   = "/var/lib/openclaw/searxng/data"

Accessing SearXNG:

Once deployed, SearXNG is available within the cluster at:

http://openclaw-searxng.<namespace>.svc.cluster.local:8080

Or from other pods in the same namespace:

http://openclaw-searxng:8080

The SearXNG secret key is auto-generated. Retrieve it with:

terraform output searxng_secret

Qdrant Vector Database (Optional)

Qdrant is a high-performance vector database for AI applications, enabling similarity search and vector embeddings storage.

Enable Qdrant:

create_qdrant = true

Storage Options:

PVC (default for production):

create_qdrant = true
qdrant_config_storage_size = "100Mi"
qdrant_storage_size        = "5Gi"

HostPath (for development):

create_qdrant = true
use_hostpath = true
qdrant_config_hostpath  = "/var/lib/openclaw/qdrant/config"
qdrant_storage_hostpath = "/var/lib/openclaw/qdrant/storage"

Accessing Qdrant:

Once deployed, Qdrant is available within the cluster at:

  • HTTP API: http://openclaw-qdrant.<namespace>.svc.cluster.local:6333
  • gRPC API: http://openclaw-qdrant.<namespace>.svc.cluster.local:6334

Or from other pods in the same namespace:

  • HTTP: http://openclaw-qdrant:6333
  • gRPC: http://openclaw-qdrant:6334

Qdrant Authentication:

Qdrant uses an API key for authentication. The API key is auto-generated. Retrieve it with:

terraform output qdrant_api_key

When making requests, include the API key in the header:

curl -H "api-key: YOUR_API_KEY" http://openclaw-qdrant:6333/collections

Ports:

  • HTTP API: 6333 (configurable via qdrant_http_port)
  • gRPC API: 6334 (configurable via qdrant_grpc_port)

LiteLLM Proxy (Optional)

LiteLLM is a lightweight LLM proxy server that provides a unified OpenAI-compatible API for 100+ LLM providers. It supports load balancing, cost tracking, and logging.

Enable LiteLLM:

create_llmlite = true

Storage Options:

PVC (default for production):

create_llmlite = true
llmlite_config_storage_size = "100Mi"

HostPath (for development):

create_llmlite = true
use_hostpath = true
llmlite_config_hostpath = "/var/lib/openclaw/llmlite/config"

Accessing LiteLLM:

Once deployed, LiteLLM is available within the cluster at:

http://openclaw-llmlite.<namespace>.svc.cluster.local:4000

Or from other pods in the same namespace:

http://openclaw-llmlite:4000

LiteLLM Configuration:

LiteLLM requires a config.yaml file to define model configurations. You can create this configuration by:

  1. Exec into the pod:

    kubectl exec -it -n openclaw deployment/openclaw-llmlite -- /bin/bash

  2. Edit /app/config.yaml with your model configurations:

    model_list:
  - model_name: azure-gpt-4o
    litellm_params:
      model: azure/<your-azure-model-deployment>
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2025-01-01-preview"

  3. Set environment variables:

    export AZURE_API_BASE="https://your-resource.openai.azure.com/"
export AZURE_API_KEY="your-api-key"

LiteLLM Authentication:

LiteLLM uses a master key for authentication. The master key is auto-generated. Retrieve it with:

terraform output llmlite_master_key

When making requests, include the master key in the Authorization header:

curl -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "azure-gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}' \
  http://openclaw-llmlite:4000/chat/completions

Database Support (Optional):

LiteLLM can optionally use a database for persistent storage of logs, spend tracking, and virtual keys:

llmlite_database_url = "postgresql://user:password@host:5432/litellm"

Port:

  • API: 4000 (configurable via llmlite_port)

Run Onboarding

Option 1: Sequential Apply (Simplest for Initial Setup)

Run onboarding first, then deploy gateway:

# Step 1: Create onboarding job only
terraform apply -var="create_onboarding_job=true" -var="create_gateway_deployment=false"

# Step 2: Complete onboarding interactively
kubectl attach -n $(terraform output namespace) openclaw-onboarding -i -c onboard

When prompted during onboarding:

  • Gateway bind: lan
  • Gateway auth: token
  • Gateway token: (use the output from terraform output gateway_token)
  • Tailscale exposure: Off
  • Install Gateway daemon: No
# Step 3: Clean up job (it will complete on its own, but you can delete)
kubectl delete job -n $(terraform output namespace) openclaw-onboarding --ignore-not-found=true

# Step 4: Deploy gateway (now with initialized config)
terraform apply -var="create_onboarding_job=false" -var="create_gateway_deployment=true"

Option 2: Manual with Terraform

terraform apply -var="create_onboarding_job=true"
kubectl attach -n $(terraform output namespace) openclaw-onboarding -i -c onboard

When prompted during onboarding:

  • Gateway bind: lan
  • Gateway auth: token
  • Gateway token: (use the output from terraform output gateway_token)
  • Tailscale exposure: Off
  • Install Gateway daemon: No

IMPORTANT: Complete onboarding before the gateway starts.

Option 3: Existing Config (Production)

For production or greenfield deployments with existing config:

  1. Copy existing config to PVCs (create a temporary job or use kubectl cp)
  2. Set create_onboarding_job = false
  3. Run terraform apply

Commands

Note: CLI Commands Feature

The cli_commands variable is defined but not yet implemented in the current version. To run CLI commands, use kubectl exec or the tools.sh script:

kubectl exec -n openclaw deployment/openclaw-gateway -- node dist/index.js providers status
# or
./tools.sh providers status

Initial Deployment

terraform init
terraform plan
terraform apply

Check Status

terraform output
kubectl get pods -n $(terraform output namespace)
kubectl get svc -n $(terraform output namespace)

Update Configuration

terraform apply

Scale Gateway

terraform apply -var="gateway_replicas=3"

Destroy Resources

terraform destroy

Configuration Changes

You can update any configuration variable and re-apply:

# Change container image
terraform apply -var="container_image=my-registry/openclaw:v2.0"

# Switch to hostPath
terraform apply -var="use_hostpath=true"

# Update namespace
terraform apply -var="namespace=openclaw-prod"

Outputs

After deployment, Terraform outputs:

terraform output namespace           # openclaw
terraform output gateway_token       # [sensitive - use terraform output gateway_token ]
terraform output gateway_service     # Service name and type
terraform output storage_backend     # PVC or hostPath
terraform output storage_config_info # Storage configuration details
# PVC outputs (null if using hostPath):
terraform output config_pvc          # openclaw-config-pvc
terraform output workspace_pvc       # openclaw-workspace-pvc
# hostPath outputs (null if using PVC):
terraform output config_hostpath
terraform output workspace_hostpath
# Optional services:
terraform output browserless_token   # [sensitive] Browserless auth token
terraform output searxng_secret      # [sensitive] SearXNG secret key
terraform output qdrant_api_key      # [sensitive] Qdrant API key
terraform output qdrant_service      # Qdrant service endpoint
terraform output llmlite_master_key  # [sensitive] LiteLLM master key
terraform output llmlite_service     # LiteLLM service endpoint

Provider Setup

Using Default Kubeconfig

Terraform will use ~/.kube/config by default. Set kubeconfig_path if needed:

kubeconfig_path = "/path/to/kubeconfig"

Using Environment Variables

export KUBECONFIG=/path/to/kubeconfig

Troubleshooting

Job Stuck in Pending

kubectl get job -n $(terraform output namespace)
kubectl describe job <job-name> -n $(terraform output namespace)

PVC Pending

kubectl get pvc -n $(terraform output namespace)
# Ensure your cluster has a default StorageClass

Gateway Pod Not Starting

kubectl logs -n $(terraform output namespace) openclaw-gateway
kubectl describe pod -n $(terraform output namespace) -l app=openclaw-gateway

Examples

Minimal Deployment

# terraform.tfvars
namespace = "openclaw"

Production Deployment

# terraform.tfvars
namespace               = "openclaw"
container_image         = "ghcr.io/openclaw/openclaw:latest"
busybox_image           = "busybox:latest"
gateway_replicas        = 3
service_type            = "LoadBalancer"
config_storage_size     = "10Gi"
workspace_storage_size  = "50Gi"
node_selector = {
  "openclaw-enabled"   = "true"
  "node.kubernetes.io/instance-type" = "c5.xlarge"
}

Development Deployment

# terraform.tfvars (with PVCs)
namespace               = "openclaw-dev"
container_image         = "my-registry/openclaw:dev"
gateway_replicas        = 1
service_type            = "NodePort"
create_onboarding_job   = true
# terraform.tfvars (with hostPath - simpler for development)
namespace               = "openclaw-dev"
container_image         = "my-registry/openclaw:dev"
gateway_replicas        = 1
service_type            = "NodePort"
use_hostpath            = true
fix_hostpath_permissions = true
config_hostpath         = "/var/lib/openclaw/config"
workspace_hostpath      = "/var/lib/openclaw/workspace"
create_onboarding_job   = false  # Set to true for initial setup workflow
create_gateway_deployment = true

With Optional Services

# terraform.tfvars (with browserless, SearXNG, and Qdrant)
namespace               = "openclaw"
container_image         = "ghcr.io/openclaw/openclaw:latest"
gateway_replicas        = 1
service_type            = "LoadBalancer"

# Enable browserless Chrome
create_browserless      = true
browserless_replicas    = 1
browserless_shm_size    = "2Gi"

# Enable SearXNG search engine
create_searxng          = true
searxng_replicas        = 1
searxng_config_storage_size = "100Mi"
searxng_data_storage_size   = "500Mi"

# Enable Qdrant vector database
create_qdrant           = true
qdrant_replicas         = 1
qdrant_config_storage_size = "100Mi"
qdrant_storage_size     = "5Gi"

# Enable LiteLLM proxy
create_llmlite          = true
llmlite_replicas        = 1
llmlite_config_storage_size = "100Mi"
# Optional: Database URL for persistent storage
# llmlite_database_url = "postgresql://user:password@host:5432/litellm"

# Optional: Expose LiteLLM on node IP (0 = disabled, >0 = port number)
# llmlite_host_port = 0

## Documentation

- Full docs: https://docs.openclaw.ai
- Providers: https://docs.openclaw.ai/providers
- Gateway configuration: https://docs.openclaw.ai/configuration

## Migrating from Previous Structure

If you have an existing Terraform state from the previous (flat) project structure, follow these steps to migrate:

### Quick Migration (Recommended)

Use the Makefile target for automatic migration:

```bash
make migrate-state

This will:

  1. Move terraform.tfstate to terraform/
  2. Move terraform.tfstate.backup to terraform/ (if exists)
  3. Move .terraform/ directory to terraform/
  4. Move .terraform.lock.hcl to terraform/
  5. Move terraform.tfvars to terraform/
  6. Run terraform init in the new location

Option 1: Move State File (Local State)

If you're using local state (default), simply move the state file to the new location:

# From the repository root
mv terraform.tfstate terraform/
mv terraform.tfstate.backup terraform/  # If exists
mv .terraform terraform/                 # Move the .terraform directory
mv .terraform.lock.hcl terraform/        # Move the lock file

# Then reinitialize
cd terraform
terraform init

Option 2: Re-initialize (Remote State)

If you're using remote state (e.g., S3, Consul, Terraform Cloud):

# Navigate to the new terraform directory
cd terraform

# Initialize (will configure the backend)
terraform init

# Terraform will detect the existing remote state

Option 3: Fresh State (Development/Testing)

For development environments where you want to start fresh:

# From the repository root
rm -f terraform.tfstate terraform.tfstate.backup
rm -rf .terraform

# Initialize and apply
cd terraform
terraform init
terraform apply

Note: This will cause Terraform to recreate all resources. Only use this for development environments.

Verifying the Migration

After migrating, verify everything is working:

cd terraform
terraform plan

The plan should show "No changes" if the migration was successful.