Nimbus Development Plan

Overview

Nimbus is a standalone Elixir daemon for provisioning and managing elastic CI runners. It provides standalone value as an open-source tool while creating a pathway to Tuist's managed offerings (similar to Grafana Cloud model).

Product Vision

Standalone Value Proposition: "Elastic CI runners for any Git forge, using your cloud provider"

Adoption Pathway

Stage	Deployment	What Tuist Provides	User Value
Stage 1: Discovery	Self-hosted daemon	OSS tool, docs	Cost savings + cloud credit usage
Stage 2: Simplification	Tuist-hosted daemon + control plane	Managed daemon, UI, monitoring	Less ops burden
Stage 3: Full Managed	Tuist infrastructure	Complete runner infrastructure	Zero-config convenience
Stage 4: Premium	Tuist platform	Runners + caching + insights	Full build optimization

Architecture

┌─────────────────────────────────────────────────────────┐
│            Tuist Control Plane (Optional)               │
│  - Multi-tenant daemon orchestration                    │
│  - Billing/metering                                     │
│  - UI dashboard                                         │
│  - Daemon hibernation (inactive → $0.001/month)        │
│  - Optimization recommendations                         │
└─────────────────┬───────────────────────────────────────┘
                  │ HTTP API
┌─────────────────▼───────────────────────────────────────┐
│         Nimbus Daemon (per tenant, isolated)            │
│  - Git forge integration (GitHub, GitLab, Forgejo)     │
│  - Cloud provider integration (AWS, GCP, Hetzner)      │
│  - Runner lifecycle management                         │
│  - Job queue management                                │
│  - Health monitoring & telemetry                       │
│  - RESTful API for control plane                       │
│  - Per-daemon storage (SQLite or provided DB)          │
└─────────────────┬───────────────────────────────────────┘
                  │ Provisions
┌─────────────────▼───────────────────────────────────────┐
│          Cloud Provider (AWS/GCP/Hetzner/etc)          │
│  - EC2/Compute Engine instances                        │
│  - Ephemeral environments                              │
│  - Auto-cleanup                                        │
└─────────────────────────────────────────────────────────┘

Economics (Hosted Mode)

Per-Tenant Daemon Cost (EC2 bin packing on t4g.large):

Active daemon (10% of tenants): ~$0.60/month
Hibernated daemon (90% of tenants): ~$0.001/month (state in S3)
Blended cost: ~$0.10/tenant/month

Pricing: $10/month per tenant → ~95% gross margin

Hibernation Strategy:

Daemon hibernates after 15 minutes of inactivity
State saved to S3 (~$0.001/month storage)
Cold start on job arrival: 1-2 seconds
50 daemons per t4g.large instance ($48/month)

Technical Architecture

Core Principles

Standalone Daemon: Each tenant gets an isolated Elixir process/daemon
Per-Daemon Storage: SQLite by default, or configurable storage connection
Unified Machine Model: Abstract away provider-specific details (e.g., AWS dedicated hosts)
Telemetry-driven: Emit events for all significant operations
Lean State: Minimize stored data - prefer querying cloud provider APIs with tags
API-First: RESTful API for control plane integration and management
Hibernation Support: Daemons can hibernate and restore state for cost efficiency

Module Structure

Nimbus
├── Nimbus.Application (OTP app, supervision tree)
├── Nimbus.Daemon (per-tenant daemon logic)
│   ├── Nimbus.Daemon.Supervisor (per-daemon supervision tree)
│   ├── Nimbus.Daemon.State (hibernation/restore)
│   └── Nimbus.Daemon.Config (per-daemon configuration)
├── Nimbus.API (HTTP API for control plane)
│   ├── Nimbus.API.Router (endpoint routing)
│   ├── Nimbus.API.Health (health checks)
│   └── Nimbus.API.Machines (machine management)
├── Nimbus.Storage (per-daemon storage)
│   ├── Nimbus.Storage.SQLite (default implementation)
│   └── Nimbus.Storage.Postgres (optional for larger deployments)
├── Nimbus.Provider (behavior for cloud providers)
│   ├── Nimbus.Provider.AWS (Phase 1)
│   ├── Nimbus.Provider.Hetzner (Future)
│   ├── Nimbus.Provider.GCP (Future)
│   └── Nimbus.Provider.Local (testing/development)
├── Nimbus.Forge (Git forge integrations)
│   ├── Nimbus.Forge.GitHub (Phase 1)
│   ├── Nimbus.Forge.GitLab (Future)
│   └── Nimbus.Forge.Forgejo (Future)
├── Nimbus.Machine
│   ├── Nimbus.Machine.Provisioner
│   ├── Nimbus.Machine.Lifecycle
│   └── Nimbus.Machine.SSH
├── Nimbus.Telemetry (instrumentation)
└── Nimbus.CLI (command-line interface for self-hosted)

Data Models

Note: Each daemon represents a single tenant, so tenant_id is a daemon-level configuration rather than a data model.

Daemon Configuration

# Loaded at daemon startup from config file or environment
%Nimbus.Daemon.Config{
  daemon_id: "daemon-uuid",
  tenant_id: "tenant-123",  # For control plane tracking
  tenant_name: "Acme Corp",

  storage: %{
    type: :sqlite,  # or :postgres
    path: "/var/lib/nimbus/nimbus.db"  # or connection string
  },

  api: %{
    port: 4000,
    host: "0.0.0.0",
    secret_key_base: "..."
  }
}

Provider Configuration

# Stored in per-daemon database, can have multiple providers
%ProviderConfig{
  id: "provider-456",
  name: "AWS US-East",
  type: :aws,  # :aws, :hetzner, :gcp, :azure
  credentials: %{
    access_key_id: "AKIA...",
    secret_access_key: "..."
  },
  config: %{
    region: "us-east-1",
    tags: %{"managed_by" => "nimbus", "tenant" => "tenant-123"}
  }
}

Forge Configuration

# Stored in per-daemon database, typically one per daemon
%ForgeConfig{
  id: "forge-789",
  type: :github,  # :github, :gitlab, :forgejo
  credentials: %{
    # GitHub App
    app_id: "123456",
    installation_id: "789012",
    private_key: "-----BEGIN RSA PRIVATE KEY-----..."
    # Or for GitLab/Forgejo
    # token: "glpat-..."
  },
  org: "tuist",  # or group/instance URL for GitLab/Forgejo
  webhook_secret: "..."
}

Machine

# Stored in per-daemon database
%Machine{
  id: "machine-uuid",
  provider_id: "provider-456",

  # Unified fields
  os: :macos,  # :macos, :linux
  arch: :arm64,  # :arm64, :x86_64
  state: :ready,  # :provisioning, :image_installing, :ready, :running, :stopping, :terminated
  ip_address: "1.2.3.4",
  ssh_public_key: "ssh-rsa ...",
  labels: ["macos", "xcode-15"],

  # Image configuration (optional - nil for machines without images)
  image: %{
    id: "ami-123abc",  # AMI ID, Docker image name, etc.
    type: :ami,  # :ami | :docker | nil
    state: :ready,  # :provisioning | :ready
    installed_at: ~U[2025-01-15 10:30:00Z]
  },

  # Timestamps (queried from provider, not stored)
  created_at: ~U[2025-01-15 10:00:00Z],

  # Provider-specific metadata
  provider_metadata: %{
    instance_id: "i-123abc",
    host_id: "h-456def",  # AWS Mac only
    minimum_allocation_hours: 24  # Provider-specific
  }
}

Notes:

Machine state includes :image_installing to track image setup progress
image field tracks software configuration (AMIs for macOS, Docker for Linux)
Linux machines with pre-built images transition directly to :ready
macOS machines go through :provisioning → :image_installing → :ready
Future: May extend to support multiple VMs per physical machine (2 VMs/Mac limit)

Storage Implementation

Each daemon has its own storage (SQLite by default, Postgres optional):

defmodule Nimbus.Storage do
  @callback list_providers() :: {:ok, [ProviderConfig.t()]} | {:error, term()}
  @callback get_provider(provider_id :: String.t()) :: {:ok, ProviderConfig.t()} | {:error, :not_found}
  @callback create_provider(attrs :: map()) :: {:ok, ProviderConfig.t()} | {:error, term()}
  @callback update_provider(provider_id :: String.t(), attrs :: map()) :: {:ok, ProviderConfig.t()} | {:error, term()}
  @callback delete_provider(provider_id :: String.t()) :: :ok | {:error, term()}

  @callback get_forge_config() :: {:ok, ForgeConfig.t()} | {:error, :not_found}
  @callback create_forge_config(attrs :: map()) :: {:ok, ForgeConfig.t()} | {:error, term()}
  @callback update_forge_config(attrs :: map()) :: {:ok, ForgeConfig.t()} | {:error, term()}

  @callback list_machines() :: {:ok, [Machine.t()]} | {:error, term()}
  @callback get_machine(machine_id :: String.t()) :: {:ok, Machine.t()} | {:error, :not_found}
  @callback create_machine(attrs :: map()) :: {:ok, Machine.t()} | {:error, term()}
  @callback update_machine(machine_id :: String.t(), attrs :: map()) :: {:ok, Machine.t()} | {:error, term()}
  @callback delete_machine(machine_id :: String.t()) :: :ok | {:error, term()}
end

Provider Behavior

Each cloud provider implements:

defmodule Nimbus.Provider do
  @callback provision(provider_config :: ProviderConfig.t(), specs :: map()) ::
    {:ok, Machine.t()} | {:error, term()}

  @callback terminate(provider_config :: ProviderConfig.t(), machine :: Machine.t()) ::
    :ok | {:error, term()}

  @callback can_terminate?(machine :: Machine.t()) ::
    {:ok, true} | {:error, :minimum_allocation_period, hours_remaining: integer()}

  @callback list_machines(provider_config :: ProviderConfig.t()) ::
    {:ok, [Machine.t()]} | {:error, term()}

  @callback get_machine(provider_config :: ProviderConfig.t(), machine_id :: String.t()) ::
    {:ok, Machine.t()} | {:error, term()}
end

Daemon HTTP API

The daemon exposes a RESTful API for control plane integration and management:

# Health & Status
GET  /health          → Daemon health check
GET  /status          → Daemon status (active/hibernated, uptime, etc.)

# Configuration (read-only in production, writable in dev/testing)
GET  /config          → Daemon configuration
GET  /providers       → List configured providers
GET  /forge           → Forge configuration (credentials masked)

# Machine Management
POST /machines        → Provision new machine
GET  /machines        → List all machines
GET  /machines/:id    → Get machine details
DELETE /machines/:id  → Terminate machine

# Hibernation (for control plane)
POST /hibernate       → Hibernate daemon (save state, shutdown)
POST /restore         → Restore from hibernation

# Telemetry
GET  /metrics         → Prometheus-compatible metrics

Machine Lifecycle Flow

1. Request machine via HTTP API or CLI
   POST /machines with specs (os, arch, provider_id, labels)
   ├─> Validate provider exists in daemon storage
   ├─> Get provider credentials from daemon storage
   └─> Call Provider.provision/2

2. Provider provisions infrastructure
   AWS Mac: Allocate dedicated host → Launch instance
   AWS Linux: Launch instance
   Hetzner: Create server
   ├─> Tag with daemon_id for discovery
   └─> Return Machine struct

3. Machine.Lifecycle sets up runner
   ├─> Wait for machine to be accessible
   ├─> SSH into machine
   ├─> Install dependencies (homebrew, etc.)
   ├─> Download Git forge runner agent
   ├─> Get registration token from Forge API
   ├─> Register runner with forge
   ├─> Initialize machine state file (SQLite on machine)
   └─> Emit telemetry: [:nimbus, :machine, :ready]

4. Runner operates (managed by Git forge)
   ├─> Forge assigns jobs
   └─> Runner executes jobs

5. Request termination via HTTP API or CLI
   DELETE /machines/:id
   ├─> Check can_terminate? (24h minimum for AWS Mac)
   ├─> Unregister runner from forge
   ├─> Terminate instance (and release host if needed)
   ├─> Remove from daemon storage
   └─> Emit telemetry: [:nimbus, :machine, :terminated]

Telemetry Events

Nimbus emits telemetry events for all significant operations:

# Machine lifecycle
[:nimbus, :machine, :provision_start]
[:nimbus, :machine, :provision_success]
[:nimbus, :machine, :provision_failure]
[:nimbus, :machine, :setup_start]
[:nimbus, :machine, :setup_success]
[:nimbus, :machine, :setup_failure]
[:nimbus, :machine, :ready]
[:nimbus, :machine, :terminate_start]
[:nimbus, :machine, :terminate_success]
[:nimbus, :machine, :terminate_failure]

# Forge operations
[:nimbus, :forge, :register_runner_start]
[:nimbus, :forge, :register_runner_success]
[:nimbus, :forge, :register_runner_failure]
[:nimbus, :forge, :unregister_runner_start]
[:nimbus, :forge, :unregister_runner_success]
[:nimbus, :forge, :unregister_runner_failure]

# Cloud provider operations
[:nimbus, :cloud_provider, :api_call_start]
[:nimbus, :cloud_provider, :api_call_success]
[:nimbus, :cloud_provider, :api_call_failure]

# SSH operations
[:nimbus, :ssh, :connect_start]
[:nimbus, :ssh, :connect_success]
[:nimbus, :ssh, :connect_failure]
[:nimbus, :ssh, :command_start]
[:nimbus, :ssh, :command_success]
[:nimbus, :ssh, :command_failure]

Each event includes metadata like tenant_id, machine_id, duration, error, etc.

Self-Hosted CLI Experience

For standalone/self-hosted deployments:

# Installation
brew install tuist/tap/nimbus
# or
curl -sSL https://get.nimbus.dev | sh

# Initialize daemon configuration
nimbus init
# Creates ~/.nimbus/config.toml with prompts for:
# - Daemon ID/name
# - Storage type (sqlite/postgres)
# - API port
# - Cloud provider credentials
# - Git forge configuration

# Start daemon
nimbus start
# Starts daemon on configured port (default: 4000)
# Logs to ~/.nimbus/logs/
# State in ~/.nimbus/nimbus.db (SQLite)

# Manage providers
nimbus provider add aws --name "AWS US-East" --region us-east-1
nimbus provider list
nimbus provider remove <id>

# Manage machines
nimbus machine provision --provider aws --os macos --arch arm64
nimbus machine list
nimbus machine get <id>
nimbus machine terminate <id>

# Daemon management
nimbus status       # Check daemon health
nimbus logs         # View daemon logs
nimbus stop         # Stop daemon gracefully
nimbus restart      # Restart daemon

Phase 1: MVP (Standalone Daemon)

Scope

Deployment Model: Self-hosted standalone daemon Cloud Provider: AWS EC2 Mac (mac2.metal on dedicated hosts) Git Forge: GitHub (via GitHub App) Storage: SQLite (per-daemon) API: HTTP REST API + CLI Features: Basic lifecycle, 24h minimum tracking, SSH-based setup

Implementation Tasks

1. Core Daemon Infrastructure

2. HTTP API

Set up Plug/Bandit web server
Implement Nimbus.API.Router (endpoint routing)
Implement Nimbus.API.Health (health checks)
Implement Nimbus.API.Machines (CRUD endpoints)
Implement Nimbus.API.Providers (read-only endpoints)
Implement Nimbus.API.Config (daemon config endpoint)
Add authentication/authorization (API keys)

3. CLI

Set up CLI framework (mix escript or Burrito)
Implement nimbus init (interactive config setup)
Implement nimbus start/stop/restart/status
Implement nimbus provider commands
Implement nimbus machine commands
Implement nimbus logs (tail daemon logs)
Add shell completions (bash/zsh/fish)

4. AWS Provider

Implement Nimbus.Provider.AWS
Handle EC2 dedicated host allocation
Handle mac2.metal instance provisioning
Tag resources with daemon_id and nimbus metadata
Implement machine discovery via AWS API
Implement can_terminate? with 24h check
Handle host + instance cleanup

5. GitHub Forge Integration

Implement GitHub App authentication
Implement runner registration token API
Implement runner registration API
Implement runner unregistration API
Handle API errors and retries

6. Machine Setup (SSH)

7. Testing

Unit tests for XDG module
Unit tests for Machine.Setup module (with mocked installers)
Unit tests for remaining core modules
Mocked AWS API tests
Mocked GitHub API tests
Integration tests (may require real AWS/GitHub sandbox)
CLI integration tests
API endpoint tests

8. Documentation

Module documentation (@moduledoc)
Function documentation (@doc)
Self-hosted deployment guide
Configuration file documentation
CLI command reference
API endpoint reference
Provider setup guides (AWS, GitHub)

Dependencies

# mix.exs dependencies

# HTTP Server & API
{:plug, "~> 1.15"},
{:bandit, "~> 1.0"},  # HTTP server
{:jason, "~> 1.4"},  # JSON encoding/decoding
{:cors_plug, "~> 3.0"},  # CORS support

# Storage
{:ecto_sql, "~> 3.11"},
{:ecto_sqlite3, "~> 0.14"},  # SQLite adapter
{:postgrex, "~> 0.17"},  # Optional: Postgres adapter

# Cloud Providers
{:ex_aws, "~> 2.5"},
{:ex_aws_ec2, "~> 2.0"},
{:hackney, "~> 1.18"},

# Git Forges
{:req, "~> 0.4"},  # HTTP client for GitHub/GitLab APIs

# SSH & Process Management
{:sshex, "~> 2.2"},  # For SSH operations
{:muontrap, "~> 1.5"},  # For process management

# Configuration & Validation
{:nimble_options, "~> 1.1"},  # Config validation
{:toml, "~> 0.7"},  # TOML config parsing

# Telemetry & Monitoring
{:telemetry, "~> 1.2"},
{:telemetry_metrics, "~> 0.6"},
{:telemetry_poller, "~> 1.0"},

# CLI (optional, for escript build)
{:owl, "~> 0.9"},  # CLI framework with interactive prompts

Phase 2: Control Plane & Hibernation

Scope

New Components:

Control plane orchestrator (Elixir/Phoenix app)
Daemon hibernation & restore logic
Multi-tenant daemon management
UI dashboard (Phoenix LiveView)
Billing/metering integration

Tasks

1. Daemon Hibernation

Implement Nimbus.Daemon.State.save() (persist to S3/storage)
Implement Nimbus.Daemon.State.restore() (load from storage)
POST /hibernate endpoint (graceful shutdown + save)
POST /restore endpoint (load state + restart)
Background hibernation after N minutes idle

2. Control Plane

Phoenix app for control plane
Daemon registry (track daemon IDs, status, endpoints)
Daemon lifecycle management (spawn, monitor, hibernate)
Proxy API requests to tenant daemons
LiveView dashboard (tenant overview, machines, costs)

3. Multi-Tenant Orchestration

EC2 instance management (bin packing daemons)
Daemon spawning via systemd/K8s
Health monitoring & auto-restart
Load balancing across orchestrator instances

Phase 3: Enhanced Features

Warm Pools

Pre-provision N machines per tenant/provider
Maintain minimum pool size
Automatic replenishment
Pool sizing strategies

Additional Cloud Providers

Hetzner dedicated servers
AWS EC2 Linux instances
GCP (future)
Azure (future)

Additional Git Forges

GitLab (via personal access token or OAuth)
Forgejo (similar to GitLab)

Automatic Lifecycle Management

Auto-terminate idle machines (after minimum period)
Auto-provision based on queue depth
Cost optimization strategies
Usage analytics and reporting

Advanced Features

Completed

Project Setup

Set up mise.toml with lockfile enabled
Created Elixir application structure using mix
Created CLAUDE.md and README.md documentation
Designed architecture and data models
Set up Quokka (code formatter/linter) and Mimic (mocking library)
Configured .formatter.exs with Quokka plugin
Generated .credo.exs configuration
Updated CLAUDE.md with development workflow and pre-commit checklist

Core Infrastructure (Phase 1)

Define core data structures (Tenant, Provider.Config, Forge.Config, Machine)
Define Nimbus.Storage behavior with delegation functions
Define Nimbus.Provider behavior with delegation functions
Implement Nimbus.Provider.Local for development/testing
Set up Nimbus.Telemetry with event helpers and convenience macro
Add telemetry dependency (~> 1.2)
Add MuonTrap for process management (~> 1.5)
Implement public API (Nimbus module) with provision/terminate/list/get functions

In Progress

Open Questions

SSH Key Management: Should we support multiple SSH keys per tenant for different purposes?
GitHub Runner Scope: Organization-level vs repository-level runners?
Error Handling: Retry strategies for transient failures (AWS API throttling, SSH timeouts)?
Logging: Use Logger or rely purely on telemetry?
Machine Naming: Convention for naming machines/runners (e.g., "nimbus-{tenant}-{uuid}")?
macOS VM Concurrency Limits:
- Legal/Licensing: Apple's EULA restricts macOS virtualization - only allowed on Apple hardware, and with specific conditions:
  - macOS can be virtualized on Apple Silicon using macOS 12.0.1+ (Virtualization.framework)
  - Up to 2 VM instances per physical Mac
  - Each VM requires a separate license
- AWS EC2 Mac Limitations: AWS mac2.metal is bare metal (not virtualized), so Apple's 2-VM limit doesn't apply. However:
  - 24-hour minimum allocation per dedicated host
  - One instance per host (bare metal)
  - No concurrent VMs on same host - each tenant gets full dedicated host
- Question: Do we need to track/enforce any concurrency limits per tenant? Or rely on AWS account limits?
macOS Image Management:
- Problem: Unlike Linux (Docker images), macOS images are installed separately:
  - AWS provides AMIs (Amazon Machine Images) with pre-installed macOS versions
  - Additional software (Xcode, simulators) must be installed after provisioning via SSH
  - Images are large (50GB+) and installation is slow (30+ minutes for Xcode)
- Modeling Approach (DECIDED):
  - Option A (MVP): Add image_id to specs, track image lifecycle in Machine struct
```
specs = %{
  os: :macos,
  arch: :arm64,
  image_id: "ami-123abc",  # macOS 14.2 base
  image_type: :ami,
  setup_script: "install_xcode_15.sh"  # Run via SSH after provision
}

# Machine struct tracks image state
%Machine{
  state: :image_installing,  # or :ready when complete
  image: %{
    id: "ami-123abc",
    type: :ami,
    state: :provisioning,  # transitions to :ready
    installed_at: nil
  }
}
```
- Design Decision:
  - Machine struct now includes image field to track software configuration
  - New machine state: :image_installing (between :provisioning and :ready)
  - Linux: Can transition directly to :ready (pre-built images)
  - macOS: Goes through image installation phase (Xcode, etc.)
- Future: Can split into separate Host/VM concepts when supporting 2 VMs per physical Mac
- Question: Should we cache/reuse provisioned machines with software pre-installed (warm pool), or always provision fresh?

Architectural Evolution

From Library to Standalone Daemon (January 2025)

Previous Architecture: Nimbus was designed as an Elixir library to be embedded into host applications (like Tuist server), with storage abstraction expecting the integrator to provide implementation.

New Architecture: Nimbus is now a standalone per-tenant daemon with:

Standalone value: Useful independently for elastic CI runners
Grafana Cloud model: Can be self-hosted or managed by Tuist
Per-daemon isolation: Each tenant gets isolated BEAM process
Built-in storage: SQLite by default, Postgres optional
HTTP API + CLI: RESTful API and command-line interface
Hibernation support: Cost-efficient for hosted deployments

Rationale:

Library-only approach has no standalone value
Hard to market/grow adoption without independent utility
Per-tenant daemon enables hosted SaaS model (like Grafana Cloud)
Creates natural pathway: OSS → Managed control plane → Full Tuist platform
Better economics: ~$0.10/tenant/month with hibernation

Migration Impact:

Old Nimbus.Storage behavior removed (storage now built-in)
Old Nimbus.provision_machine(tenant_id, ...) becomes daemon API: POST /machines
Configuration moves from host app to per-daemon config files
Tenant concept becomes implicit (one daemon = one tenant)

Notes

AWS EC2 Mac Specifics

AWS Mac dedicated hosts have 24-hour minimum allocation (billing constraint)
Bare metal instances (mac2.metal, mac2-m2.metal, mac2-m2pro.metal)
One instance per dedicated host (no concurrent VMs)
Uses Nitro System but not virtualized - direct hardware access
Must allocate dedicated host first, then launch instance on that host

macOS Licensing & Virtualization

Apple's EULA: macOS virtualization only on Apple hardware
Maximum 2 concurrent VMs per physical Mac (using Virtualization.framework)
AWS EC2 Mac is compliant (bare metal on Apple hardware)
Each macOS instance requires separate license

Image Management

macOS images distributed as AMIs (Amazon Machine Images)
Base OS only - additional software installed post-provision
Large images (50GB+) with slow installation times (30+ min for Xcode)
Xcode includes: IDE, SDKs, simulators, command-line tools (~40GB installed)

Architecture Notes

Machine discovery uses cloud provider tags instead of storing state
Integrator provides storage implementation and SSH keys
Nimbus manages complete lifecycle including forge integration

Machine State Persistence Strategy

Design Decision: State Lives on the Machine

We persist detailed machine state on the machine itself (via SSH-accessible file/database), rather than in the integrating application's storage. This aligns with our "lean state" principle.

Two-Layer State Model:

Basic State (from Cloud Provider API)
- Machine existence, IP address, running/stopped status
- Discovered via Provider.list_machines() and Provider.get_machine()
- Available immediately after provisioning
- Source of truth: Cloud provider tags/API
Detailed State (from Machine Filesystem)
- Image installation progress
- Runner registration status
- Setup step completion
- Application-specific metadata
- Available once SSH-accessible
- Source of truth: File/database on machine

State Persistence Options:

Option A: JSON File on Disk

/var/lib/nimbus/state.json

Pros: Simple, human-readable, no dependencies, easy to write
Cons: Poor queryability (must read entire file), harder to debug at scale
Example: cat state.json dumps everything, can't filter or query specific fields

Option B: SQLite Database (CHOSEN)

/var/lib/nimbus/state.db

Pros: Queryable via SSH, atomic writes, transactional, battle-tested
Cons: Binary format (but sqlite3 CLI is ubiquitous), need schema
Queryability is key for debugging provisioning issues

Example queries:

# Check failed steps
sqlite3 state.db 'SELECT * FROM setup_steps WHERE status = "failed"'

# See timing information
sqlite3 state.db 'SELECT step, started_at, completed_at FROM setup_steps'

# Check current state
sqlite3 state.db 'SELECT state, updated_at FROM machine_state'

Option C: etcd/Consul (Embedded)

Pros: Distributed, consistent, good for multi-VM scenarios
Cons: Heavy dependency, overkill for single machine
Only relevant for future multi-VM per physical machine scenarios

Decision: SQLite (Option B)

Queryability: Can SSH in and query specific state, filter results, check timing
Debugging: When setup fails, can quickly identify which step and when
Atomic writes: Built-in transaction support prevents corruption
Future-proof: Can add logs, metrics, multiple tables as needed
Ubiquitous: sqlite3 CLI is available on all Unix systems
The slight complexity of SQL schema is worth the debugging benefits

SQLite Schema (Draft):

-- Machine state table (singleton - one row)
CREATE TABLE machine_state (
  machine_id TEXT PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  provider_id TEXT NOT NULL,
  state TEXT NOT NULL,  -- provisioning, image_installing, ready, running, stopping, terminated
  created_at TEXT NOT NULL,  -- ISO8601 timestamp
  updated_at TEXT NOT NULL
);

-- Image installation state (singleton - one row if image is used)
CREATE TABLE image_state (
  id INTEGER PRIMARY KEY CHECK (id = 1),  -- Enforce single row
  image_id TEXT NOT NULL,
  image_type TEXT NOT NULL,  -- ami, docker
  state TEXT NOT NULL,  -- provisioning, ready
  started_at TEXT,
  installed_at TEXT
);

-- Setup steps tracking (multiple rows)
CREATE TABLE setup_steps (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  step TEXT NOT NULL,  -- install_homebrew, install_xcode, register_runner
  status TEXT NOT NULL,  -- pending, in_progress, completed, failed
  started_at TEXT,
  completed_at TEXT,
  error_message TEXT,
  seq INTEGER NOT NULL  -- Order of execution
);

CREATE INDEX idx_setup_steps_status ON setup_steps(status);
CREATE INDEX idx_setup_steps_seq ON setup_steps(seq);

-- Runner registration state (singleton - one row)
CREATE TABLE runner_state (
  id INTEGER PRIMARY KEY CHECK (id = 1),  -- Enforce single row
  registered BOOLEAN NOT NULL DEFAULT 0,
  runner_id TEXT,
  registration_token TEXT,  -- TODO: Encrypt?
  labels TEXT,  -- JSON array: ["macos", "xcode-15"]
  registered_at TEXT
);

Example Elixir representation after reading from SQLite:

# Read from SQLite via SSH, construct Machine struct
%Machine{
  id: "machine-uuid",
  tenant_id: "tenant-123",
  provider_id: "provider-456",
  state: :image_installing,
  # ... other fields from provider API ...

  # Enriched from SQLite:
  image: %{
    id: "ami-123abc",
    type: :ami,
    state: :provisioning,
    installed_at: nil
  },

  setup_progress: [
    %{step: "install_homebrew", status: :completed, started_at: "...", completed_at: "..."},
    %{step: "install_xcode", status: :in_progress, started_at: "...", completed_at: nil},
    %{step: "register_runner", status: :pending, started_at: nil, completed_at: nil}
  ]
}

Implementation Flow:

1. Provider.provision() creates machine in cloud
   → Returns Machine struct with basic info from provider API
   → State: :provisioning, SSH not accessible yet

2. Wait for SSH to become accessible (poll/retry)
   → Once accessible, initialize state persistence
   → Create /var/lib/nimbus/state.db (SQLite) with schema
   → Insert initial state: machine_id, tenant_id, provider_id, state: :provisioning

3. Begin setup process
   → Update state to :image_installing
   → Record setup steps as they progress
   → State file tracks: current step, progress, timestamps

4. Setup completes
   → Update state to :ready
   → Record completion timestamp

5. get_machine() / list_machines() enriches data
   → Query provider API for basic info (running, IP, etc.)
   → If SSH-accessible: query state.db and merge with provider data
   → If SSH-unavailable: return provider data only (fallback)
   → Example query: `ssh machine "sqlite3 /var/lib/nimbus/state.db 'SELECT * FROM machine_state'"`

Open Questions:

State initialization timing: Should we initialize state file immediately after SSH becomes available, or wait until setup begins?
State encryption: Should sensitive data (runner tokens, etc.) be encrypted in the state file?
State backup: Should we periodically sync state to integrator's storage for disaster recovery?
State format: JSON vs SQLite vs other? DECIDED: JSON with atomic writes (see above)
State location: /var/lib/nimbus/ vs /opt/nimbus/ vs home directory?
Handling missing state: If state file is missing/corrupted, do we recreate from provider API + reinitialize?
Local provider: How do we handle state for local machines without actual SSH? (Mock file in temp directory?)

Implementation Notes (For Later):

# SQLite operations via SSH
defmodule Nimbus.Machine.State do
  @state_db "/var/lib/nimbus/state.db"

  # Initialize database with schema
  def init_via_ssh(machine) do
    commands = [
      "mkdir -p /var/lib/nimbus",
      "sqlite3 #{@state_db} '#{create_schema_sql()}'"
    ]

    Enum.each(commands, fn cmd ->
      Nimbus.Provider.Local.exec_command(machine, cmd)
    end)
  end

  # Update machine state
  def update_state_via_ssh(machine, state) do
    sql = """
    UPDATE machine_state
    SET state = '#{state}', updated_at = datetime('now')
    WHERE machine_id = '#{machine.id}'
    """

    exec_sql_via_ssh(machine, sql)
  end

  # Read machine state
  def read_state_via_ssh(machine) do
    sql = "SELECT * FROM machine_state"

    case exec_sql_via_ssh(machine, sql) do
      {:ok, output} -> parse_sqlite_output(output)
      error -> error
    end
  end

  defp exec_sql_via_ssh(machine, sql) do
    cmd = "sqlite3 #{@state_db} \"#{escape_sql(sql)}\""
    Nimbus.Provider.Local.exec_command(machine, cmd)
  end
end

All database operations will be executed via SSH commands, not direct database connections.

Uh oh!

FilesExpand file tree

PLAN.md

Latest commit

History

PLAN.md

File metadata and controls

Nimbus Development Plan

Overview

Product Vision

Adoption Pathway

Architecture

Economics (Hosted Mode)

Technical Architecture

Core Principles

Module Structure

Data Models

Daemon Configuration

Provider Configuration

Forge Configuration

Machine

Storage Implementation

Provider Behavior

Daemon HTTP API

Machine Lifecycle Flow

Telemetry Events

Self-Hosted CLI Experience

Phase 1: MVP (Standalone Daemon)

Scope

Implementation Tasks

1. Core Daemon Infrastructure

2. HTTP API

3. CLI

4. AWS Provider

5. GitHub Forge Integration

6. Machine Setup (SSH)

7. Testing

8. Documentation

Dependencies

Phase 2: Control Plane & Hibernation

Scope

Tasks

1. Daemon Hibernation

2. Control Plane

3. Multi-Tenant Orchestration

Phase 3: Enhanced Features

Warm Pools

Additional Cloud Providers

Additional Git Forges

Automatic Lifecycle Management

Advanced Features

Completed

Project Setup

Core Infrastructure (Phase 1)

In Progress

Open Questions

Architectural Evolution

From Library to Standalone Daemon (January 2025)

Notes

AWS EC2 Mac Specifics

macOS Licensing & Virtualization

Image Management

Architecture Notes

Machine State Persistence Strategy