Skip to content

architecture.md

Latest commit

 

History

History
324 lines (254 loc) · 9.15 KB

architecture.md

File metadata and controls

324 lines (254 loc) · 9.15 KB
Raw

Architecture

5-Spot is built as a Kubernetes controller using the kube-rs framework.

High-Level Architecture

flowchart TB
    subgraph Kubernetes["Kubernetes Cluster"]
        API[Kubernetes API Server]
        
        subgraph CRDs["5-Spot CRDs"]
            SM[ScheduledMachine]
        end
        
        subgraph CAPI["Cluster API Resources"]
            Machine[Machine]
            Bootstrap[Bootstrap Config<br/>e.g., K0sWorkerConfig]
            Infra[Infrastructure<br/>e.g., RemoteMachine]
        end
        
        Node[Node]
    end
    
    subgraph Operator["5-Spot Controller"]
        Controller[Controller Loop]
        Reconciler[Reconciler]
        Scheduler[Schedule Evaluator]
        Creator[Resource Creator]
    end
    
    subgraph External["External"]
        PhysicalMachine[Physical Machine]
    end
    
    Controller -->|watch| API
    API --> SM
    Reconciler --> Scheduler
    Reconciler --> Creator
    Creator -->|create/delete| Bootstrap
    Creator -->|create/delete| Infra
    Creator -->|create/delete| Machine
    Machine --> Node
    Node -.->|joins| PhysicalMachine
    
    SM -->|owns| Bootstrap
    SM -->|owns| Infra
    SM -->|owns| Machine
Loading

Watch Topology

5-Spot is an event-driven controller — it reacts to Kubernetes API events rather than polling. The primary controller watches ScheduledMachine resources; two secondary .watches() on CAPI Machine and core Node trigger reconciles of the owning ScheduledMachine whenever a downstream resource changes.

flowchart LR
    subgraph Primary["Primary Watch"]
        SM[ScheduledMachine CR]
    end

    subgraph Secondary["Secondary Watches (.watches)"]
        M[CAPI Machine<br/><i>filtered by label</i><br/>5spot.eribourg.dev/scheduled-machine]
        N[Node<br/><i>cluster-wide</i>]
    end

    subgraph Mappers["Reverse Mappers (pure fns)"]
        MM[machine_to_scheduled_machine<br/><i>reads the label</i>]
        NM[node_to_scheduled_machines<br/><i>matches status.nodeRef.name<br/>against Controller Store</i>]
    end

    SM -->|ObjectRef| Q[Reconcile Queue]
    M --> MM --> Q
    N --> NM --> Q

    Q --> R[reconcile_scheduled_machine]
Loading

Why this matters. When CAPI populates status.nodeRef on a Machine, or an operator drains a Node via kubectl cordon, the reconciler is notified in under a second through the watch stream — there is no polling loop. The label filter on the Machine watch means we only receive events for Machines we created; the Node watch uses the Controller's own reflector store to map a Node event back to the ScheduledMachine whose status.nodeRef.name matches, so we never issue an extra API list per event.

Reverse mappers are pure functions in src/reconcilers/helpers.rs (machine_to_scheduled_machine, node_to_scheduled_machines), unit-tested in isolation. They return an empty Vec when no match is found — the controller treats empty as "no enqueue," so an unrelated Machine or Node event costs nothing beyond the label lookup.

Component Details

Controller

The main entry point that:

  • Watches for ScheduledMachine resource changes (primary watch)
  • Watches CAPI Machine resources filtered by the 5spot.eribourg.dev/scheduled-machine label (secondary watch)
  • Watches core Node resources cluster-wide (secondary watch)
  • Manages reconciliation queue
  • Handles multi-instance distribution via consistent hashing
  • Provides health and metrics endpoints

Reconciler

Implements the reconciliation loop:

  1. Fetch current ScheduledMachine state
  2. Check for kill switch or disabled schedule
  3. Evaluate schedule against current time (in configured timezone)
  4. Determine desired state (active/inactive)
  5. Create or delete CAPI resources as needed
  6. Update status, conditions, and references

Schedule Evaluator

Evaluates time-based schedules:

  • Parses day ranges (e.g., mon-fri, with wrap-around support)
  • Parses hour ranges (e.g., 9-17, with wrap-around support)
  • Handles timezone conversions using IANA timezone database
  • Determines if current time is within schedule

Resource Creator

Creates and manages CAPI resources from inline specs:

  • Creates Bootstrap resource from bootstrapSpec
  • Creates Infrastructure resource from infrastructureSpec
  • Creates CAPI Machine with references to both
  • Sets owner references for automatic garbage collection
  • Tracks created resource references in status

Resource Creation Flow

flowchart TD
    SM[ScheduledMachine] --> |contains| BS[bootstrapSpec<br/>inline config]
    SM --> |contains| IS[infrastructureSpec<br/>inline config]
    
    subgraph Creation["When Schedule Active"]
        BS --> |creates| BR[Bootstrap Resource<br/>e.g., K0sWorkerConfig]
        IS --> |creates| IR[Infrastructure Resource<br/>e.g., RemoteMachine]
        BR --> |referenced by| M[CAPI Machine]
        IR --> |referenced by| M
    end
    
    M --> |provisions| N[Node]
    
    subgraph Status["Status Updated"]
        SM -.-> |bootstrapRef| BR
        SM -.-> |infrastructureRef| IR
        SM -.-> |machineRef| M
        SM -.-> |nodeRef| N
    end
Loading

Reconciliation Flow

sequenceDiagram
    participant API as Kubernetes API
    participant Ctrl as Controller
    participant Rec as Reconciler
    participant Sched as Schedule Evaluator
    participant Res as Resource Creator
    
    API->>Ctrl: ScheduledMachine Event
    Ctrl->>Rec: Reconcile Request
    Rec->>API: Get ScheduledMachine
    
    alt killSwitch = true
        Rec->>Res: Delete all resources immediately
        Rec->>API: Update status (Terminated)
    else schedule.enabled = false
        Rec->>API: Update status (Disabled)
    else
        Rec->>Sched: Evaluate Schedule
        Sched-->>Rec: inSchedule: true/false
        
        alt inSchedule = true
            Rec->>Res: Ensure resources exist
            Res->>API: Create Bootstrap from bootstrapSpec
            Res->>API: Create Infrastructure from infrastructureSpec
            Res->>API: Create Machine with refs
            Rec->>API: Update status (Active)
        else inSchedule = false
            Rec->>Res: Remove resources
            Res->>API: Delete Machine
            Res->>API: Delete Bootstrap
            Res->>API: Delete Infrastructure
            Rec->>API: Update status (Inactive)
        end
    end
    
    Rec->>API: Requeue after interval
Loading

Multi-Instance Support

5-Spot supports running multiple instances for high availability:

  • Consistent Hashing: Resources are distributed based on name hash
  • Instance ID: Each instance has a unique ID (0 to N-1)
  • No Overlap: Each resource is managed by exactly one instance
  • Environment Variables: OPERATOR_INSTANCE_ID and OPERATOR_INSTANCE_COUNT
flowchart LR
    subgraph Resources
        R1[SM: worker-a]
        R2[SM: worker-b]
        R3[SM: worker-c]
        R4[SM: worker-d]
        R5[SM: worker-e]
        R6[SM: worker-f]
    end
    
    subgraph Instances
        I0[Instance 0]
        I1[Instance 1]
        I2[Instance 2]
    end
    
    R1 -->|hash % 3 = 0| I0
    R2 -->|hash % 3 = 1| I1
    R3 -->|hash % 3 = 2| I2
    R4 -->|hash % 3 = 0| I0
    R5 -->|hash % 3 = 1| I1
    R6 -->|hash % 3 = 2| I2
Loading

Owner References & Garbage Collection

5-Spot uses Kubernetes owner references for automatic cleanup:

flowchart TD
    SM[ScheduledMachine<br/>owner] 
    SM -->|ownerRef| B[Bootstrap Resource]
    SM -->|ownerRef| I[Infrastructure Resource]
    SM -->|ownerRef| M[CAPI Machine]
    
    subgraph GC["Garbage Collection"]
        D[Delete ScheduledMachine]
        D --> DB[Bootstrap deleted]
        D --> DI[Infrastructure deleted]
        D --> DM[Machine deleted]
    end
Loading

When a ScheduledMachine is deleted, Kubernetes automatically garbage collects all owned resources.

Data Flow

flowchart LR
    subgraph Input
        CR[ScheduledMachine CR]
        Time[Current Time]
        TZ[Timezone]
    end
    
    subgraph Processing
        SE[Schedule Evaluation]
        RC[Reconciliation]
    end
    
    subgraph Output
        Resources[CAPI Resources]
        Status[Status Update]
        Events[Kubernetes Events]
        Metrics[Prometheus Metrics]
    end
    
    CR --> SE
    Time --> SE
    TZ --> SE
    SE --> RC
    RC --> Resources
    RC --> Status
    RC --> Events
    RC --> Metrics
Loading

Error Handling

Error Type Handling Requeue
Transient API errors Automatic retry 30s with backoff
Schedule parse errors Status updated with error No requeue
Resource creation failures Retry with backoff Up to 5m max
Permanent errors Manual intervention required No automatic retry

Observability

Health Endpoints

  • /healthz - Liveness probe (port 8081)
  • /readyz - Readiness probe (port 8081)

Metrics

  • /metrics - Prometheus metrics (port 8080)
  • Reconciliation duration, success/failure counts
  • Resource counts by phase

Events

Kubernetes events are emitted for:

  • Machine creation/deletion
  • Schedule activation/deactivation
  • Errors and warnings

Related