Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
331 lines (265 loc) · 22.6 KB

ARCHITECTURE.md

File metadata and controls

331 lines (265 loc) · 22.6 KB
Raw

Elisa Architecture

Elisa is a kid-friendly IDE that orchestrates AI agent teams to build real software and hardware nuggets. Kids compose nugget specs using visual blocks (Blockly); the backend decomposes specs into task DAGs, executes them via the Claude Agent SDK, and streams results back in real-time.

System Topology

Electron main process (electron/main.ts)
  |-> Loads API key from encrypted store (OS keychain via safeStorage)
  |-> Picks a free port
  |-> Starts Express server in-process
  |-> Opens BrowserWindow -> http://localhost:{port}

frontend/ (React 19 + Vite)         backend/ (Express 5 + TypeScript)
+-----------------------+           +---------------------------+
| Blockly Editor        |  REST     | Express Server            |
| (BlockCanvas)         |---------->| POST /api/sessions/:id/*  |
|                       |           |                           |
| MissionControl        |  WS      | Orchestrator              |
| (TaskDAG, CommsFeed,  |<---------|  -> AutoTestMatcher        |
|  Metrics, Deploy)     |  events  |  -> MetaPlanner (Claude)   |
|                       |           |  -> MeetingTriggerWiring   |
| SystemPanel           |           |  -> PortalService (MCP/CLI)|
| (architecture         |           |  -> AgentRunner (SDK)      |
|  explorer)            |           |  -> TestRunner (pytest/node)|
| BottomBar             |           |  -> GitService (simple-git)|
| (Trace, Board, Learn, |           |  -> HardwareService       |
|  Progress, Health,    |           |  -> TeachingEngine         |
|  Tokens — contextual  |           |  -> HealthTracker          |
|  visibility)          |           |  -> TraceabilityTracker    |
|                       |           |  -> DeviceRegistry         |
|                       |           |     (plugin manifests)     |
|                       |           |                           |
| FlashWizardModal      |           |                           |
| (multi-device flash)  |           |                           |
+-----------------------+           +---------------------------+
                                              |
                                    runs agents via SDK query() API
                                    per task (async streaming)

In production, Express serves everything: /api/* (REST), /ws/* (WebSocket), and /* (built frontend static files). CORS is unnecessary (same-origin). In dev mode, the frontend runs on Vite (port 5173) with proxy to backend (port 8000), and CORS is enabled.

Monorepo Layout

elisa/
  package.json       Root package: Electron deps, build/dev/dist scripts
  electron/          Electron main process, preload, settings dialog
  frontend/          React SPA - visual block editor + real-time dashboard
  backend/           Express server - orchestration, agents, hardware
  scripts/           Build tooling (esbuild backend bundler)
  devices/           Device plugins (manifest + templates + prompts)
  docs/              Documentation (INDEX.md, guides, API reference, plans)

Root package.json manages Electron and build tooling. Frontend and backend remain independent Node.js projects with their own package.json.

Planning Mode Flow (PRD-004)

Planning Mode (conversational specification assistant):
  Idea (free text) + optional canvas context (mid-project)
    -> POST /api/sessions/:id/planning/start
    -> PlanningService -> Claude SDK (structured outputs, effort: medium)
    -> WS: planning_mode_started, planning_turn (streamed message)
    -> WS: planning_question (widget spec + mutation map)
    -> Structured answer: client-side mutation (no API round-trip)
    -> Free-text answer: Claude SDK call -> next question
    -> Readiness detected: planning_ready (goal solid, 2+ promises, portals, skills, deploy)
    -> POST /api/sessions/:id/planning/generate
    -> Final Claude SDK call -> CanvasBlockSpec (validated against PRD-003 primitives)
    -> planToBlocks: Blockly workspace population (merge for mid-project)
    -> Plan + conversation persisted to .elisa/plan.json

Data Flow: Build Session Lifecycle

1. User arranges blocks in Blockly editor
2. Click GO -> blockInterpreter converts workspace to NuggetSpec JSON
3. POST /api/sessions (create) -> POST /api/sessions/:id/start (with spec)
4. Backend Orchestrator.run():
   a. AUTO-TEST: autoMatchTests() auto-generates behavioral tests for when_then
                 requirements missing test_id (Explorer level only; no-op at Builder/Architect)
   b. PLAN:     MetaPlanner calls Claude API to decompose spec into task DAG
                MetaPlanner auto-selects game framework (Phaser/p5/Three.js) if not specified
                Multi-file task planning enforced: each task owns one file, no concurrent edits
   c. FRAMEWORK: Copy selected framework lib to workspace lib/ (if applicable)
   d. MEETINGS: evaluateAndInvite('plan_ready') checks meeting triggers
                (currently no meetings fire at plan_ready; all moved to per-task/deploy events)
   e. TRACE:    Build traceability map from spec requirements <-> behavioral tests
   f. PORTALS:  Initialize MCP/CLI portal adapters if spec declares portals
   g. EXECUTE:  Streaming-parallel pool (Promise.race, up to 3 concurrent tasks)
                Each agent gets: role prompt + framework context + task description + context from prior tasks
                Agent output streams via SDK -> WebSocket events to frontend
                Git commit after each completed task (serialized via mutex)
                Token budget tracked per agent; warning at 80%, halt on exceed
                Per-task meeting triggers: design review (task_starting), mid-build
                  meetings at 25%/50%/60% completion (task_completed)
   h. TEST:    TestRunner executes pytest/node, parses results + coverage
               Traceability tracker updates requirement-test status
   i. GATE:    Test pass rate gate -- auto-fix if below threshold
               (explorer: no gate, builder: 50%/1 fix, architect: 80%/2 fixes)
   j. HEALTH:  HealthTracker computes score (0-100) and grade (A-F)
               HealthHistoryService persists trend data for Architect level
   k. DEPLOY:  Surface before_deploy rules as deploy_checklist event
               evaluateAndInvite('deploy_started') for device/web deploy meetings
               If web: build -> find serve dir -> start local HTTP server -> open browser
               If devices: resolveDeployOrder -> flash wizard per device
               If CLI portals: execute via CliPortalAdapter (no shell)
   l. COMPLETE: evaluateAndInvite('session_complete') -> Architecture agent meeting
5. session_complete event with summary
6. Post-build actions (optional):
   - FIX: POST /api/sessions/:id/fix with bugReport -> Orchestrator.runFix()
          creates targeted fix task -> re-runs tests -> emits fix_* events
   - LAUNCH: POST /api/sessions/:id/launch -> finds index.html in workspace
             (dist/ > build/ > public/ > src/ > root) -> spawns local serve
             process -> returns preview URL (no rebuild required)
   - CHAT: POST /api/sessions/:id/chat with { message }
           -> IterativeChatService.processMessage()
           -> builds context (file manifest + structural digest + conversation history)
           -> runs agent via SDK query() against existing workspace
           -> detects changed files, re-runs tests if present
           -> emits chat_processing, chat_agent_output, chat_response,
              chat_tests_completed, chat_preview_refresh events
           -> session stays in 'done' state (loops for multiple chat turns)

Human gates can pause execution at any point, requiring user approval via REST endpoint.

Composition Flow (PRD-003 Phase 2)

Workspace mode toggle: 'specify' (default) | 'compose'
  -> Compose mode loads composition toolbox (nugget_ref blocks only)
  -> User arranges nugget blocks spatially (vertical = sequential, side-by-side = parallel)
  -> nuggetInterpreter converts layout to ComposeRequest
  -> Composition review modal shows arrangement before build
  -> interfaceInference auto-infers provides/requires from NuggetSpec
  -> CompositionService.compose() merges nuggets + resolves interfaces
  -> Composition workspace persisted as composition.json in .elisa file

Shipped nuggets (Dashboard, API, Sensor, Gateway) available out of the box via shippedNuggets.ts.

Device Plugin System

Device plugins live in devices/<plugin-id>/ and follow a manifest-driven convention:

  • device.json — Zod-validated manifest (board info, capabilities, Blockly blocks, deploy config)
  • prompts/agent-context.md — Injected into builder agent prompts for hardware-specific guidance
  • templates/ — MicroPython templates for code generation
  • lib/ — Shared MicroPython libraries flashed alongside user code

The DeviceRegistry service loads all plugins at startup, validates manifests, and provides:

  • Block definitions for dynamic Blockly registration in the frontend
  • Agent context injection via getAgentContext() in formatTaskPrompt()
  • Deploy ordering via resolveDeployOrder() (provides/requires DAG)

Communication Protocol

Channel Direction Purpose
REST client -> server Commands: create session, start build, fix, launch, gate responses, question answers
WebSocket server -> client Events: task progress, agent output, test results, teaching moments, errors

WebSocket path: /ws/session/:sessionId

In dev mode, Vite (port 5173) proxies /api/* and /ws/* to backend (port 8000). In production (Electron), Express serves the frontend statically on the same port -- no proxy needed.

Core Abstractions

NuggetSpec

JSON schema produced by blockInterpreter from Blockly workspace. Drives the entire pipeline. Contains: goal, requirements (with test traceability), style, agents, deployment target, workflow flags (feedback loops, system level, behavioral tests), skills, rules, portals, devices, runtime config (agent name, voice, greeting), knowledge (backpack sources, study mode), composition (provides/requires interfaces for Spec Graph), framework (optional: phaser/p5/threejs/none -- auto-selected by MetaPlanner if not set).

Task DAG

Directed acyclic graph of tasks with dependencies. Generated by MetaPlanner. Executed in topological order by Orchestrator. Uses Kahn's algorithm (utils/dag.ts).

Build Session

In-memory state for one execution run. Tracks: session ID, phase, tasks, agents, commits, events, teaching moments, test results, token usage. No database - everything lives in memory.

Agent Roles

  • Builder: Writes source code
  • Tester: Writes and runs tests
  • Reviewer: Reviews code quality
  • Custom: User-defined persona

Each agent runs via the Claude Agent SDK's query() API with role-specific system prompts injected from backend/src/prompts/.

State Machine

idle -> planning -> executing -> testing -> deploying -> done
                       ^                                   |    |
                   human gates (pause/resume via REST)     |  keep working
                                                           |    |
                                                           |    v
                                                           | design (iterative build)
                                                           |
                                                           +-> fix (POST /fix: targeted bug fix -> re-test)
                                                           +-> launch (POST /launch: serve without rebuild)
                                                           +-> chat (POST /chat: iterative conversation loop, stays in done)

Note: `reviewing` is a transient state during human gate pauses within execution, not a separate pipeline phase. Reviewer agents execute as tasks within the execute phase. During execution, HITL checkpoints may pause task processing for kid approval (visual, choice, or progress checkpoints).

Key Patterns

  • Event-driven UI: All frontend state updates flow through WebSocket event handlers. No polling.
  • Agent isolation: Each agent task runs as a separate SDK query() call. No shared state between agents except via context summaries written to .elisa/ in the workspace.
  • Context chain: After each task, a summary is written to .elisa/context/nugget_context.md. Subsequent agents receive this as input, creating a chain of context.
  • Graceful degradation: Missing tools (git, pytest, mpremote, serialport) cause warnings, not crashes.
  • Bearer token auth: Server generates a random auth token on startup. All /api/* routes (except /api/health) require Authorization: Bearer <token>. WebSocket upgrades require ?token=<token> query param. In Electron, token is shared to renderer via IPC.
  • WebSocket heartbeat + send queue: Server sends protocol-level ping frames every 30s via ws.ping(). Connections missing a pong cycle are terminated. sendEvent() serializes all sends through a per-session FIFO queue with setImmediate yields between frames, preventing burst-flooding the Vite dev proxy even from concurrent fire-and-forget callers.
  • Content safety: All agent prompts include a Content Safety section enforcing age-appropriate output (ages 8-14). User-controlled placeholder values are sanitized before prompt interpolation. Runtime responses are post-processed through a mandatory content filter (PII redaction, inappropriate topic blocking) before delivery.
  • Iterative chat: After build completes (done state), kids can chat with the agent to fix bugs or add features via POST /api/sessions/:id/chat. IterativeChatService runs agent against the existing workspace, detects file changes, re-runs tests, and refreshes the preview. Conversation history (last 20 turns) is carried forward. The session remains in done state throughout.
  • Kid-friendly HITL: During execution, checkpointService.shouldFireCheckpoint() evaluates each completed task. Three checkpoint types: visual (design keyword match), choice (agent-written decision file in .elisa/decisions/), progress (fires at ~33% and ~66% completion). Max 3 checkpoints per build (HITL_MAX_CHECKPOINTS_PER_BUILD). Checkpoint blocks execution via Promise until kid responds or 120s timeout (HITL_CHECKPOINT_TIMEOUT_MS). Kid responds via POST /api/sessions/:id/checkpoint with approve/reject/choice. CheckpointModal renders the checkpoint UI.
  • Abort propagation: Orchestrator's AbortController signal is forwarded to each agent's SDK query() call. On cancel or error, agents are aborted immediately.
  • API key management: In dev, read from ANTHROPIC_API_KEY env var. In Electron, encrypted via OS keychain (safeStorage) and stored locally. When the key is changed at runtime via /api/internal/config, the Anthropic client singleton is reset so all services pick up the new key immediately. Child processes (test runners, flash scripts, builds) receive sanitized env without the API key.

Storage

  • Session state: In-memory Map<sessionId, Session> with optional JSON persistence for crash recovery. Cleanup timer (5 min) starts on WS connect; cancelled at build start, re-armed after build completes. Meeting accepts and fix/launch requests also reset the timer.
  • Session persistence: Builds use persistent project directories under ~/Elisa/projects/{slug}/. Orchestrator resolves dir via resolveProjectDir(goal) with collision-avoiding suffixes. Comprehensive metadata written to {nuggetDir}/.elisa/session.json at phase transitions (plan, execute, test, completion). Graceful shutdown: Electron before-quit calls POST /api/internal/shutdown -> store.checkpointAll(). GET /api/projects lists all saved projects. POST /api/sessions/:id/restore rehydrates a session from a project directory.
  • Workspace: Persistent project directory under ~/Elisa/projects/ (slug derived from goal). Contains generated code, tests, git repo, .elisa/ metadata, and design artifacts. Cleanup logic skips project directories to preserve user work.
  • localStorage: Workspace JSON, skills, and rules auto-saved in browser (elisa:workspace, elisa:skills, elisa:rules). Restored on page load.
  • Nugget files: .elisa zip format for export/import (workspace + skills + rules + generated code)
  • No database

Agent Runtime (PRD-001)

After a build deploys an agent to a target device (BOX-3, web, etc.), the Agent Runtime manages live conversations. Services live in backend/src/services/runtime/:

  • AgentStore — Provisions agent identities from NuggetSpec (system prompt, greeting, tools, API key)
  • ConversationManager — Per-agent conversation sessions with windowed turn history and TTL-based cleanup (30-min stale session sweep)
  • TurnPipeline — Core text loop: load identity + history -> Claude API -> store turn -> track usage
  • SafetyGuardrails — Safety prompt injected into every agent system prompt
  • KnowledgeBackpack — Per-agent document store with TF-IDF search for context injection
  • StudyMode — Spaced-repetition quiz generation from backpack sources
  • ContentFilter — Mandatory post-response PII redaction and inappropriate topic blocking (regex-based). Blocks responses containing inappropriate topics by replacing with agent's fallback response.
  • UsageLimiter — Tiered usage limits (free/basic/unlimited) with daily turn and monthly token caps
  • ConsentManager — COPPA parental consent tracking (full_transcripts / session_summaries / no_history)
  • DisplayManager — BOX-3 display command generator (idle, conversation, thinking, error, status, menu screens)

REST API: POST /v1/agents (provision), PUT /v1/agents/:id (update), DELETE /v1/agents/:id, POST /v1/agents/:id/turn/text, GET /v1/agents/:id/history, GET /v1/agents/:id/heartbeat.

Hardware Path

ESP32 support via serialport library:

  1. Detect boards by USB VID:PID (Heltec LoRa, ESP32-S3, CH9102)
  2. Compile MicroPython via py_compile
  3. Flash via mpremote or esptool (selected by FlashStrategy)
  4. Serial monitor at 115200 baud, streamed to frontend via WebSocket

Device Plugin Deploy

Multi-device builds use the device plugin system:

  1. DeployPhase.shouldDeployDevices() checks if spec has devices array
  2. resolveDeployOrder() sorts devices by provides/requires dependency DAG
  3. selectFlashStrategy(method) dispatches to MpremoteFlashStrategy or EsptoolFlashStrategy
  4. FlashWizardModal prompts the user to connect each device sequentially (flash_prompt event)
  5. Each device is flashed with files from its plugin manifest (flash_progress / flash_complete events)
  6. Shared libraries from devices/_shared/ are included automatically

BOX-3 Deploy (PRD-002)

The ESP32-S3-BOX-3 voice agent plugin uses esptool for binary firmware flash:

  1. Device manifest declares deploy.method: "esptool" and deploy.requires: ["agent_id", "api_key", "runtime_url"]
  2. If runtime_provision.required: true, RuntimeProvisioner.provision() creates an agent identity and returns credentials
  3. EsptoolFlashStrategy resolves esptool, detects serial port, and flashes the pre-built firmware binary
  4. Runtime config (agent_id, api_key, runtime_url) written as runtime_config.json alongside firmware
  5. On redeploy, redeployClassifier.classifyChanges() determines whether to reflash or just update config
  6. Art Agent meeting (art-agent type) triggers on deploy_started for BOX-3 theme customization

Agent Meeting System

AI agent personas pop up during builds to collaborate with kids via chat + interactive canvases.

Meeting Lifecycle

invited -> (accept) -> active -> (end) -> completed
        -> (decline) -> declined

Team Concept

Agents are either always-on defaults (fire without blocks) or opt-in (require team_member Blockly blocks).

Always-On Trigger
Buddy 25% tasks done
Scribe 50% tasks done
Blueprint session_complete
Bug Detective convergence_stalled
Pixel (design review) task_starting + design keywords

Opt-in agents: Marketing, Social Media, Styler, Pixel (art/theme), Interface Designer, and kid-defined custom agents.

Meeting Type Registry

Static types registered at startup via register<Type>Meeting(registry). Dynamic types created per-session via registerDynamic(sessionId, specs) for custom kid-defined agents with auto-assigned triggers based on canvas type.

Canvas Types

canvasRegistry.ts maps canvas type strings to React components. 13 types: agent-studio, blueprint, bug-detective, campaign, code-explorer, design-preview, explain-it, interface-designer, launch-pad, live-preview, test-dashboard, theme-picker, whiteboard.

Persistent Team Tab

TeamPanel provides always-available team access. TeamMemberList shows agents with invite badges. TeamConversation reuses ChatPanel + CanvasPanel via MeetingLayout. Auto-dismiss on MeetingInviteToast preserves invites in queue for Team tab.

Module-Level Documentation

Deeper context for each subsystem lives in CLAUDE.md files within each directory:

  • frontend/CLAUDE.md - Frontend architecture, component tree, state management
  • backend/CLAUDE.md - Backend architecture, services, API surface
  • backend/src/services/CLAUDE.md - Service responsibilities and interactions
  • frontend/src/components/CLAUDE.md - Component hierarchy and patterns

Electron Packaging

Elisa is distributed as an Electron desktop app. The build pipeline:

  1. npm run build:frontend -- Vite builds React SPA into frontend/dist/
  2. npm run build:backend -- esbuild bundles Express server into backend/dist/server-entry.js
  3. npm run build:mingit -- downloads MinGit for Windows (git + bash, ~39MB). Copies sh.exe to bash.exe. Skipped on macOS.
  4. npm run build:electron -- tsc compiles electron/main.ts and preload.ts
  5. npm run dist -- electron-builder packages into installer (NSIS on Windows, DMG on macOS). afterPack hook renames vendor/ -> node_modules/ and verifies cli.js exists for the Claude Agent SDK. MinGit included as extraResource on Windows.

Prerequisites: Node.js (LTS) must be installed on the system. The Windows installer bundles MinGit (git + bash) so Git is not required separately. If Node.js is missing, a non-blocking dialog prompts the user to install it on first launch. Python projects require a separate Python install (see #246). Web preview uses a built-in static file server (no npx serve dependency).

Dev mode (npm run dev at root): runs backend + frontend only (no Electron). npm run dev:electron runs backend, frontend, and Electron concurrently. Electron loads http://localhost:5173 (Vite HMR). Production: Electron loads http://localhost:{free port} where Express serves everything.