AI Orchestrator

AI-powered system controller that turns natural language into working software.

Say what you want built. The orchestrator generates it, runs it, and manages it — all from one place. No context-switching between chat, editor, terminal, and browser.

My Journey: Want the backstory? Read JOURNEY.md — how this project evolved from a naive terminal experiment into a cross-platform AI workbench, and the lessons learned along the way.

What It Does

Generates full applications from a single prompt (Python, web apps, CLIs, games, utilities)
Two AI controllers — Hummingbird (CLI) and Sparrow (GUI workbench)
Plugin system with 9 built-in plugins for system control, UI automation, backup, email, and more
StarkyGUI workbench — a unified desktop surface where apps, browser, and plugins all live as panels
Learns from you — teach it new command mappings that persist across sessions
Cross-platform — Windows, Linux, macOS

Quick Start

1. Clone & install

git clone https://github.com/YOUR_USERNAME/ai-orchestrator.git
cd ai-orchestrator
python setup.py --skip-key-setup --system-deps --ui-deps --yes

Or manually:

pip install -r requirements.txt

2. Set your OpenAI API key

Pick one method (first match wins):

Priority	Method	Location
1	Environment variable	`export OPENAI_API_KEY=sk-...`
2	System keyring	service `ai_orchestrator`, key `openai_api_key`
3	Config file	`~/.config/ai_orchestrator/openai_api_key`
4	Local file (dev only)	`cloud_agent/apikey.txt`

3. Run

# Interactive mode
python orchestrator.py

# One-shot prompt
python orchestrator.py "build me a todo app with a dark theme"

# Force CLI controller (Hummingbird)
python orchestrator.py --hummingbird "make a calculator"

Platform launchers:

# Windows
run_orchestrator.bat "make a snake game"
.\run_orchestrator.ps1 "make a snake game"

# Linux / macOS
./run_orchestrator.sh "make a snake game"

Architecture

┌──────────────────────────────────────────────────────────┐
│                    Natural Language Prompt                │
└────────────────────────┬─────────────────────────────────┘
                         │
                ┌────────▼────────┐
                │ System Controller│  (translates text → slash commands)
                │   (gpt-5-mini)  │
                └───┬─────────┬───┘
                    │         │
         ┌──────────▼──┐  ┌──▼──────────┐
         │ Hummingbird  │  │   Sparrow    │
         │  (CLI mode)  │  │(Workbench)   │
         └──────┬───────┘  └──────┬───────┘
                │                 │
         ┌──────▼─────────────────▼──────┐
         │         Plugin System          │
         │  system │ tools │ uicontrol    │
         │  backup │ email │ appgen │ ... │
         └──────────────┬────────────────┘
                        │
              ┌─────────▼─────────┐
              │   Host OS / GUI    │
              │  (files, windows,  │
              │   network, apps)   │
              └────────────────────┘

Two-Controller Model

The orchestrator has two AI controllers that each own their environment:

	Hummingbird	Sparrow
Environment	Host OS (terminal, filesystem)	StarkyGUI workbench (panels)
Style	Conversational, Jarvis-like	Panel-driven, workspace-based
Entry	`python orchestrator.py`	`/starkygui`
Prefix	Direct text	`/sparrow ...` commands

They can hand off tasks to each other via an explicit bridge:

# From CLI → Workbench
sparrow: open the browser and search for hummingbirds

# From Workbench → CLI
hummingbird: install numpy

Controllers

Hummingbird (CLI Controller)

Text-first, host-OS controller. Runs in the terminal and manages your system.

Capabilities:

Conversational natural language processing
Multi-step task decomposition via TaskBroker
Foreground/background task management with queue
Approval-gated execution for dangerous operations
Training system for learned command mappings
Cross-environment bridge to Sparrow

File: runtime/hummingbird.py

Sparrow (Workbench Controller)

GUI-driven controller that lives inside StarkyGUI. Manages panels, windows, and workbench apps.

Capabilities:

Panel management (open, move, resize, arrange)
Window arrangement (auto, side-by-side, stack, grid)
Built-in web browser panel
App launching and management
Workspace save/restore
Dev Trainer mode for teaching new commands interactively

File: runtime/sparrow.py

Task Broker

Breaks complex multi-step requests into subtasks with capability-aware routing.

File: runtime/task_broker.py

Plugins

All plugins auto-load from plugins/*/plugin.py. No plugin can crash the host.

Plugin	Description	Key Commands
system	Window management, WiFi, Bluetooth, audio, brightness, DND	`/winlist`, `/wifi on\|off\|list`, `/dnd on\|off`, `/bluetooth on\|off`, `/brightness N`
tools	Capability scanning and dependency installation	`/tools list`, `/tools install`, `/tools scan`
uicontrol	UI automation — screenshots, OCR, mouse/keyboard	`/uicontrol shot`, `/uicontrol click-text TEXT`, `/uicontrol type TEXT`
starkygui	Launch and manage the StarkyGUI workbench	`/starkygui web\|desktop\|vs\|install`
backup	Snapshot backups with retention policy and scheduling	`/backup`, `/install-nightly-backup [HH:MM]`, `/uninstall-nightly-backup`
dashboard	Terminal log dashboard	`/dashboard`
email_calendar	IMAP email and CalDAV calendar integration	`/email setup\|inbox`, `/calendar setup\|today\|week`
installai	AI-guided package installer with approval gating	Smart install prompts
orchy_app_gen	Full app generation using a heavy LLM model	`/appgen PROMPT`

Plugin Protocol

Every plugin implements this interface (all methods optional except name):

class Plugin(Protocol):
    name: str
    about: str

    def handle_command(line: str, cache: list) -> tuple[bool, list]
    def handle_natural(text: str) -> bool
    def commands_schema() -> list[str]
    def metadata() -> dict

See plugins/base.py for the full protocol. Plugins can also be installed via pip entry points.

StarkyGUI Workbench

A single-surface desktop where everything runs as panels:

App Generator — generate apps from prompts directly in the workbench
Browser — built-in web browser with history
System Control — WiFi, Bluetooth, DND, brightness
Tools — capability scanning and management
UI Control — grid-based AI-driven clicks and typing
Dashboard — system info and logs
Backup Manager — snapshot management
Email/Calendar — IMAP and CalDAV integration
Help — built-in help panel

Launch it:

/starkygui          # default (web)
/starkygui web      # web-based UI
/starkygui desktop  # PySide6 desktop app
/starkygui vs       # VS Code extension

Technology: Qt WebEngine host with JSON API bridge. Native app embedding on Windows.

Files:

starkygui_app/web_main.py — main Qt WebEngine host
starkygui_app/native_app_host.py — Windows native app embedding
starkygui_plugins/ — workbench panel plugins

Commands Reference

Core CLI Commands

/help                             Show help
/history                          List recent projects
/open N                           Open project N in file manager
/run N                            Run project N
/appgen PROMPT                    Generate an app (heavy model)
/web QUERY|URL                    Open web browser
/plugins list [--about]           List all plugins
/starkygui [web|desktop|vs]       Launch the workbench
/tools list|install|scan          Capability management
/dashboard                        Terminal log dashboard
/backup                           Create a backup snapshot
/dnd on|off                       Do Not Disturb toggle
/wifi on|off|list|connect SSID    WiFi control
/set model ID                     Change default LLM model
/set temperature X                Change default temperature
/quit                             Exit

Sparrow Workbench Commands

/sparrow panel PANEL                    Show a panel
/sparrow panel move PANEL X Y           Move panel to position
/sparrow panel resize PANEL W H         Resize panel
/sparrow arrange auto|side|stack|grid   Arrange all panels
/sparrow web QUERY|URL                  Open browser panel
/sparrow apps list                      List available apps
/sparrow apps open NAME                 Open an app
/sparrow plugins open|reload            Manage plugins
/sparrow workspace save NAME            Save workspace layout
/sparrow workspace restore NAME|ID      Restore workspace layout
/sparrow system sysinfo|wifi|bluetooth  System controls
/sparrow ui grid on|off                 Toggle UI control grid
/sparrow ui click-text TEXT             Click on-screen text
/sparrow ui type TEXT                   Type text

Training Commands

Teach the controllers new command mappings that persist across sessions:

learn "search for {query}" => /sparrow web {query}
learn "open {app}" => /sparrow apps open {app}
learn list                  Show all learned rules
learn forget <pattern>      Remove a rule
learn clear                 Reset all rules

Rules are stored in runtime/learned_rules.json and injected into the fast controller.

Configuration

User config is stored at ~/.config/ai_orchestrator/config.json.

LLM Settings

Key	Default	Description
`llm.model`	`gpt-5`	Default chat model
`llm.temperature`	`1.0`	Chat temperature
`llm.system_model`	`gpt-5-mini`	Fast controller model (intent routing)
`llm.system_temperature`	`0.1`	Fast controller temperature
`llm.app_gen_model`	`gpt-5`	App generation model
`llm.max_retries`	`3`	API retry count
`llm.cache_enabled`	`true`	Enable response caching

Behavior Settings

Key	Default	Description
`behavior.default_save_policy`	`A`	Save policy: Auto, Save, Delete, Keep (ask)
`behavior.auto_run`	`true`	Auto-run generated projects
`behavior.hummingbird_default`	`true`	Start in Hummingbird (CLI) mode
`behavior.hummingbird_auto_run`	`true`	Auto-execute Hummingbird tasks
`behavior.hummingbird_background`	`true`	Run tasks in background
`behavior.starky_internet_policy`	`prompt`	StarkyGUI internet access policy
`behavior.starky_appgen_target`	`panel`	Where generated apps appear

Security Settings

Key	Default	Description
`security.allow_ui_control`	`false`	Allow mouse/keyboard automation
`security.auto_arm_ui_control`	`false`	Auto-arm after first consent
`security.allow_system_actions`	`true`	Allow OS-level actions
`security.confirm_system_actions`	`true`	Require confirmation for installs
`security.remote_enabled`	`false`	Enable remote control (Discord/OpenClaw)

CLI Flags

python orchestrator.py --verbose "prompt"        # Verbose logging
python orchestrator.py --policy S "prompt"       # Force save policy
python orchestrator.py --no-run "prompt"         # Generate but don't run
python orchestrator.py --model gpt-5 "prompt"    # Override model
python orchestrator.py --hummingbird "prompt"    # Force CLI mode

Generated Projects

Projects are created in two directories:

Directory	Purpose
`SAVED/`	Persistent projects (kept across sessions)
`RUNNING/`	Temporary projects (deleted per save policy)

Each generated project includes:

run.sh / run.bat / run.command — platform launchers
HOW_TO_RUN.txt — human-readable instructions (for app projects)
Source files with stack-specific assets (venv, package.json, etc.)

Supported stacks: Python, Node.js (npm/yarn/pnpm), Go, Rust, Java, C++, web (HTML/CSS/JS)

Project Structure

ai-orchestrator/
├── orchestrator.py              # Main entry point
├── orchestrator_cli.py          # Backup CLI wrapper
├── config.py                    # Configuration management
├── setup.py                     # Environment setup & dependency installer
├── gui_shell.py                 # GUI shell for interactive sessions
│
├── cloud_agent/                 # OpenAI API integration
│   ├── cloud_client.py          # API client wrapper
│   └── system_controller.py     # Natural language → slash command translator
│
├── runtime/                     # Controllers & persistent state (gitignored)
│   ├── hummingbird.py           # CLI controller
│   ├── sparrow.py               # Workbench controller
│   ├── task_broker.py           # Multi-step task decomposition
│   ├── learned_rules.py         # Training system for command mappings
│   ├── memory_store.py          # Persistent memory
│   ├── plan_runner.py           # Plan execution engine
│   └── controller_contexts/     # Controller training context files
│
├── plugins/                     # Plugin system
│   ├── base.py                  # Plugin protocol definition
│   ├── registry.py              # Plugin loader & dispatcher
│   ├── system/                  # System control plugin
│   ├── tools/                   # Capability management plugin
│   ├── uicontrol/               # UI automation plugin
│   ├── starkygui/               # StarkyGUI launcher plugin
│   ├── backup/                  # Backup plugin
│   ├── dashboard/               # Dashboard plugin
│   ├── email_calendar/          # Email & calendar plugin
│   ├── installai/               # AI installer plugin
│   └── orchy_app_gen/           # App generation plugin
│
├── starkygui_app/               # StarkyGUI desktop application
│   ├── web_main.py              # Qt WebEngine host with JSON API bridge
│   ├── native_app_host.py       # Windows native app embedding
│   └── native_window_manager.py # Cross-platform window operations
│
├── starkygui/                   # VS Code extension for StarkyGUI
├── starkygui_plugins/           # Workbench panel plugins
├── starkygui_web/               # Web-based StarkyGUI
├── starkygui_electron/          # Electron wrapper (experimental)
│
├── utils/                       # Utility modules
│   ├── backup.py                # Snapshot backup with retention
│   ├── runner.py                # Command execution
│   ├── prompt.py                # Interactive prompts
│   ├── connected.py             # Email (IMAP) & calendar (CalDAV) setup
│   ├── ui_dashboard.py          # Terminal UI dashboard
│   └── helper.py                # General utilities
│
├── orchestrator_bridge/         # Cross-environment bridge server
├── integrations/                # Third-party integrations (Discord, OpenClaw)
├── scripts/                     # Utility scripts
├── local_agent/                 # Local system automation module
├── tests/                       # Test suite
│
├── docs/                        # Documentation
│   ├── commands.md              # Full command reference
│   ├── HELP.md                  # Built-in help text
│   ├── HUMMINGBIRD.md           # Hummingbird controller docs
│   ├── STARKYGUI.md             # StarkyGUI docs
│   ├── STARKYPLUGINS.md         # Plugin development guide
│   ├── uicontrol_grid.md        # UI control grid system
│   └── workbench_plan.md        # Workbench architecture
│
├── requirements.txt             # Python dependencies
├── requirements-dev.txt         # Dev dependencies
├── pyproject.toml               # Tool configuration (black, ruff, mypy)
├── pytest.ini                   # Test configuration
├── run_orchestrator.sh          # Linux/macOS launcher
├── run_orchestrator.bat         # Windows batch launcher
└── run_orchestrator.ps1         # Windows PowerShell launcher

Developing Plugins

CLI Plugins

Create a folder under plugins/ with a unique name
Add plugin.py implementing the Plugin protocol from plugins/base.py
The registry auto-discovers it on next launch

StarkyGUI Panel Plugins

Create a folder under starkygui_plugins/
Add plugin.json + optional plugin.js + plugin.css
Register in starkygui_plugins/index.json
Reload via the Plugins panel or /sparrow plugins reload

See docs/STARKYPLUGINS.md for the full guide and starkygui_plugins/sample_clock/ for an example.

Testing

python -m pytest -v                           # Run all tests
python -m pytest tests/test_orchestrator.py   # Specific test file

Security Notes

UI control (mouse/keyboard automation) is off by default and requires explicit consent
System actions (install, update, remove) require user confirmation
Remote control (Discord/OpenClaw) is disabled by default
All dangerous operations go through approval gating
API keys are never hardcoded — use environment variables or the system keyring

Requirements

Python 3.10+
OpenAI API key
Optional: PySide6/PyQt5 (for StarkyGUI desktop), system packages for UI automation (see setup.py)

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
cloud_agent		cloud_agent
docs		docs
integrations		integrations
local_agent		local_agent
orchestrator_bridge		orchestrator_bridge
plugins		plugins
scripts		scripts
starkygui		starkygui
starkygui_app		starkygui_app
starkygui_electron		starkygui_electron
starkygui_plugins		starkygui_plugins
starkygui_web		starkygui_web
tests		tests
utils		utils
.gitignore		.gitignore
JOURNEY.md		JOURNEY.md
README.md		README.md
config.py		config.py
gui_shell.py		gui_shell.py
orchestrator.py		orchestrator.py
orchestrator_cli.py		orchestrator_cli.py
pyproject.toml		pyproject.toml
pytest.ini		pytest.ini
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
run_orchestrator.bat		run_orchestrator.bat
run_orchestrator.ps1		run_orchestrator.ps1
run_orchestrator.sh		run_orchestrator.sh
setup.py		setup.py

Folders and files

Latest commit

History

Repository files navigation

AI Orchestrator

What It Does

Quick Start

1. Clone & install

2. Set your OpenAI API key

3. Run

Architecture

Two-Controller Model

Controllers

Hummingbird (CLI Controller)

Sparrow (Workbench Controller)

Task Broker

Plugins

Plugin Protocol

StarkyGUI Workbench

Commands Reference

Core CLI Commands

Sparrow Workbench Commands

Training Commands

Configuration

LLM Settings

Behavior Settings

Security Settings

CLI Flags

Generated Projects

Project Structure

Developing Plugins

CLI Plugins

StarkyGUI Panel Plugins

Testing

Security Notes

Requirements

License

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages