A lightweight, protocol-driven multi-agent framework for Apple platforms — on-device agent orchestration, MCP tools, realtime voice, and first-class tracing.
Status: work in progress. Built incrementally, one reviewed component at a time. The public API is not yet stable.
Versioning: until the first stable release, install with
branch: "main"(as in the quick start below). Swift releases will use bare semver tags (0.1.0,0.2.0, …) — thetypescript_*/python_*tags in this monorepo are not SwiftPM-resolvable.
- iOS 16+ / macOS 14+
- Swift 6.2 toolchain (Xcode 26+), Swift 6 language mode
The whole framework runs on iOS 16+, including the bundled FileChatStorage (JSON-file
persistence). The one piece gated higher is DeviceChatStorage (SwiftData), which needs iOS 17+ /
macOS 14+. iOS 15 is not supported: the MCP Swift SDK and core Duration/ContinuousClock/
URL.appending(path:) all require iOS 16.
A multi-agent framework built to run on device: small protocols you extend, swappable agents, classifier routing, and pluggable chat storage — plus what an on-device assistant needs:
- 🧠 Swappable agents — pick a simple single-LLM agent, or a grounded Brain → Presenter agent that answers only from what the tools actually returned.
- 🔌 Remote or on-device models — use any remote, OpenAI-compatible LLM today; run models fully on device (soon).
- 🧰 Tools from any source — Model Context Protocol (MCP) servers, native Swift functions, or a mix; the agent doesn't care where a tool comes from.
- 🪟 Text or rich UI — answers come back as plain text or as text plus interactive widgets, your choice per agent.
- 🎙️ Realtime voice — natural spoken conversations with interrupt-to-speak, as a first-class part of the framework.
- 📈 Built-in tracing — see every step of a run, locally during development or shipped to your observability tool of choice.
- 💾 Local-first chat history — conversations persist on device, with storage you can swap out.
- Classify — user input arrives and the Classifier reads it (with the chat history) to route the turn.
- Select — it picks the best agent for the job: a plain
Agent, aGroundedAgent, or one of your own. - Process — the chosen agent runs the turn (calling tools as needed) and produces the reply.
- Store & respond — the turn is saved to the chat history and the answer goes back to the user.
Single-agent apps skip the classifier entirely — the orchestrator just runs your one agent.
GroundedAgent is the framework's anti-hallucination pattern: two LLMs, not one.
- Brain — one system prompt (shared across all tools) with full instructions, the chat history, and the tools. It decides what to fetch, calls the tools, and produces the tool output — but never speaks to the user.
- Presenter — no tools, no chat history, no tool responses, no generic instructions. Just the tool output plus a small prompt chosen for the specific data being shown. With nothing to call and only the output in front of it, it can't invent values beyond what was actually fetched.
A no-tool turn answers in one pass — the Brain replies directly, no Presenter. The same Brain → tool output → Presenter core also powers the realtime voice runtime.
By default the Presenter also sees this turn's question (tagged separately from the data) so it
can answer directly; pass presenterInput: .dataOnly for strict data-only presenting. It never
sees the chat history or the Brain's transcript in either mode.
The Brain runs one system prompt for all tools — role, the full tool catalog, and the rules for gathering and resolving follow-ups:
You are the data brain of a shopping assistant.
GATHER the facts needed to answer the user — never write the final reply.
Tools:
search_products(query, max_price) → matching products
get_product(id) → details · price · rating · stock
get_order(id) → order + delivery status
Rules:
- Call whatever tools you need; you may chain several.
- Use the chat history to resolve follow-ups ("cheaper ones?", "is it in stock?").
- Never invent values. If a tool returns nothing, note that.
- Do NOT address the user or format anything — the presenter does that.
In a real assistant this is the big one — easily several hundred to a couple thousand tokens once the full tool catalog, domain rules, and output policy are in.
The Presenter instead picks one small prompt for the data being shown (a few dozen tokens each) — here, the one for product results:
You are presenting product search results.
Use ONLY the data block provided — never invent a price, rating, name, or stock status.
Lead with the best match: its name and price, then one standout detail.
Two short sentences, natural tone. Do not call tools.
The same answer can come back as plain text, or as text plus UI elements (cards, lists, and the like) — you choose per agent. When a tool has a UI to show, the framework can surface it as a widget next to the reply, or fold everything into a text-only answer.
→ How a tool UI flows through the agent: the Tool UIs docs.
A complete first run — a tiny command-line executable you can swift run.
1. Create the package — Package.swift:
// swift-tools-version: 6.0
import PackageDescription
let package = Package(
name: "quickstart",
platforms: [.macOS(.v14)],
dependencies: [
.package(url: "https://github.com/2FastLabs/agent-squad", branch: "main")
],
targets: [
.executableTarget(name: "quickstart", dependencies: [
.product(name: "AgentSquad", package: "agent-squad")
])
]
)2. Write it — Sources/quickstart/main.swift. One agent, no classifier (the common case):
import AgentSquad
import Foundation
guard let apiKey = ProcessInfo.processInfo.environment["OPENAI_API_KEY"] else { fatalError("set OPENAI_API_KEY") }
let model = ChatCompletionsClient(model: "gpt-4o-mini", apiKey: apiKey)
let agent = Agent(name: "Shop", description: "Shopping assistant", model: model)
let orchestrator = Orchestrator(
agents: [agent],
store: try DeviceChatStorage(userId: "u1", inMemory: true)
// classifier defaults to nil → no routing hop; the one agent always answers
)
for try await event in orchestrator.route(.text("wireless headphones under €100?"),
userId: "u1", sessionId: "s1") {
if case .textDelta(let token) = event { print(token, terminator: "") }
}
print()3. Run it:
OPENAI_API_KEY=sk-… swift runGive the orchestrator more than one agent and an LLMClassifier, and it routes each turn to the
best fit — the call site doesn't change:
let shop = Agent(name: "Shop", description: "Product search, prices, recommendations.", model: model)
let support = Agent(name: "Support", description: "Orders, returns, and account help.", model: model)
let orchestrator = Orchestrator(
agents: [shop, support], // the first agent (shop) is the default/fallback
classifier: LLMClassifier(model: model), // picks the agent for each turn
store: try DeviceChatStorage(userId: "u1", inMemory: true)
)
for try await event in orchestrator.route(.text("where is my order #1234?"),
userId: "u1", sessionId: "s1") {
if case .textDelta(let token) = event { print(token, terminator: "") }
}From here: give an agent a ToolProvider for tools — build native ones with ToolKit (Tool.local
for Swift code, Tool.http/HTTPToolGroup for APIs), connect an MCP server with MCPServer(url:),
or mix them with AggregateToolProvider — or swap Agent for GroundedAgent. None of it changes
the call site.
The package ships as separate libraries so the core stays dependency-free; you import only the integrations you need (each isolates its own dependencies):
| Library | import |
Pulls in | Contents |
|---|---|---|---|
AgentSquad (core) |
import AgentSquad |
nothing external | protocols, agents, orchestrator, native tools (ToolKit, Tool.local/.http, HTTPToolGroup, AggregateToolProvider), DakeraRetriever (Dakera memory retriever), FileChatStorage, DeviceChatStorage (iOS 17+), InMemoryChatStorage, TransformingChatStorage, SummarizingChatStorage, OSLogTracer |
AgentSquadMCP |
import AgentSquadMCP |
the official MCP Swift SDK | MCPServer (alias of MCPToolProvider) — connect any MCP server with MCPServer(url:), with MCP Apps UI support |
AgentSquadAudio |
import AgentSquadAudio |
AVFoundation (Apple platforms only) | VoiceProcessedAudioIO (echo-cancelled capture + playback on one engine — the recommended wiring), MicCapture, AudioPlayback for the realtime voice runtime — requires NSMicrophoneUsageDescription in your Info.plist |
So an app that doesn't use MCP never downloads the MCP SDK. Future optional integrations
(e.g. AgentSquadLangfuse for trace export) follow the same pattern — the core never grows a
dependency. Add a product to your target's dependencies to use it:
.target(name: "MyApp", dependencies: [
.product(name: "AgentSquad", package: "agent-squad"),
.product(name: "AgentSquadMCP", package: "agent-squad"), // only if you want MCP tools
])The audio layer captures through Apple's Voice-Processing I/O unit by default — the native equivalent of what ChatGPT's voice mode gets via WebRTC. The signal sent to the speaker is used as a hardware reference to subtract the assistant's own voice from the mic, plus noise suppression and automatic gain control. Without it, the assistant hears itself through the speaker and interrupts its own answers.
The audio layer is configurable in four independent levels — each level keeps everything the previous levels give you:
Level 0 — defaults. VoiceProcessedAudioIO runs capture and playback on one
AVAudioEngine, so the assistant's audio renders through the voice-processed output and is by
construction in the echo canceller's reference path. Pass one instance as both input: and
output::
import AgentSquadAudio
let io = VoiceProcessedAudioIO()
let runtime = RealtimeRuntime(session: assistant, input: io, output: io)
try await runtime.start()The separate MicCapture + AudioPlayback pair still works (both echo-cancelled on the capture
side by default) — but with two engines the echo reference is taken at the device level, which is
route-dependent. Prefer VoiceProcessedAudioIO for voice sessions.
Level 1 — tune voice processing (or turn it off):
// Keep AEC but disable gain control and minimize how much the system ducks playback volume:
let io = VoiceProcessedAudioIO(voiceProcessing: .init(automaticGainControl: false, duckingLevel: .min))
// Raw capture — no AEC at all (the previous default; split pair only):
let rawMic = MicCapture(voiceProcessing: nil)If the Voice-Processing unit can't be enabled, start() throws
MicCaptureError.voiceProcessingUnavailable(_:) rather than silently degrading — catch it and
retry with voiceProcessing: nil if raw capture is an acceptable fallback for your app.
Level 2 — own the AVAudioSession, or reach the raw engine. All three audio classes take
an AudioSessionPolicy (if you use the split pair, give both the same one so they can't fight):
// Your app already manages its AVAudioSession (music, video, CallKit…) — AgentSquad won't touch it.
// You must configure AND activate the session yourself before runtime.start().
let io = VoiceProcessedAudioIO(sessionPolicy: .external)
// Or let AgentSquad drive the timing but with YOUR configuration (iOS):
let io2 = VoiceProcessedAudioIO(sessionPolicy: .custom { session in
try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.allowBluetoothHFP]) // no speaker override
try session.setActive(true)
})The configureEngine hook hands you the underlying AVAudioEngine at the right lifecycle
moment (after voice processing is enabled, before the tap is installed), so any AVFoundation
API stays reachable without forking the class:
let io = VoiceProcessedAudioIO(configureEngine: { engine in
// e.g. inspect engine.inputNode, insert effect nodes, adopt future AVFoundation APIs…
})Level 3 — replace the audio path entirely. AudioInput/AudioOutput are protocols and
RealtimeRuntime accepts any conformer — bring WebRTC capture, CallKit audio, or a fully custom
pipeline and keep the rest of the stack (transport, VAD, tools, tracing):
final class MyCapture: AudioInput { /* frames, start(), stop() */ }
let runtime = RealtimeRuntime(session: assistant, input: MyCapture(), output: AudioPlayback())A custom AudioOutput can also implement playedMilliseconds() (defaults to nil) — the
session uses it to send conversation.item.truncate on barge-in, so the model's context only
keeps what the user actually heard. The built-in outputs measure this automatically.
Validate on a real device. The simulator performs no echo cancellation at all, so AEC can't be tested there. Expect voice-processed audio to sound "call-like" and slightly quieter — use
duckingLevel: .minto compensate. Never enable voice processing on the playback engine; it belongs on capture only.
Reasoning-capable Realtime models (e.g. gpt-realtime-2) can "think" before each response; how
much is a tunable — RealtimeReasoningEffort: .minimal/.low/.medium/.high/.xhigh
(server default .low). OpenAIVoiceAssistant exposes it at two levels:
let assistant = OpenAIVoiceAssistant(
name: "advisor", transport: transport, tools: tools,
userId: "u1", sessionId: "s1",
reasoning: .low, // session default for every response
toolReasoningEffort: ["get_insights": .medium] // per-tool escalation (see below)
)toolReasoningEffort solves a latency/depth dilemma: you rarely know a turn needs deep reasoning
before it starts — but the turn reveals itself by the tools it calls. When a turn calls a listed
tool, every subsequent response of that turn is created with the mapped effort (highest wins
if several match), so the response that synthesizes a heavyweight tool's payload thinks harder
while quick lookups keep the session default. The escalation is per turn and resets when the turn
completes.
Leave both unset for non-reasoning models (e.g. gpt-realtime-1.5) — nothing is sent on the wire.
Two peer runtimes share one set of contracts but not a control loop: a turn-based Orchestrator
(classify? → dispatch → stream → persist) and a long-lived RealtimeSession for voice. Agents,
tools (ToolProvider), storage (ChatStorage), and tracing (Tracer) are all swappable protocols;
optional integrations (MCP, tracing exporters) ship as separate products so the core stays
dependency-free.
This repo ships a skill — a single, assistant-agnostic guide at SKILL.md with the
mental model, the real API signatures, task recipes, and the gotchas, for both the built-in types and
writing your own.
It isn't auto-installed — point your assistant at it. For example, tell it:
Read
SKILL.mdbefore writing any AgentSquad code.
Works with any assistant (Claude, Cursor, Copilot, …). Claude Code users can also drop it in as a
skill: copy it to .claude/skills/agent-squad-swift/SKILL.md.
Apache 2.0 — see the repository LICENSE.