Client Setup

OlliteRT implements the OpenAI Chat Completions / Responses / Audio APIs and the Anthropic Messages API. Any tool or library that lets you set a custom base URL will work out of the box — no plugins, no adapters, no cloud required.

Connection Settings
Home Assistant
- Conversation Agent
- Voice Transcription (STT)
Open WebUI
OpenClaw
Claude Code
Anthropic SDKs
Python (OpenAI SDK)
curl

Connection Settings

Every integration uses the same three values:

Setting	Value	Where to find it
Base URL	`http://PHONE_IP:8000/v1`	Status screen → tap the endpoint to copy
API Key	Your bearer token (if auth is enabled — enter any value if the client requires it but auth is disabled)	Settings → Server Configuration
Model	e.g. `Gemma-4-E2B-it`	Any model name shown on the Models screen

Tip

The port (8000) is configurable in Settings → Server Configuration. Replace PHONE_IP with your device's local IP address shown on the Status screen.

If you're having trouble connecting, see Troubleshooting → Connection Issues.

Home Assistant

OlliteRT integrates with Home Assistant in two ways — as a conversation agent for LLM-powered automations, and as a speech-to-text engine for voice pipelines. Each can be set up independently or combined in the same Assist pipeline.

See also: Home Assistant REST API — monitor server status, reload models, and adjust inference settings directly from HA automations using built-in REST sensors and commands.

Conversation Agent

Note

Home Assistant currently requires a custom integration for OlliteRT to work with conversation. The recommended options are Custom Conversation (recommended), Home LLM, Local OpenAI LLM, and Extended OpenAI Conversation.

Pick one of the following integrations, install and configure it, then continue with the common steps below.

Option A — Custom Conversation (recommended)

Step 1 — Install Custom Conversation via HACS.

Step 2 — Configure the integration (Settings → Devices & Services → Add Integration → search "Custom Conversation" → Choose OpenAI as LLM Provider):

Field	Value
Base URL	`http://PHONE_IP:8000/v1`
API Key	Your bearer token (if auth is enabled — enter any value if required but auth is disabled)
Model	`Gemma-4-E2B-it` (or any model name shown on the Models screen)

Step 3 — Open Custom Conversation Integration settings, click on settings button under Services → Custom Conversation and choose Assist as the API to expose to the LLM and select the desired LLM from the dropdown. Then click on save.

Step 4 — Continue with After Installation to set up the Assist pipeline and test.

Note

Custom Conversation was easiest to setup without any additional configuration and produced the most consistent results during testing. This however, does not mean that it is the best integration for every use case. Your mileage may vary.

Option B — Home LLM

Home LLM is built specifically for local/self-hosted models and uses Home Assistant's native Assist tool-calling API for device control.

Step 1 — Install Home LLM via HACS.

Step 2 — Add the integration (Settings → Devices & Services → Add Integration → search "Local LLM"):

Select "OpenAI Compatible 'Conversations' API" as the backend
Configure the connection:

Setting	Value
API Hostname	`PHONE_IP` (without http:// and port number)
API Port	`8000`
API Key	Your bearer token (if auth is enabled — enter any value if the client requires it but auth is disabled)
API Path	`v1`
Model	e.g. `Gemma-4-E2B-it`

Step 3 — Configure the conversation agent (in the integration options after adding):

Select model from the dropdown (auto-populated from OlliteRT's /v1/models) → Press Next → Select Assist as Selected LLM API(s) → Scroll down and Enable Legacy Tool Calling → Press Submit.

Step 4 — Continue with After Installation to set up the Assist pipeline and test.

Option C — Local OpenAI LLM

Step 1 — Install Local OpenAI LLM via HACS.

Step 2 — Configure the integration (Settings → Devices & Services → Add Integration):

Field	Value
Base / Server URL	`http://PHONE_IP:8000/v1`
API Key	Your bearer token (if auth is enabled — enter any value if required but auth is disabled)
Model	`Gemma-4-E2B-it` (or any model name shown on the Models screen)

Step 3 — Add a conversation agent (in the integration options after adding):

Select model from the dropdown (auto-populated from OlliteRT's /v1/models)→ Select Assist under Tool Providers → Press Submit.

Step 4 — Continue with After Installation to set up the Assist pipeline and test.

Option D — Extended OpenAI Conversation

Step 1 — Install Extended OpenAI Conversation via HACS.

Step 2 — Configure the integration (Settings → Devices & Services → Add Integration):

Field	Value
Base / Server URL	`http://PHONE_IP:8000/v1`
API Key	Your bearer token (if auth is enabled — enter any value if required but auth is disabled)
Model	`Gemma-4-E2B-it` (or any model name shown on the Models screen)

Important

Configuring a model for Extended OpenAI Conversation needs to be done in the Integration settings > Cog Wheel icon > chat_model field. By default integration uses gpt-4o-mini which is not available on OlliteRT. Change it to Gemma-4-E2B-it or any other model from the Models screen, otherwise Assist will not work.

Step 3 — Continue with After Installation to set up the Assist pipeline and test.

After Installation

These steps apply to all integrations above.

Create an Assist pipeline — Settings → Voice assistants → Add Assistant. Select your chosen integration as the Conversation Agent.

Expose entities — Settings → Voice Assistants → Expose tab. Only exposed entities are available to the model. Keep the number reasonable to avoid exceeding the model's context window.

Test it — Open the Assist UI and try "Turn on [Name of exposed entity]" to verify tool calling works. See Troubleshooting → Tool Calling if it doesn't respond correctly.

Tip

For better tool calling results:

Keep entity names and aliases short and simple — the model matches concise names (e.g. "Living Room Light") more reliably than long or complex ones
Some integrations limit temperature to 0–1, but Gemma models support 0–2. Enable Ignore Client Sampler Parameters in Settings → Model Behaviour to discard client-sent values and use your own per-model inference settings instead

Voice Transcription (STT)

Note

Home Assistant currently requires a custom integration for Speech-to-Text to work with OlliteRT. The recommended option is OpenAI STT for Home Assistant.

Step 1 — Install OpenAI STT for Home Assistant via HACS and Restart Home Assistant

Step 2 — Configure the integration (Settings → Devices & Services → Add Integration → search "OpenAI STT"):

Field	Value
Name	Any name to identify this STT provider (e.g. `OlliteRT STT`)
API URL	`http://PHONE_IP:8000/v1`
API Key	Your bearer token (if auth is enabled — enter any value if required but auth is disabled)
Model	Auto-discovered from OlliteRT — select the model from the dropdown

Step 3 — Enable Force Transcription in OlliteRT Settings → Home Assistant. This instructs the model to transcribe audio rather than respond to it conversationally.

Step 4 — Set the STT provider in your Assist pipeline (Settings → Voice assistants → edit your assistant):

Step 5 — Test it. Use the voice input in the Assist UI and verify the transcribed text appears correctly. See Troubleshooting → Voice Transcription if it doesn't respond correctly.

Open WebUI

Open WebUI is a self-hosted ChatGPT-style interface. OlliteRT appears as an OpenAI-compatible backend.

Setup:

Go to Settings → Connections
Under OpenAI API, click + to add a new connection
Set the URL to http://PHONE_IP:8000/v1
Enter your bearer token as the API key (if auth is enabled)
Click the refresh icon — your loaded model will appear in the model selector

Tip

Open WebUI supports streaming, markdown rendering, and conversation history — making it a great desktop companion for OlliteRT. Voice input and audio transcription work out of the box — no additional configuration needed. Just use the voice mode button in the chat UI.

Tool Calling:

Important

To use tools (function calling) with OlliteRT in Open WebUI, you need to set the Function Calling mode to Native. By default, Open WebUI does not send tool calls natively to the backend.

The recommended approach is to set it globally for all models:

Go to Admin Panel → Settings → Models
Click the Settings button (top right of the models list)
Under Model Parameters, set Function Calling to Native
Save

This applies to all existing and future models. You can also override per-model (edit a specific model → Model Parameters → Function Calling → Native) or per-chat (Chat Controls → Advanced Params → Function Calling → Native).

See the Open WebUI docs on Native mode for more details.

OpenClaw

OpenClaw is a self-hosted personal AI assistant gateway. Instead of a web chat UI, it routes AI responses through messaging platforms you already use — WhatsApp, Telegram, Discord, and 30+ others.

Note

While OpenClaw works with OlliteRT, results may vary due to the small context windows of on-device models. OpenClaw's multi-platform routing and conversation management can consume significant context, leaving less room for actual responses.

Tool calling will highly likely not work — the tool definitions combined with OpenClaw's own context will exceed most models' context windows.

Using a larger model with a bigger context window can improve the experience. See the Model Guide for context window sizes.

Connect to OlliteRT by adding a custom provider in ~/.openclaw/openclaw.json:

{
  models: {
    providers: {
      ollitert: {
        baseUrl: "http://PHONE_IP:8000/v1",
        apiKey: "your-token",
        api: "openai-completions"
      }
    }
  }
}

OpenClaw has built-in provider plugins for Ollama, vLLM, LM Studio, and others. Any server exposing /v1/chat/completions works via the custom provider config above.

Claude Code

Claude Code targets the Anthropic Messages API. OlliteRT exposes that API on /v1/messages, so Claude Code can drive your phone with no proxy.

Warning

Experimental — not recommended for daily coding work. Claude Code ships with a multi-thousand-token system prompt and a large set of tool definitions (Bash, Edit, Read, Write, Grep, Glob, Task, etc.) that it sends on every request. On-device models in the Gemma-4-E2B / 3n class do not have the context budget or instruction-following headroom to drive that workload reliably — expect long prefill, frequent tool-call mistakes, and the LiteRT-LM #2418 parse failures noted below. Coding harnesses with a small system prompt and a narrower tool surface (for example Pi Agent) running against the OpenAI-compatible /v1/chat/completions endpoint are a much better fit for this hardware. Treat Claude Code support here as a smoke-test for the Anthropic API, not a production workflow.

Setup — set two environment variables before launching Claude Code:

ANTHROPIC_BASE_URL=http://PHONE_IP:8000 \
ANTHROPIC_AUTH_TOKEN=your-token \
claude

ANTHROPIC_AUTH_TOKEN is mapped to the x-api-key header. The /v1 segment is appended automatically — set the base URL to the host root, not to …/v1. By default OlliteRT does not require auth, so the value is ignored — pass any non-empty string (Claude Code requires the variable to be set). If you've turned on Require Bearer Token under Settings → Server Configuration, the value must exactly match the token configured there. The phone never relays the token to the real Anthropic API.

Pick the right model in OlliteRT first — Claude Code sends a long system prompt and many tools (Bash, Edit, Read, etc.), so a Gemma-4-E2B-it or larger is the practical floor.

Warning

Tool calls can fail with HTTP 500 on Gemma 4. When the model emits a tool call whose argument is a string with quoted content (most Bash / Edit calls), LiteRT-LM 0.11.0 / 0.12.0's native function-call parser raises INVALID_ARGUMENT and the request errors out. Tracking upstream: google-ai-edge/LiteRT-LM#2418. Workaround: turn off Settings → Schema Injection in OlliteRT so tool calls flow through the text-mode parser instead.

Tip

If a request appears to hang for 30–60 s before producing output, that is on-device prefill — not a network issue. OlliteRT emits Anthropic ping events every 10 s during prefill so Claude Code's SSE timeout doesn't trigger; you can confirm the stream is alive in Settings → Logs.

Anthropic SDKs

The official Anthropic Python SDK and TypeScript SDK both accept a base_url override and work without modification.

Python:

from anthropic import Anthropic

client = Anthropic(
    base_url="http://PHONE_IP:8000",
    api_key="your-token",  # ignored when auth is disabled; must match the configured token when auth is enabled in Settings
)

resp = client.messages.create(
    model="Gemma-4-E2B-it",
    max_tokens=256,
    messages=[{"role": "user", "content": "Say hello"}],
)
print(resp.content[0].text)

TypeScript:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  baseURL: "http://PHONE_IP:8000",
  apiKey: "your-token", // ignored when auth is disabled; must match the configured token when auth is enabled in Settings
});

const msg = await client.messages.create({
  model: "Gemma-4-E2B-it",
  max_tokens: 256,
  messages: [{ role: "user", content: "Say hello" }],
});
console.log(msg.content[0].type === "text" ? msg.content[0].text : "");

Streaming works via client.messages.stream(...) (Python) and client.messages.stream({...}) (TypeScript) without further configuration.

Python (OpenAI SDK)

The official OpenAI Python SDK works with OlliteRT by pointing it at your device's URL.

Install:

pip install openai

Streaming:

from openai import OpenAI

client = OpenAI(
    base_url="http://PHONE_IP:8000/v1",
    api_key="your-token"  # if bearer auth is enabled
)

for chunk in client.chat.completions.create(
    model="Gemma-4-E2B-it",
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}],
    stream=True
):
    print(chunk.choices[0].delta.content or "", end="")

Non-streaming:

response = client.chat.completions.create(
    model="Gemma-4-E2B-it",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Tool calling:

response = client.chat.completions.create(
    model="Gemma-4-E2B-it",
    messages=[{"role": "user", "content": "What time is it?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_current_time",
            "description": "Get the current date and time",
            "parameters": {"type": "object", "properties": {}}
        }
    }]
)

message = response.choices[0].message
if message.tool_calls:
    print(f"Tool call: {message.tool_calls[0].function.name}")

Note

Tool calling is experimental. By default it uses SDK schema injection for structured output; disable Tool Schema Injection in Settings → Model Behaviour to fall back to prompt-based parsing. See Troubleshooting → Tool Calling if results are unexpected.

curl

Non-streaming:

curl http://PHONE_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Gemma-4-E2B-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Streaming:

curl http://PHONE_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Gemma-4-E2B-it",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

With bearer auth:

curl http://PHONE_IP:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-token" \
  -d '{
    "model": "Gemma-4-E2B-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

List models:

curl http://PHONE_IP:8000/v1/models

Health check:

curl http://PHONE_IP:8000/health?metrics=true

See the API Reference for the full list of endpoints and parameters.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Client Setup

Table of Contents

Connection Settings

Home Assistant

Conversation Agent

Option A — Custom Conversation (recommended)

Option B — Home LLM

Option C — Local OpenAI LLM

Option D — Extended OpenAI Conversation

After Installation

Voice Transcription (STT)

Open WebUI

OpenClaw

Claude Code

Anthropic SDKs

Python (OpenAI SDK)

curl

Uh oh!

FilesExpand file tree

CLIENT_SETUP.md

Latest commit

History

CLIENT_SETUP.md

File metadata and controls

Client Setup

Table of Contents

Connection Settings

Home Assistant

Conversation Agent

Option A — Custom Conversation (recommended)

Option B — Home LLM

Option C — Local OpenAI LLM

Option D — Extended OpenAI Conversation

After Installation

Voice Transcription (STT)

Open WebUI

OpenClaw

Claude Code

Anthropic SDKs

Python (OpenAI SDK)

curl