Skip to content

TOOLS.md

Latest commit

 

History

History
180 lines (137 loc) · 8.4 KB

TOOLS.md

File metadata and controls

180 lines (137 loc) · 8.4 KB
Raw

Tool calling & MCP

The 27B models are trained for agentic tool use. This page covers the three ways to use that: the raw API, MCP servers in the llama-server chat UI, and the Open WebUI demo. (Running the servers is covered in the README; model-tuning knobs in AGENTS.md.)

Native tool calling (OpenAI-compatible API)

start_llama_server.sh runs the 27B with --jinja, so /v1/chat/completions accepts the standard OpenAI tools array and returns structured tool_calls (no prompt hacks needed). The MLX server (start_mlx_server.sh) emits native tool_calls too.

curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather for a city",
      "parameters": {
        "type": "object",
        "properties": {"city": {"type": "string"}},
        "required": ["city"]
      }
    }
  }]
}'

The response's choices[0].message.tool_calls carries the call; append your tool result as a "role": "tool" message and continue the conversation as usual.

MCP prompt cost (read this before enabling servers)

Every MCP server that is active in a chat has its tool schemas rendered into that chat's prompt, adding prefill time before the first token — noticeable on Macs and CPU boxes. That is why the demos ship everything per-chat opt-in: the servers are preconfigured and one toggle away, but you only pay for what you turn on.

Measured from the live servers (2026-07; will drift as they update their tools):

MCP server Tools Approx. prompt tokens
Hugging Face 8 ~2,600 (biggest: hf_fs ~670, hub_repo_details ~490, hub_repo_search ~440)
DeepWiki 3 ~400
Brave Search 3 (web/news/summarizer) ~2,900 (its full 8-tool set is ~29k - brave_place_search alone is ~20k - so the demo enables only these three via --enabled-tools)

The cost is one-time per server run, not per chat: the schemas form a stable prompt prefix and llama-server reuses the cached prefix across new chats, as long as the enabled set (and order) doesn't change. On slow hardware prefer a small fixed subset over toggling servers between chats; on a fast GPU turning everything on is cheap.

MCP in the llama-server chat UI

The built-in UI (http://localhost:8080) has an MCP client with an agentic loop. The start scripts pass --webui-config-file scripts/webui-config.json, which preconfigures two servers:

  • Hugging Face (https://huggingface.co/mcp) — search models, datasets and Spaces on the Hub.
  • DeepWiki (https://mcp.deepwiki.com/mcp) — ask questions about any public GitHub repository (Cognition's auto-generated repo wikis).

They appear in the MCP selector in the message box (and in Settings → MCP Client), but no tool schemas are sent until you turn a server on for a chat:

  • Toggling a server inside an open conversation enables it for that conversation only.
  • Toggling on the new-chat screen makes it your default for all future chats (stored in the browser; toggle it off the same way to stop).

agenticMaxTurns (default 10) bounds the tool loop. All of this state lives in the browser (localStorage), per-user and per-machine.

Troubleshooting: every new chat prefills ~2–3k tokens. A server was toggled on from the new-chat screen at some point, making it this browser's default for new chats. Toggle it off in the new-chat screen's MCP selector, or clear site data for localhost:8080 (browser DevTools → Application/Storage → Clear site data). A private/incognito window shows what a fresh browser would see; there is no way to reset browser state from the server side.

Optional: web search (Brave) in the llama-server UI

The llama-server UI ships out-of-the-box features only, so it does not auto-start a bridge — but you can add Brave web search yourself with a Brave Search API key. Run the bridge (key via env, never committed):

npm i -g @brave/brave-search-mcp-server
BRAVE_API_KEY=your_key brave-search-mcp-server --transport http --host 127.0.0.1 --port 8001

Then add it in the UI: Settings → MCP Client → add http://127.0.0.1:8001/mcp (or add an entry to scripts/webui-config.json, keeping "enabled":true so it's a per-chat opt-in). In the Open WebUI demo this is wired for you (next section).

Tools & MCP in the Open WebUI demo

./scripts/start_openwebui.sh seeds a set of on-by-default capabilities:

  • weather, web fetch, demo SQL database — local Python tools (small schemas).
  • code interpreter (server-side Python via Jupyter — plots, pandas, yfinance).
  • time — get_current_timestamp (a builtin tool); the model is also told the current date/time in its system prompt ({{CURRENT_DATE}}), so "this year" / "recent" reasoning works without a tool call.
  • Brave web search — attached by default when configured (key + bridge, below).

Hugging Face and DeepWiki are connected but per-chat opt-in (their schemas are heavy); pick them from the tool menu in the message box when a chat needs them.

Note: with Brave on by default the base prompt is larger (~2.9k tokens for Brave's schemas). If that matters for your hardware, drop server:mcp:brave from the model toolIds in seed_openwebui.py to make Brave per-chat opt-in like the others.

web_fetch reads arbitrary public pages, but some sites (e.g. Reuters) block automated fetches and return 401/403 - that's the site, not the tool.

Optional: Brave Search (web search). Needs a Brave Search API key and a local bridge. Install the bridge once:

npm i -g @brave/brave-search-mcp-server

Then provide the key one of two ways — either is local-only and never committed:

BRAVE_API_KEY=your_key ./scripts/start_openwebui.sh

or drop it in a gitignored file so you don't have to pass it each time:

echo "your_key" > .brave_key
./scripts/start_openwebui.sh

The script starts the bridge on 127.0.0.1:8001, passes your key to it via the environment (the key never touches the repo or the config), connects it as the brave MCP server, and shuts it down with everything else on Ctrl+C.

Adding your own MCP servers

Only servers speaking streamable HTTP work directly — a stdio-only server needs a local HTTP bridge first (the Brave setup above is the pattern: run the bridge, point the config at http://127.0.0.1:<port>/mcp).

llama-server chat UI — two places:

  • Per-browser (no restart): Settings → MCP Client → add name + URL.

  • Shipped default for everyone: add an entry to the mcpServers JSON-string in scripts/webui-config.json and restart the server:

    {"id":"context7","enabled":true,"name":"Context7","url":"https://mcp.context7.com/mcp","requestTimeoutSeconds":300}

    "enabled":true only lists the server in the per-chat selector — chats still opt in individually, so this costs nothing by default. Add "useProxy":true if the server rejects browser CORS requests (llama-server ships a /cors-proxy).

Open WebUI — edit the TOOL_SERVER_CONNECTIONS JSON in scripts/start_openwebui.sh and restart. Do not add servers through the admin panel: the demo runs with ENABLE_PERSISTENT_CONFIG=false, so panel edits are lost on restart — the script is the source of truth. Entry shape:

{"id":"context7","type":"mcp","url":"https://mcp.context7.com/mcp","path":"","auth_type":"none","key":"","config":{"enable":true,"access_grants":[{"principal_type":"user","principal_id":"*","permission":"read"}]},"info":{"id":"context7","name":"Context7","description":"Library docs"}}

For servers that need a token, set "auth_type":"bearer" and inject the secret at runtime instead of hardcoding it in the tracked script: keep it in an environment variable or a gitignored file (the Brave key below uses .brave_key), and reference it from the JSON the script builds, e.g. "key":"'"'"'$MY_MCP_KEY'"'"'" inside the TOOL_SERVER_CONNECTIONS block. Never commit a literal token. config.enable: true means connected and visible in the chat tool menu — still per-chat opt-in. To attach a server to every chat, add server:mcp:<id> to the model toolIds in scripts/openwebui/seed_openwebui.py (mind the prompt-token table above).