You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/model-and-source-configuration.md
+163-6Lines changed: 163 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,16 +7,119 @@ TalkPipe LLM segments need two values for every call:
7
7
8
8
You can set these on each segment, in `~/.talkpipe.toml`, via `TALKPIPE_*` environment variables, or through ChatterLang `$key` substitution. This guide explains how those layers interact. For logging, security, and general config mechanics, see [Configuration architecture](../architecture/configuration.md).
-[RAG and vector pipelines](#rag-and-vector-pipelines)
26
+
-[Ollama server URL](#ollama-server-url)
27
+
-[Examples](#examples)
28
+
-[Troubleshooting](#troubleshooting)
29
+
-[Related documentation](#related-documentation)
30
+
31
+
---
32
+
33
+
## Day-to-day usage
34
+
35
+
TalkPipe gives you several ways to configure `model` and `source` — segment parameters, ChatterLang `$key` substitution, `TALKPIPE_*` environment variables, and `~/.talkpipe.toml`. The setup below is not the only valid one; it is intended to be the simplest, lowest-maintenance path for day-to-day usage. The rest of this guide details each layer so you can deviate when you need to.
36
+
37
+
The pattern assumes you have a single "main" LLM source for most calls and only reach for alternatives occasionally:
38
+
39
+
1.**Set provider credentials in your shell environment** for whichever sources you actually use. These are SDK-level keys, not TalkPipe config keys:
40
+
- Ollama: nothing required if running on `localhost:11434`; otherwise set `TALKPIPE_OLLAMA_SERVER_URL`.
41
+
- OpenAI: `OPENAI_API_KEY`.
42
+
- Anthropic: `ANTHROPIC_API_KEY`.
43
+
2.**Set the `default_*` keys** for the model and source you reach for most often (chat and embeddings) — once, in `~/.talkpipe.toml` or as `TALKPIPE_*` environment variables.
44
+
3.**Write pipelines without specifying `model` / `source`** by default. Only override on the specific segments where you genuinely want a different model.
45
+
46
+
### Example: Ollama-first, occasional OpenAI
47
+
48
+
To make that concrete, here is what an Ollama-first setup looks like end-to-end. Start by writing the defaults to `~/.talkpipe.toml` once and forgetting about them:
Then add provider credentials to your shell for any non-Ollama backends you might use:
58
+
59
+
```bash
60
+
export OPENAI_API_KEY=sk-...
61
+
```
62
+
63
+
With those defaults in place, day-to-day ChatterLang lets a bare `llmPrompt` pick up the configured model and source — no per-call boilerplate:
64
+
65
+
```chatterlang
66
+
INPUT FROM echo[data="Quick question"]
67
+
| llmPrompt
68
+
| print
69
+
```
70
+
71
+
When a specific call needs a stronger (or just different) model, override only on that segment with `[model=..., source=...]`:
72
+
73
+
```chatterlang
74
+
INPUT FROM echo[data="Tougher question"]
75
+
| llmPrompt[model="gpt-4o", source="openai"]
76
+
| print
77
+
```
78
+
79
+
The same pattern works in the Python pipe API. Default usage relies on the config:
80
+
81
+
```python
82
+
# skip-extract
83
+
from talkpipe.pipe import io
84
+
from talkpipe.llm.chat import LLMPrompt
85
+
86
+
pipeline = io.echo(data="Summarize the latest meeting notes.") | LLMPrompt()
87
+
list(pipeline.as_function(single_out=False)())
88
+
```
89
+
90
+
And per-call overrides are the same as in ChatterLang — just constructor arguments:
91
+
92
+
```python
93
+
# skip-extract
94
+
from talkpipe.pipe import io
95
+
from talkpipe.llm.chat import LLMPrompt
96
+
97
+
careful = io.echo(data="Draft a careful legal summary.") | LLMPrompt(
98
+
model="gpt-4o",
99
+
source="openai",
100
+
)
101
+
list(careful.as_function(single_out=False)())
102
+
```
103
+
104
+
### Why this layout
105
+
106
+
- Provider credentials belong in the environment because the underlying SDKs read them directly and they are sensitive.
107
+
-`default_*` keys belong in `~/.talkpipe.toml` because they are stable preferences, not secrets, and you want them shared across every shell, notebook, and script.
108
+
- Per-segment overrides belong in code because the choice of model is usually tied to the specific task — and the segment parameter is the highest-precedence layer, so it always wins.
109
+
110
+
The remaining sections fill in the details behind that pattern: which `source` values are actually accepted, exactly how `model` and `source` are resolved when you omit them, every configuration key that participates, and the segment-by-segment parameter reference.
111
+
10
112
---
11
113
12
114
## Supported sources
13
115
14
-
Sources are registered in `talkpipe.llm.config`:
116
+
The first piece of the pattern above is the `source` value itself. Sources are registered in `talkpipe.llm.config`:
Additional sources can be registered at runtime with `registerPromptAdapter` or `registerEmbeddingAdapter` (see [Extending TalkPipe](../architecture/extending-talkpipe.md)).
The day-to-day pattern relies on TalkPipe quietly filling in `model` and `source` when you omit them. Here is the full rule it uses.
133
+
29
134
When `LLMPrompt` or `LLMEmbed` is constructed, TalkPipe fills in missing `model` / `source` from `get_config()` (merged `~/.talkpipe.toml` plus `TALKPIPE_*` environment variables). If either is still missing, construction raises an error.
The precedence table above refers to "config" generically. This section lists the specific keys TalkPipe looks for — the ones the Day-to-day section recommended setting in `~/.talkpipe.toml`, plus the separate set used by the RAG CLIs.
167
+
61
168
### Segment defaults (`default_*`)
62
169
63
-
Used by `llmPrompt` and `llmEmbed` when `model` / `source` are omitted:
170
+
Used by `llmPrompt`, `llmVisionPrompt`, and `llmEmbed` when `model` / `source` are omitted:
| Ollama server URL |`OLLAMA_SERVER_URL`|`TALKPIPE_OLLAMA_SERVER_URL`|
72
179
180
+
`llmVisionPrompt` shares the chat defaults — there is no separate `default_vision_model_*` key. If your `default_model_name` is a text-only model (for example `llama3.2`), passing it to `llmVisionPrompt` will fail at the provider rather than at construction. In practice, set `model` (and usually `source`) explicitly on `llmVisionPrompt`, or set `default_model_name` to a vision-capable model and override text-only `llmPrompt` calls when you need a smaller model.
181
+
73
182
Example `~/.talkpipe.toml`:
74
183
75
184
```toml
@@ -99,6 +208,8 @@ If a CLI flag is omitted and the matching `DEFAULT_*` key is unset, the value pa
99
208
100
209
## Segment parameters
101
210
211
+
Those configuration keys provide the fallback values; the per-segment parameters below override them and take final precedence at construction time.
212
+
102
213
### `llmPrompt` / `LLMPrompt`
103
214
104
215
Required (directly or via config): `model`, `source`.
Memory and compaction options (`memory_mode`, `context_token_trigger`, etc.) are described in [ChatterLang memory controls](../architecture/chatterlang.md#llmprompt-conversation-memory-controls).
119
230
231
+
### `llmVisionPrompt` / `LLMVisionPrompt`
232
+
233
+
Required (directly or via config): `model`, `source`. Required as a segment parameter: `image_field` (the item field holding the image path, URL, bytes, or `ImageResult`).
234
+
235
+
`llmVisionPrompt` resolves `model` / `source` from the same `default_model_name` / `default_model_source` keys as `llmPrompt`. There is no separate vision-specific default, so if your chat default is text-only you should override `model` (and usually `source`) on `llmVisionPrompt`:
236
+
237
+
```chatterlang
238
+
INPUT FROM loadImage[path="/path/to/diagram.png", set_as="image"]
chatterlang_script --script 'INPUT FROM prompt[data="Hi"] | llmPrompt | print'
174
309
```
175
310
311
+
Because `TALKPIPE_*` variables override the TOML file (see [Precedence](#precedence-highest-first)), this pattern is convenient for building a single generic TalkPipe container image that does not bake in any particular model or backend. The base image ships pipelines that omit `model` / `source`, and derived images — or runtime `docker run -e ...` / Kubernetes env — parameterize the deployment by setting `TALKPIPE_default_model_name`, `TALKPIPE_default_model_source`, and the embedding equivalents (plus `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `TALKPIPE_OLLAMA_SERVER_URL` as needed). For example, two derived images can target different backends from the same base:
`OPENAI_API_KEY` is intentionally left to the runtime (a Docker secret, Kubernetes secret, or `--env` flag) rather than baked into the image.
331
+
176
332
### 4. ChatterLang `$key` and CLI overrides
177
333
178
334
```bash
@@ -197,7 +353,8 @@ segment = LLMPrompt(system_prompt="You are helpful.")
197
353
| Symptom | What to check |
198
354
|---------|----------------|
199
355
|`Model name and source must be provided`| Set `model` and `source` on the segment, or add `default_model_name` and `default_model_source` (or embedding equivalents for `llmEmbed`). |
200
-
|`Unknown source`| Chat: use `ollama`, `openai`, or `anthropic`. Embeddings: only `ollama` is registered unless you added adapters. |
356
+
|`Unknown source`| Chat / vision: use `ollama`, `openai`, or `anthropic`. Embeddings: use `ollama` or `openai` (or register additional adapters). |
357
+
|`llmVisionPrompt` errors at the provider with model-not-found / unsupported-input |`llmVisionPrompt` reads `default_model_name` / `default_model_source` (the chat defaults). Set `model` and `source` explicitly on the segment, or change the chat defaults to a vision-capable model. |
201
358
| Ollama connection refused | Run `ollama serve` or set `OLLAMA_SERVER_URL` / `TALKPIPE_OLLAMA_SERVER_URL`. |
202
359
| OpenAI / Anthropic auth errors | Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`; these are not read from `TALKPIPE_*` model keys. |
203
360
| RAG CLI uses unexpected models | Check `DEFAULT_*` keys and CLI flags; then check segment `default_*` fallbacks. |
0 commit comments