You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: 'Overview of configurable workers in the Spice runtime.'
5
+
---
6
+
7
+
Workers define a pattern of usage for one or more other spicepod components. The type of workers include:
8
+
-[Routers](/docs/components/workers/routers.md): Define routing rules how one or more [llms](/docs/components/models/index.md) can be combined into a logically single model.
description: 'Instructions for defining routing rules for LLMs within Spice.'
4
+
sidebar_label: 'LLM Routing'
5
+
sidebar_position: 1
6
+
---
7
+
8
+
Model routing workers define how one or more [llms](/docs/components/models/index.md) can be combined into a logically single model. Workers are defined as components in the `spicepod.yaml`.
9
+
10
+
**Example:**
11
+
12
+
```yaml
13
+
workers:
14
+
- name: round-robin
15
+
description: |
16
+
Call models 'foo' & 'bar' in round robin.
17
+
models:
18
+
- from: foo
19
+
- from: bar
20
+
- name: fallback
21
+
description: |
22
+
Call 'bar'. On error, call 'foo'. Failing that 'baz'.
23
+
models:
24
+
- from: foo
25
+
order: 2
26
+
- from: bar
27
+
order: 1
28
+
- from: baz
29
+
order: 3
30
+
- name: weighted
31
+
description: |
32
+
Routes 80% of traffic to 'foo'.
33
+
models:
34
+
- from: foo
35
+
order: 4
36
+
- from: bar
37
+
order: 1
38
+
```
39
+
40
+
For a complete specification of the routing rule options and syntax see the [specifications](/docs/reference/spicepod/workers.md#models).
41
+
42
+
## Calling a Model Routing Worker
43
+
44
+
Model routing workers act like any other LLM defined in `.models`. For example, they can be called via the OpenAI compatible HTTP endpoints.
45
+
```bash
46
+
curl http://localhost:8090/v1/chat/completions \
47
+
-H "Content-Type: application/json" \
48
+
-d '{
49
+
"model": "fallback",
50
+
"messages": [{ "role": "user", "content": "Tell me a joke"}]
0 commit comments