Skip to content

Commit abbb4c0

Browse files
committed
add all worker docs
1 parent 40d544a commit abbb4c0

3 files changed

Lines changed: 61 additions & 1 deletion

File tree

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
title: 'Workers'
3+
sidebar_label: 'Workers'
4+
description: 'Overview of configurable workers in the Spice runtime.'
5+
---
6+
7+
Workers define a pattern of usage for one or more other spicepod components. The type of workers include:
8+
- [Routers](/docs/components/workers/routers.md): Define routing rules how one or more [llms](/docs/components/models/index.md) can be combined into a logically single model.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
title: 'Model Routing Workers'
3+
description: 'Instructions for defining routing rules for LLMs within Spice.'
4+
sidebar_label: 'LLM Routing'
5+
sidebar_position: 1
6+
---
7+
8+
Model routing workers define how one or more [llms](/docs/components/models/index.md) can be combined into a logically single model. Workers are defined as components in the `spicepod.yaml`.
9+
10+
**Example:**
11+
12+
```yaml
13+
workers:
14+
- name: round-robin
15+
description: |
16+
Call models 'foo' & 'bar' in round robin.
17+
models:
18+
- from: foo
19+
- from: bar
20+
- name: fallback
21+
description: |
22+
Call 'bar'. On error, call 'foo'. Failing that 'baz'.
23+
models:
24+
- from: foo
25+
order: 2
26+
- from: bar
27+
order: 1
28+
- from: baz
29+
order: 3
30+
- name: weighted
31+
description: |
32+
Routes 80% of traffic to 'foo'.
33+
models:
34+
- from: foo
35+
order: 4
36+
- from: bar
37+
order: 1
38+
```
39+
40+
For a complete specification of the routing rule options and syntax see the [specifications](/docs/reference/spicepod/workers.md#models).
41+
42+
## Calling a Model Routing Worker
43+
44+
Model routing workers act like any other LLM defined in `.models`. For example, they can be called via the OpenAI compatible HTTP endpoints.
45+
```bash
46+
curl http://localhost:8090/v1/chat/completions \
47+
-H "Content-Type: application/json" \
48+
-d '{
49+
"model": "fallback",
50+
"messages": [{ "role": "user", "content": "Tell me a joke"}]
51+
}'
52+
```

website/docs/reference/spicepod/workers.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ A unique identifier for this worker component.
4848

4949
Additional details about the worker, useful for displaying to users
5050

51-
### `models`
51+
### `models` {#models}
5252

5353
A list of model configurations that define how the model worker behaves.
5454

0 commit comments

Comments
 (0)