You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For more detail, refer to the `model`[reference specification](/reference/spicepod/models.md).
23
+
* LLM Format(s) may require additional files (e.g. `tokenizer_config.json`).
23
24
24
-
## Model Sources
25
-
26
-
import DocCardList from '@theme/DocCardList';
27
-
28
-
<DocCardList />
25
+
The model type is inferred based on the model source and files. For more detail, refer to the `model`[reference specification](/reference/spicepod/models.md).
description: 'Learn how to override default LLM hyperparameters in Spice.'
5
+
sidebar_position: 1
6
+
pagination_prev: null
7
+
pagination_next: null
8
+
---
9
+
10
+
### Chat Completion Parameter Overrides
11
+
[`v1/chat/completion`](/api/http/chat-completions) is an OpenAI compatible endpoint.
12
+
13
+
It supports all request body parameters defined in the [OpenAI reference documentation](https://platform.openai.com/docs/api-reference/chat/create). Spice can configure different defaults for these request parameters.
14
+
```yaml
15
+
models:
16
+
- name: pirate-haikus
17
+
from: openai:gpt-4o
18
+
params:
19
+
openai_temperature: 0.1
20
+
openai_response_format: { "type": "json_object" }
21
+
```
22
+
To specify a default override for a parameter, use the `openai_` prefix followed by the parameter name. For example, to set the `temperature` parameter to `0.1`, use `openai_temperature: 0.1`.
23
+
24
+
### System Prompt
25
+
In addition to any system prompts provided in message dialogue, or added by model providers, Spice can configure an additional system prompt.
26
+
```yaml
27
+
models:
28
+
- name: pirate-haikus
29
+
from: openai:gpt-4o
30
+
params:
31
+
system_prompt: |
32
+
Write everything in Haiku like a pirate
33
+
```
34
+
35
+
Any request to [HTTP `v1/chat/completion`](/api/http/chat-completions) will include the configured system prompt.
description: 'Learn how to configure language models in Spice.'
5
+
sidebar_position: 7
6
+
pagination_prev: null
7
+
pagination_next: null
8
+
---
9
+
10
+
Spice supports language models (LLMs) from several sources (see [model components](/components/models/index.md)) and provides configuration for how inference will be performed in the Spice runtime. This includes:
11
+
- Providing tools to the language model, enabling it to interact with the Spice runtime.
12
+
- Specifying system prompts and overriding defaults for [`v1/chat/completion`](/api/http/chat-completions.md).
description: 'Learn how LLMs can interact with the spice runtime.'
5
+
sidebar_position: 2
6
+
pagination_prev: null
7
+
pagination_next: null
8
+
---
9
+
10
+
Spice provides a set of tools that let LLMs interact with the runtime. To provide these tools to a Spice model, specify them in its `params.spice_tools`.
11
+
```yaml
12
+
models:
13
+
- name: sql-model
14
+
from: openai:gpt-4o
15
+
params:
16
+
spice_tools: list_datasets, sql, table_schema
17
+
18
+
- name: full-runtime
19
+
from: openai:gpt-4o
20
+
params:
21
+
spice_tools: auto # Use all available tools
22
+
```
23
+
24
+
## Available tools
25
+
- `list_datasets`: List all available datasets in the runtime.
26
+
- `sql`: Execute SQL queries on the runtime.
27
+
- `table_schema`: Get the schema of a specific SQL table.
28
+
- `document_similarity`: For datasets with an embedding column, retrieve documents based on an input query. It is equivalent to [/v1/search](/api/http/search).
Copy file name to clipboardExpand all lines: spiceaidocs/docs/reference/spicepod/models.md
+19-6Lines changed: 19 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,6 +16,15 @@ The model specifications are in early preview and are subject to change.
16
16
Spice supports both traditional machine learning (ML) models and language models (LLMs). The configuration allows you to specify either type from a variety of sources. The model type is automatically determined based on the model source and files.
|`name`| Unique, readable name for the model within the Spicepod. |
22
+
|`from`| Source-specific address to uniquely identify a model |
23
+
|`description`| Additional details about the model, useful for displaying to users |
24
+
|`datasets`| Datasets that the model depends on for inference |
25
+
|`files`| Specify additional files, or override default files needed by the model |
26
+
|`params`| Additional parameters to be passed to the model |
27
+
19
28
## `models`
20
29
21
30
The `models` section in your configuration allows you to specify one or more models to be used with your datasets.
@@ -41,13 +50,13 @@ models:
41
50
42
51
### `from`
43
52
44
-
The `from` field specifies both the source of the model, and the unique identifier of the model (relative to the source). The `from` value expects the following format
53
+
The `from` field specifies both the source of the model (e.g Huggingface, or a local file), and the unique identifier of the model (relative to the source). The `from` value expects the following format
45
54
46
55
```yaml
47
56
- from: <model_source>/<model id>
48
57
```
49
58
50
-
### Model Source
59
+
#### Model Source
51
60
52
61
The `<model_source>` prefix of the `from` field indicates where the model is sourced from:
53
62
@@ -56,7 +65,7 @@ The `<model_source>` prefix of the `from` field indicates where the model is sou
56
65
- `openai`- OpenAI (or compatible) models
57
66
- `spiceai`- Spice AI models
58
67
59
-
### Model ID
68
+
#### Model ID
60
69
61
70
The `<model_id>` suffix of the `from` field is a unique (per source) identifier for the model:
62
71
@@ -65,13 +74,17 @@ The `<model_id>` suffix of the `from` field is a unique (per source) identifier
65
74
- For Hugging Face: A repo_id and, optionally, revision hash or tag.
- For local files: Represents the absolute or relative path to the model weights file on the local file system. See [below](#files) for the accepted model weight types and formats.
77
+
- For local files: Represents the absolute or relative path to the model weights file on the local file system. See [below](#files) for the accepted model weight types and formats.
69
78
- For OpenAI: Only supports LMs. For OpenAI models, valid IDs can be found in their model [documentation](https://platform.openai.com/docs/models/continuous-model-upgrades). For OpenAI compatible providers, specify the value required in their `v1/chat/completion` [payload](https://platform.openai.com/docs/api-reference/chat/create#chat-create-model).
70
79
71
80
### `name`
72
81
73
82
A unique identifier for this model component.
74
83
84
+
### `description`
85
+
86
+
Additional details about the model, useful for displaying to users
87
+
75
88
### `files`
76
89
77
90
Optional. A list of files associated with this model. Each file has:
@@ -106,8 +119,8 @@ Optional. A map of key-value pairs for additional parameters specific to the mod
106
119
107
120
### `datasets`
108
121
109
-
Optional. A list of [dataset names](./datasets.md#name) that this model should be applied to. For ML models, this preselects the dataset to use for inference.
122
+
Optional. A list of [dataset names](./datasets.md#name) that this model should be applied to. For ML models, this preselects the dataset to use for inference.
110
123
111
124
### `dependsOn`
112
125
113
-
Optional. A list of dependencies that must be loaded and available before this model.
126
+
Optional. A list of dependencies that must be loaded and available before this model.
0 commit comments