Docs for embeddings (#554)

* docs for embeddings

* Update local.md

* Apply suggestions from code review

Co-authored-by: Scott Lyons <scott@spice.ai>

---------

Co-authored-by: Scott Lyons <scott@spice.ai>

Author: Jeadie

spiceaidocs/docs/components/catalogs/index.md

@@ -2,7 +2,7 @@
 title: 'Catalog Connectors'
 sidebar_label: 'Catalog Connectors'
 description: ''
-sidebar_position: 5
+sidebar_position: 4
 pagination_prev: null
 pagination_next: null
 ---

spiceaidocs/docs/components/embeddings/huggingface.md

@@ -0,0 +1,26 @@
+---
+title: 'HuggingFace Text Embedding Models'
+sidebar_label: 'HuggingFace'
+sidebar_position: 2
+---
+
+To run an embedding model from HuggingFace, specify the `huggingface` path in `from`. This will handle downloading and running the embedding model locally.
+```yaml
+embeddings:
+  - from: huggingface:huggingface.co/sentence-transformers/all-MiniLM-L6-v2
+    name: all_minilm_l6_v2
+```
+
+Supported models include:
+ - All models tagged as [text-embeddings-inference](https://huggingface.co/models?other=text-embeddings-inference) on Huggingface
+ - Any Huggingface repository with the correct files to be loaded as a [local embedding model](/components/embeddings/local.md).
+
+
+With the same semantics as [language models](/components/models/huggingface#access-tokens), `spice` can run private HuggingFace embedding models:
+```yaml
+embeddings:
+  - from: huggingface:huggingface.co/secret-company/awesome-embedding-model
+    name: top_secret
+    params:
+      hf_token: ${ secrets:HF_TOKEN }
+```

spiceaidocs/docs/components/embeddings/index.md

@@ -0,0 +1,35 @@
+---
+title: 'Embedding Models'
+sidebar_label: 'Embeddings'
+description: ''
+sidebar_position: 6
+pagination_prev: null
+pagination_next: null
+---
+
+Embedding models are used to convert raw text into a numerical representation that can be used by machine learning models.
+
+Spice supports running embedding models locally, or use remote services such as OpenAI, or [la Plateforme](https://console.mistral.ai/).
+
+Embedding models are defined in the `spicepod.yaml` file as top-level components.
+
+```yaml
+embeddings:
+  - from: huggingface:huggingface.co/sentence-transformers/all-MiniLM-L6-v2
+    name: all_minilm_l6_v2
+
+  - from: openai:text-embedding-3-large
+    name: xl_embed
+    params:
+      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }
+
+  - name: my_model
+    from: file:model.safetensors
+    files:
+      - path: config.json
+      - path: models/embed/tokenizer.json
+```
+
+Embedding models can be used either by:
+ - An OpenAI-compatible [endpoint](/api/http/embeddings.md)
+ - By augmenting a dataset with column-level [embeddings](/reference/spicepod/datasets.md#embeddings), to provide vector-based [search functionality](/features/search/index.md#vector-search).

spiceaidocs/docs/components/embeddings/local.md

@@ -0,0 +1,21 @@
+---
+title: 'Local Filesystem Embedding Models'
+sidebar_label: 'Local'
+sidebar_position: 3
+---
+
+Embedding models can be run with files stored locally.
+
+```yaml
+embeddings:
+  - name: all_minilm_l6_v2
+    from: file:model.safetensors
+    files:
+      - path: /Users/jeadie/Github/spiceai/models/embed/config.json
+      - path: models/embed/tokenizer.json
+```
+
+## Required Files
+ - Model file, one of: `model.safetensors`, `pytorch_model.bin`.
+ - A tokenizer file with the filename `tokenizer.json`.
+ - A config file with the filename `config.json`.

spiceaidocs/docs/components/embeddings/openai.md

@@ -0,0 +1,34 @@
+---
+title: 'OpenAI (or Compatible) Embedding Models'
+sidebar_label: 'OpenAI'
+sidebar_position: 1
+---
+
+To use a hosted OpenAI (or compatible) embedding model, specify the `openai` path in `from`. 
+
+For a specific model, include it as the model ID in `from` (see example below). Defaults to `"text-embedding-3-small"`.
+These parameters are specific to OpenAI models:
+
+| Parameter | Description | Default |
+| ----- | ----------- | ------- |
+| `openai_api_key` | The OpenAI API key.        | -                           |
+| `openai_org_id` | The OpenAI organization id. | -                           |
+| `openai_project_id` | The OpenAI project id.  | -                           |
+| `endpoint` | The OpenAI API base endpoint.    | `https://api.openai.com/v1` |
+
+
+Example:
+
+```yaml
+models:
+  - from: openai:text-embedding-3-large
+    name: xl_embed
+    params:
+      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }
+
+  - name: mistral
+    from: openai:mistral-embed
+    params:
+      endpoint: https://api.mistral.ai/v1
+      api_key: ${ secrets:SPICE_MISTRAL_API_KEY }
+```

spiceaidocs/docs/components/models/index.md

@@ -2,6 +2,7 @@
 title: 'AI/ML Models'
 sidebar_label: 'AI/ML Models'
 description: ''
+sidebar_position: 5
 ---
 
 Spice supports traditional machine learning (ML) models and language models (LLMs).

spiceaidocs/docs/components/secret-stores/index.md

@@ -2,7 +2,7 @@
 title: 'Secret Stores'
 sidebar_label: 'Secret Stores'
 description: ''
-sidebar_position: 4
+sidebar_position: 3
 pagination_prev: null
 pagination_next: null
 ---

spiceaidocs/docs/components/views/index.md

@@ -2,6 +2,7 @@
 title: 'Views'
 sidebar_label: 'Views'
 description: 'Documentation for defining Views'
+sidebar_position: 7
 ---
 
 Views in Spice are virtual tables defined by SQL queries. They simplify complex queries and support reuse across applications.