Merge branch 'lancedb:main' into feat/embedding-model-voyage-multimodal-3.5

Author: fzowl

AGENTS.md

@@ -0,0 +1,20 @@
+# LanceDB Mintlify Documentation
+
+This is a documentation site for [LanceDB](https://docs.lancedb.com).
+
+## Languages used
+
+- Code examples are primarily in three language SDKs: Python, TypeScript and Rust.
+- Best practices for linting, formatting and code complexity for each respective language apply.
+- Write idiomatic code as far as possible
+
+## Running Python code
+
+When running Python code, we have to cater to users of both pip and uv.
+
+- Use 4 spaces to represent a tab (do not use tab characters)
+- Always attempt to first run *any* Python code via the local virtual environment
+  - Look for a local virtual environment (typically in `.venv` or `venv`)
+  - Activate the environment, so that you can run multiple code exampes in the same environment
+- Avoid using `uv run` directly, as you have issues running it in your sandbox
+- Only fall back to the system `python3` to run code if the above steps don't work

CLAUDE.md

@@ -0,0 +1,20 @@
+# LanceDB Mintlify Documentation
+
+This is a documentation site for [LanceDB](https://docs.lancedb.com).
+
+## Languages used
+
+- Code examples are primarily in three language SDKs: Python, TypeScript and Rust.
+- Best practices for linting, formatting and code complexity for each respective language apply.
+- Write idiomatic code as far as possible
+
+## Running Python code
+
+When running Python code, we have to cater to users of both pip and uv.
+
+- Use 4 spaces to represent a tab (do not use tab characters)
+- Always attempt to first run *any* Python code via the local virtual environment
+  - Look for a local virtual environment (typically in `.venv` or `venv`)
+  - Activate the environment, so that you can run multiple code exampes in the same environment
+- Avoid using `uv run` directly, as you have issues running it in your sandbox
+- Only fall back to the system `python3` to run code if the above steps don't work

README.md

@@ -1,24 +1,36 @@
 # LanceDB Mintlify Documentation
 
-Home of the new LanceDB documentation on Mintlify.
+Home of the [LanceDB](https://lancedb.com/) documentation. Built using [Mintlify](https://www.mintlify.com/).
 
 ## Development
 
 Install the [Mintlify CLI](https://www.npmjs.com/package/mintlify) to preview the documentation changes locally. To install, use the following command
 
-```
+```bash
 npm i -g mintlify
 ```
 
 Run the following commands at the root of the documentation (`/docs/` in this repo, where `docs.json` is located).
 
-```
+```bash
 cd docs
 mint dev
 ```
 
+Check broken links (applies to internal links within this docs site only):
+
+```bash
+mint broken-links
+```
+
 ## Generate snippets
 
+To generate snippets, use `uv` to sync your local Python environment so that you can run the Python script described below.
+
+```bash
+uv sync
+```
+
 The Python, TypeScript and Rust code snippets used in the documentation are tested prior to use in the docs. These tests are located in the `tests/` directory. Run the tests locally for each language
 when building the docs locally.
 
@@ -38,17 +50,20 @@ make rs
 make snippets
 ```
 
+The generated snippets are placed in the appropriate file in `/docs/snippets/` directory, making them
+available for importing in the corresponding file.
+
 The following sequence of steps are run:
 
-1. Run tests for py, ts, rs files to verify that the code works
-2. Generate MDX snippets
-3. Import MDX snippets in the corresponding documentation page
-4. Add the NDX snippet inside a `<CodeBlock>` component in Mintlify
+1. Run tests for py, ts, rs files that contain new code you added, and verify that the tests pass locally
+2. Generate MDX snippets via the `make snippets` command
+3. Import MDX snippets in the corresponding MDX docs page
+4. Include the MDX snippet as a parameter inside a `<CodeBlock>` JSX component in Mintlify
 
-This ensures that the code in the docs is in line with the latest LanceDB API
+Creating and using snippets for code blocks in the MDX files helps ensure that we are placing
+code that's been tested (per recent LanceDB releases) in the hands of users.
 
 > [!NOTE]
-> Do not add code snippets manually inside triple-backticks! Write the tests to the `tests` directory,
-> then generate the snippets programmatically via the Makefile commands. This helps ensure that
-> the documentation shows code that was actually run by a human, and by CI.
-
+> As far as possible, do not add code snippets manually inside triple-backticks! Write the tests for
+> the required language in `tests/*` directory, then generate the snippets programmatically via the Makefile
+> commands.

WRITING.md

@@ -0,0 +1,199 @@
+# Writing Guide
+
+This is a documentation site built in [Mintlify](https://www.mintlify.com/docs). Writing in Mintlify is similar to `README.md`
+docs you may be used to writing in markdown -- the main difference is that Mintlify uses MDX files (Markdown + JSX) files
+instead of regular markdown files.
+
+It's worth going through the key sections in the Mintlify [docs](https://www.mintlify.com/docs) before you begin writing.
+To begin writing, create a new MDX file in the appropriate location and follow the steps below.
+
+## 1. Create the MDX file
+
+As far as possible, docs are organized at the top level by concept, in the `/docs/` directory. In certain cases,
+an additional level of nesting into an inner subdirectory is okay to avoid cluttering in the sidebar. However, it's
+recommended to avoid nesting more than 2 levels deep as this affects the flow and discoverability from a user
+perspective.
+
+## 2. Update `docs.json`
+
+The sitemap is defined in JSON format in `docs.json`. You can organize new content into tab groups and pages, as per the
+structure shown in the existing `docs.json`. To view the new page in the sidebar and in the local preview, it must be
+referenced in `docs.json` in the appropriate location.
+
+## 3. Add frontmatter
+
+Frontmatter is written in YAML, and is compulsory for all MDX files that contain documentation. It's recommended to _always_
+have at least the first three keys (`title`, `sidebarTitle` and `description`) for readability and SEO on a given docs page.
+Specifying an `icon` helps readers associate a familiar image with the page title in the sidebar. For searchability within
+the docs, you can optionally specify the `keywords` field and pass in a list of keyword strings. When a user searches for
+those strings, the page is prioritized in the search box.
+
+Here's an example:
+
+```yml
+---
+title: "Lance format"
+sidebarTitle: "Lance format"
+description: "Open-source lakehouse format for multimodal AI."
+icon: "/static/assets/logo/lance-logo-gray.svg"
+keywords: ["lance"]
+---
+```
+
+> [!NOTE]
+> The example above showed a custom SVG icon in the `/static/assets/` directory of this repo, but you can pick stock
+> icons from [fontawesome.com](https://fontawesome.com/icons) by searching for a high-level concept by name.
+
+## 4. Begin writing
+
+Writing in Mintlify is similar to conventional markdown, except that you have access to JSX-based (React) components that
+make it much simple to add documentation-friendly functionality and aesthetics to the docs page. Components are a very powerful
+addition to the writing experience, and are covered in detail on the [Mintlify docs](https://www.mintlify.com/docs/components/accordions).
+
+Below is an example of a `Card`, which emphasizes content, while providing a clickable URL out of the given page.
+```jsx
+<Card
+    title="Quickstart"
+    icon="rocket"
+    href="/quickstart"
+>
+Get started with LanceDB in minutes.
+</Card>
+```
+
+The best part about components is that they are composable. You can embed one component inside another and achieve the functionality of both. The example below shows an `Card` at the top level, with an `Accordion` inside it.
+
+```mdx
+<Card
+    title="Quickstart"
+    icon="rocket"
+    href="/quickstart"
+>
+Get started with LanceDB in minutes.
+
+<Accordion>
+Collapsible text content here....
+</Accordion>
+
+</Card>
+```
+
+## 5. Mathematical equations
+
+Math equations are supported via standard KaTeX plugins. You can write any LaTeX-style equation and get it rendered on the
+page by enclosing it in `$$` symbols.
+
+```mdx
+$$
+E = mc^2
+$$
+```
+
+## 6. Code snippets
+
+Code snippets are where Mintlify probably differs the most from markdown. There are several ways to write code snippets, but
+this section describes how we do it specifically in these LanceDB docs.
+
+### Option 1: `CodeGroup` components
+
+The preferred way to include a code snippet is to enter it within <CodeGroup> tags, as follows:
+
+```mdx
+<CodeGroup>
+```python Python icon="python"
+import lancedb
+```
+
+```typescript TypeScript icon="square-js"
+import * as lancedb from "@lancedb/lancedb";
+```
+
+```rust Rust icon="rust"
+use lancedb::connect;
+```
+</CodeGroup>
+
+
+This will allow you to include code snippets from multiple languages, grouped together on the docs page so that the user
+can click on their language of choice via tabs.
+
+### Option 2: `CodeBlock` components within `CodeGroup`
+
+As engineers, we may want to write a testable snippet in code in the `tests/py`, `tests/ts`, or `tests/rs` directory.
+These directories contain test files in each language that contain valid, tested code, which are fenced within comment markers
+so that they can be parsed by a [snippet generation script](./scripts/mdx_snippets_gen.py).
+
+The snippet generation script is run to extract the relevant snippets from the file (based on the fenced comment markers
+indicating `start` and `end` in each test file).
+
+Here's how you'd call the snippet into a code block in the MDX file:
+
+```mdx
+import { PyConnect, TsConnect, RsConnect } from '/snippets/connection.mdx';
+
+<CodeGroup >
+    <CodeBlock filename="Python" language="Python" icon="python">
+    {PyConnect}
+    </CodeBlock>
+
+    <CodeBlock filename="TypeScript" language="TypeScript" icon="square-js">
+    {TsConnect}
+    </CodeBlock>
+
+    <CodeBlock filename="Rust" language="Rust" icon="rust">
+    {RsConnect}
+    </CodeBlock>
+</CodeGroup >
+```
+
+### Option 3: Vanilla backticks
+
+This is the least preferred approach, as it doesn't let you group together code snippets from multiple languages effectively.
+Note that Mintlify offers some additional features compared to traditional markdown even when using triple backticks.
+
+In the example below, we may have a long code snippet that we want to collapse (to show a few lines in the rendered page).
+This is useful for example code or data snippets that are quite long.
+
+```json camelot.json icon="brackets-curly" expandable=true
+[
+  {
+    "id": 1,
+    "name": "King Arthur",
+    "role": "King of Camelot",
+    "description": "The legendary ruler of Camelot, wielder of Excalibur, and leader of the Knights of the Round Table.",
+    "vector": [0.72, -0.28, 0.60, 0.86],
+    "stats": { "strength": 2, "courage": 5, "magic": 1, "wisdom": 4 }
+  }
+]
+```
+
+Using vanilla backticks is okay when the code snippets like JSON blobs can get really long, and we only want to show a
+preview to the reader. Enabling `expandable=true` allows readers to see the whole block when they click on the "expand"
+button on the page.
+
+## 7. Run local deployment
+
+After you update the `docs.json` page with the path to the new MDX file, you can debug the site on the local deployment.
+
+```bash
+# cd to the docs/ directory
+cd docs
+# Run local server
+mint dev
+```
+This will run a local deployment on `localhost:3000`, which is useful for debugging and testing purposes.
+
+You can check for broken lines in the site by running the following command.
+
+```bash
+mint broken-links
+```
+
+> [!NOTE]
+> The broken link checker **only** checks for internal (relative) links to other pages within this docs repo.
+> It cannot check external site links.
+
+## 8. Commit to the docs repo
+
+Once you've finished writing and reviewing the content yourself, submit a PR to the [repo](https://github.com/lancedb/docs)
+for review. If you're an external contributor, we thank you for your contribution to LanceDB!

docs/api-reference/index.mdx

@@ -17,7 +17,7 @@ Python, Typescript and Rust SDKs are officially supported by LanceDB.
 | SDK Reference | Description |
 |:--------------|-------------------|
 | [Python SDK](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration |
-| [Typescript SDK](https://lancedb.github.io/lancedb/js/) | A Typescipt wrapper around the Rust library, built with `napi-rs`
+| [Typescript SDK](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs`
 | [Rust SDK](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance |
 
 ## Examples in other languages

docs/docs.json

@@ -1,7 +1,6 @@
 {
   "$schema": "https://mintlify.com/docs.json",
   "appearance": {
-    "default": "light",
     "strict": false
   },
   "theme": "mint",
@@ -277,7 +276,8 @@
                   "tutorials/agents/time-travel-rag/index",
                   "tutorials/agents/multimodal-agent/index"
                 ]
-              }
+              },
+              "tutorials/feature-engineering/index"
             ]
           }
         ]

docs/embedding/index.mdx

@@ -9,13 +9,17 @@ Modern machine learning models can be trained to convert raw data into embedding
 of floating point numbers. The position of an embedding in vector space captures the semantics of
 the data, so vectors that are close to each other are considered similar.
 
-LanceDB provides an embedding function registry that automatically generates vector embeddings
+LanceDB provides an embedding function registry in OSS as well as its Cloud and Enterprise versions
+([see below](#embeddings-in-lancedb-cloud-and-enterprise)).
+that automatically generates vector embeddings
 during data ingestion and querying. The API abstracts embedding generation, allowing you to focus
 on your application logic.
 
-## Embedding function registry
+## Embedding Registry
 
-You can get a supported embedding function from the registry, and then use it in your table schema. 
+<Badge color="green">OSS</Badge>
+
+In LanceDB OSS, you can get a supported embedding function from the registry, and then use it in your table schema. 
 Once configured, the embedding function will automatically generate embeddings when you insert data
 into the table. And when you query the table, you can provide a query string or other input, and the
 embedding function will generate an embedding for it.
@@ -98,12 +102,14 @@ LanceDB supports most popular embedding providers.
 
 You can find all supported embedding models in the [integrations](/integrations/embedding) section.
 
-## Embedding function on LanceDB cloud
-When using embedding functions on LanceDB cloud, during the ingestion time the embeddings are
-generated on the client side, and stored in the cloud. We don't yet support model inference on the
-cloud side so automatic query generation during search is not supported. You can manually generate
-the embeddings for your queries using the same embedding function and pass the vector to the search
-function.
+## Embeddings in LanceDB Cloud and Enterprise
+Currently, the embedding registry on LanceDB <Badge color="purple">Cloud</Badge> or
+<Badge color="red">Enterprise</Badge> supports automatic generation of embeddings during data ingestion,
+generated on the client side (and stored on the remote table). We don't yet support automatic query-time
+embedding generation when sending queries, though this is planned for a future release.
+
+For now, you can manually generate the embeddings at query time using the same embedding function that
+was used during ingestion, and pass the embeddings to the search function.
 
 <CodeGroup>
 ```python Python icon="python"
@@ -132,8 +138,8 @@ results = table.search(query_vector).limit(5).to_pandas()
 
 ## Custom Embedding Functions
 
-You can implement your own embedding function by inheriting from `TextEmbeddingFunction` (for text)
-or `EmbeddingFunction` (for multimodal data).
+You can always implement your own embedding function by inheriting from `TextEmbeddingFunction`
+(for text) or `EmbeddingFunction` (for multimodal data).
 
 <CodeGroup>
 ```python Python icon="python"