More documentation updates

Author: travis-bauer

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Unreleased
 
+- Documentation: fixed internal link and config overview ordering; developer handbook grammar, `$var_name`
+  explanation, and `default_embedding_model_source` key; pip install examples; RAG/config notes on
+  `DEFAULT_*` vs `default_*` keys for `serverag` vs segments; minor README and workbench markdown fixes.
+
 - README: stronger opening (audience, fit, differentiation), badges and Python version note, shorter
   architecture section, documentation map table, tightened Quick Start and config pointer, RAG “at a
   glance” under Quick Start, per-example problem/result blurbs, and status note on what is stable;

docs/README.md

@@ -1,6 +1,6 @@
 # TalkPipe Documentation
 
-Welcome to the TalkPipe documentation! This directory contains comprehensive, API references, and examples for using TalkPipe effectively.
+Welcome to the TalkPipe documentation! This directory contains guides, API references, and examples for using TalkPipe effectively.
 
 ## Quick Navigation
 
@@ -19,10 +19,10 @@ Real-world usage examples and tutorials:
 
 ### 📚 [API Reference](api-reference/)
 Complete documentation for all TalkPipe commands and components:
-- [chatterlang_serve](api-reference/chatterlang-server.md) - Run a script as an API endpoint and with a customizable web interface
+- [chatterlang_serve](api-reference/chatterlang-server.md) - Run a script as an API endpoint with a customizable web interface
 - [ChatterLang Workbench](api-reference/chatterlang-workbench.md) - Interactive web interface for writing and testing scripts interactively
 - [ChatterLang Script Runner](api-reference/chatterlang-script.md) - Run a script from the command line
-- [Documentation Generator](api-reference/talkpipe-ref.md) - Generate reference documentations for Segments and Sources
+- [Documentation Generator](api-reference/talkpipe-ref.md) - Generate reference documentation for Segments and Sources
 - [Plugin Manager](api-reference/talkpipe-plugin-manager.md) - Manage and inspect TalkPipe plugins
 
 ### 🏗️ [Architecture](architecture/)

docs/api-reference/chatterlang-workbench.md

@@ -132,7 +132,7 @@ chatterlang_workbench --reload
 
 ---
 
-*For conceptual information about ChatterLang, see [ChatterLang Architecture](../architecture/chatterlang.md). 
+*For conceptual information about ChatterLang, see [ChatterLang Architecture](../architecture/chatterlang.md).*
 
 ---
 Last Reviewed: 20250814

docs/api-reference/lazy-loading.md

@@ -574,6 +574,6 @@ print(f'Total: {len(seen)} unique names')
 
 **See Also:**
 - [Plugin Manager](talkpipe-plugin-manager.md) for managing external plugins
-- [ChatterLang Compiler](../architecture/compiler.md) for how components are resolved during compilation
+- [ChatterLang compiler layer](../architecture/chatterlang.md#2-compiler-layer) for how components are resolved during compilation
 
 Last Reviewed: 20251025

docs/architecture/configuration.md

@@ -6,9 +6,9 @@ This document describes how TalkPipe manages configuration across different envi
 
 TalkPipe uses a layered configuration system that supports multiple sources with clear precedence rules. Configuration can come from:
 
-3. **Command-line arguments** (including unknown arguments that get added to the configuration)
+1. **Command-line arguments** (including unknown arguments that get added to the configuration)
 2. **Environment variables** (with `TALKPIPE_` prefix)
-1. **Configuration files** (TOML format)
+3. **Configuration files** (TOML format)
 4. **Default values** (hard-coded in the application)
 
 ## Configuration File Formats
@@ -146,6 +146,15 @@ For application settings (logging, server ports, etc.):
 3. **Configuration file values** (`~/.talkpipe.toml`)
 4. **Application defaults** (hard-coded values)
 
+### Embedding and LLM defaults: `serverag` vs segment keys
+
+Several components resolve embedding and chat model defaults from `get_config()`. Two naming patterns appear in configuration:
+
+- **Segment defaults** — `LLMEmbed` and `LLMPrompt` fall back to these keys when `model` / `source` arguments are omitted: `default_embedding_model_name`, `default_embedding_model_source`, `default_model_name`, and `default_model_source` (see `talkpipe.util.constants`).
+- **`serverag` defaults** — When you omit `--embedding_model`, `--embedding_source`, `--completion_model`, and `--completion_source`, `serverag` reads `DEFAULT_EMBEDDING_MODEL`, `DEFAULT_EMBEDDING_SOURCE`, `DEFAULT_LLM_MODEL`, and `DEFAULT_LLM_SOURCE` from the merged config. If those keys are unset, `serverag` passes `None` into the RAG pipeline, and the segments above apply their `default_*` fallbacks.
+
+Use **`default_*`** in `~/.talkpipe.toml` for one consistent set of defaults across pipelines. Use **`DEFAULT_*`** when you want values applied at the `serverag` CLI layer (still overridable per invocation). Environment variables use the usual `TALKPIPE_` prefix and map to the exact key name after the prefix (for example, `TALKPIPE_default_model_name` or `TALKPIPE_DEFAULT_LLM_MODEL`).
+
 ### ChatterLang Script Variable Access
 
 **Configuration Variables** (accessed with `$key` syntax in scripts):

docs/contributing/developer-handbook.md

@@ -4,9 +4,9 @@ Contributor-oriented reference: glossary, repository conventions, parameter sema
 
 ## Glossary
 
-* **Unit** - A component in a pipeline that either produces or processes data.  There are two types of units, Source, and Segments.
+* **Unit** - A component in a pipeline that either produces or processes data.  There are two types of units: Sources and Segments.
 * **Segment** - A unit that reads from another Unit and may or may not yield data of its own.  All units that
-are not at the start of a pipeline is a Segment.
+are not at the start of a pipeline are Segments.
 * **Source** - A unit that takes nothing as input and yields data items.  These Units are used in the
 "INPUT FROM..." portion of a pipeline.
 
@@ -24,9 +24,9 @@ The following are the main breakdown of the codebase. These should be considered
   * Example: chatterlang_script
 * **talkpipe.operations** - Contains general algorithm implementations.  Associated segments and sources can be included next to the algorithm implementations, but the algorithms themselves should also work stand-alone.
   * Example: bloom filters
-* **talkpipe.data** - Contain components having to do with complex, type-specific data manipulation.
+* **talkpipe.data** - Contains components having to do with complex, type-specific data manipulation.
   * Example: extracting text from files.
-* **talkpipe.llm** - Contain the abstract classes and implementations for accessing LLMs, both code for accessing specific LLMs and code for doing prompting.
+* **talkpipe.llm** - Contains the abstract classes and implementations for accessing LLMs, both code for accessing specific LLMs and code for doing prompting.
   * Example: Code for talking with Ollama or OpenAI
 * **talkpipe.pipe** - Code that implements the core classes and decorators for the pipe api as well and misc implementations of helper segments and sources.
   * Example: echo and the definition of the @segment decorator
@@ -58,7 +58,7 @@ These parameter names should behave consistently across all units:
   If used, any processed output is attached to the original data using bracket notation. The original item is then emitted.
 
 - **fail_on_error**
-  If True, an operation the exception should be raised, likely aborting the pipeline.  If False, the operation should continue
+  If True, the exception should be raised, likely aborting the pipeline.  If False, the operation should continue
   and either None should be yielded or nothing, depending on the segment or source.  A warning message should be logged.
 
 - **field**
@@ -68,7 +68,7 @@ These parameter names should behave consistently across all units:
 - **field_list**
   Specifies that a list of fields can or should be provided, with each field separated
   by a comma.  In some cases, each field needs to be mapped to some other name.  In
-  those case, the field and name should be separated by a colon.  In field_lists,
+  those cases, the field and name should be separated by a colon.  In field_lists,
   the underscore (_) refers to the item as a whole.
   - For example, "X.2.0:SomeName,X.1:SomeOtherName".  If no "name" is provided,
   the fieldname itself is used.  Where only a list of fields is needed and no names,
@@ -88,11 +88,9 @@ After talkpipe is installed, a script called "chatterlang_reference_browser" is
 
 ### Standard Configuration File Items
 
-Configuration constants can be defined either in ~/.talkpipe.toml or in environment variables.  Any constant defined in an environment variable needs to be prefixed with TALKPIPE_.  So email_password, stored in an environment variable, needs to be TALKPIPE_email_password.  Note that in Chatterlang, any variable stored in the format
-can be specified as a parameter using $var_name.  This will get dereferenced to
-the environment variable TALKPIPE_var_name or var_name in talkpipe.toml.
+Configuration constants can be defined either in ~/.talkpipe.toml or in environment variables.  Any constant defined in an environment variable needs to be prefixed with TALKPIPE_.  So email_password, stored in an environment variable, needs to be TALKPIPE_email_password.  Note that in ChatterLang, any key defined in ~/.talkpipe.toml or set via a TALKPIPE_* environment variable can be referenced in scripts as a parameter using $var_name.  That reference resolves to the environment variable TALKPIPE_var_name or to var_name in talkpipe.toml.
 
-* **default_embedding_source** - The default source (e.g. ollama) to be used for creating sentence embeddings.
+* **default_embedding_model_source** - The default source (e.g. ollama) to be used for creating sentence embeddings.
 * **default_embedding_model_name** - The name of the LLM model to be used for creating sentence embeddings.
 * **default_model_name** - The default name of a LLM model to be used in chat
 * **default_model_source** - The default source (e.g. ollama) to be used in chat

docs/guides/makevectordatabase-and-serverag.md

@@ -19,7 +19,7 @@ Together they form a minimal path from raw documents to a queryable RAG interfac
 
 ## Prerequisites
 
-- **TalkPipe** with LLM support: `pip install talkpipe[ollama]` or `talkpipe[all]`
+- **TalkPipe** with LLM support: `pip install talkpipe[ollama]` or `pip install talkpipe[all]`
 - **Embedding model**: Ollama with an embedding model (e.g. `ollama pull mxbai-embed-large`)
 - **Completion model** (for serverag): Ollama with an LLM (e.g. `ollama pull llama3.2`)
 - **Configuration**: Set `DEFAULT_EMBEDDING_MODEL`, `DEFAULT_EMBEDDING_SOURCE`, `DEFAULT_LLM_MODEL`, and `DEFAULT_LLM_SOURCE` in `~/.talkpipe.toml` or pass them on the command line

docs/tutorials/README.md

@@ -16,7 +16,7 @@ Each tutorial builds on the previous. Tutorial 1's index feeds Tutorial 2; Tutor
 
 ## Get Started in 5 Minutes
 
-1. **Install**: See [Getting Started](../quickstart.md) for installation. For tutorials: `pip install talkpipe[ollama]` or `talkpipe[all]`
+1. **Install**: See [Getting Started](../quickstart.md) for installation. For tutorials: `pip install talkpipe[ollama]` or `pip install talkpipe[all]`
 2. **Run Tutorial 1** (from `docs/tutorials/Tutorial_1-Document_Indexing`):
    - `./Step_1_CreateSyntheticData.sh` — creates `stories.json` (~5–10 min)
    - `./Step_2_IndexStories.sh` — builds search index (~5 sec)

docs/tutorials/Tutorial_1-Document_Indexing/README.md

@@ -27,7 +27,7 @@ TalkPipe lets you prototype searchable document systems without external databas
 
 ## Prerequisites
 
-- **TalkPipe** installed: See [Getting Started](../../quickstart.md) for installation. For this tutorial: `pip install talkpipe[ollama]` or `talkpipe[all]`
+- **TalkPipe** installed: See [Getting Started](../../quickstart.md) for installation. For this tutorial: `pip install talkpipe[ollama]` or `pip install talkpipe[all]`
 - **Step 1 only**: Ollama installed locally with the `llama3.2` model (or adjust the script to use another model)
 
 > **Tip:** If you skip Step 1, you can use the included `stories.json` and go straight to Step 2.

docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/README.md

@@ -28,7 +28,7 @@ Keyword search breaks when users ask "find documents similar to this" or "show m
 ## Prerequisites
 
 - **Tutorial 1 completed**: `stories.json` must exist at `../Tutorial_1-Document_Indexing/stories.json`
-- **TalkPipe** installed: See [Getting Started](../../quickstart.md). For this tutorial: `pip install talkpipe[ollama]` or `talkpipe[all]`
+- **TalkPipe** installed: See [Getting Started](../../quickstart.md). For this tutorial: `pip install talkpipe[ollama]` or `pip install talkpipe[all]`
 - **Ollama** with these models:
   - `mxbai-embed-large` (embeddings): `ollama pull mxbai-embed-large`
   - `llama3.2` (Step 3 only): `ollama pull llama3.2`