docs: align documentation with public API

Author: preciz

.formatter.exs

@@ -1,4 +1,4 @@
 # Used by "mix format"
 [
-  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
+  inputs: ["{mix,.formatter}.exs", "{config,examples,lib,test}/**/*.{ex,exs}"]
 ]

CHANGELOG.md

@@ -13,43 +13,33 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 - Added `Chunx.Chunker.Recursive`, which progressively splits text at
   paragraph, sentence, punctuation, whitespace, and token boundaries.
-- Added a shared tokenizer boundary with support for native
-  `Tokenizers.Tokenizer` values and custom tokenizer adapters.
-- Added grapheme-safe handling for repeated, overlapping, and byte-level token
-  offsets.
+- Added custom tokenizer adapters through `Chunx.Tokenizer`.
 - Added an example script covering the available non-semantic chunkers.
 - Added opt-in real-model embedding integration tests, runnable with
   `mix test --include integration` or `mix test --only integration`.
-- Expanded property testing for reconstruction, offsets, overlap normalization,
-  semantic metadata, sentence grouping, tokenizer windows, Unicode, and
-  statistical invariants.
+- Added property tests for reconstruction, offsets, overlap, Unicode, tokenizer
+  windows, and semantic metadata.
 
 ### Changed
 
 - Standardized `token_count` across all chunkers to mean content tokens,
   excluding tokenizer entries without a byte span.
-- Refactored Token, Word, Sentence, Recursive, and Semantic chunking paths to
-  reduce intermediate collections and repeated traversal.
-- Optimized native token counting and ordinary token-window packing while
-  retaining the hardened Unicode fallback.
-- Simplified semantic similarity averaging without changing the adjacent
-  cosine-similarity model.
+- Made tokenizer-derived chunk boundaries grapheme-safe.
+- Reduced intermediate allocations and repeated traversal in the chunkers.
 - Updated Nx, EXLA, Bumblebee, Scholar, StreamData, and related dependencies.
 
 ### Fixed
 
 - Preserved exact byte offsets for repeated text and Unicode content.
-- Prevented SentenceChunker from stalling when overlap contains the entire
+- Prevented Sentence from stalling when overlap contains the entire
   previous chunk.
-- Preserved TokenChunker's trailing overlap window while keeping indivisible
+- Preserved Token's trailing overlap window while keeping indivisible
   graphemes intact.
-- Propagated tokenizer failures consistently instead of crashing on failed
-  encodings.
+- Propagated tokenizer failures instead of crashing on failed encodings.
 - Rejected malformed tokenizer responses and offsets with tagged errors.
-- Prevented SemanticChunker from silently dropping sentences when an embedding
+- Prevented Semantic from silently dropping sentences when an embedding
   function returns the wrong number of embeddings.
-- Corrected sentence grouping and chunk-boundary behavior to preserve all input
-  text.
+- Preserved all input text when grouping sentences and placing chunk boundaries.
 
 ## [0.1.0] - 2026-03-04
 

README.md

@@ -2,29 +2,13 @@
 
 [![test](https://github.com/preciz/chunx/actions/workflows/test.yml/badge.svg)](https://github.com/preciz/chunx/actions/workflows/test.yml)
 
-Chunx is an Elixir library for splitting text into meaningful chunks using various strategies. It's particularly useful for processing large texts for LLMs, semantic search, and other NLP tasks.
-
-## Credit
-
-This library is based on [chonkie-ai/chonkie](https://github.com/chonkie-ai/chonkie)
-
-## Features
-
-- Multiple chunking strategies:
-  - Token-based chunking
-  - Word-based chunking
-  - Sentence-based chunking
-  - Semantic chunking with embeddings
-  - Recursive chunking using structural boundaries
-
-- Configurable options for each strategy
-- Support for overlapping chunks
-- Token count tracking
-- Embedding support
+Chunx splits text by tokens, words, sentences, document structure, or semantic
+similarity. It is an Elixir implementation inspired by
+[Chonkie](https://github.com/chonkie-ai/chonkie).
 
 ## Installation
 
-Add `chunx` to your list of dependencies in `mix.exs`:
+Add Chunx to `mix.exs`:
 
 ```elixir
 def deps do
@@ -36,83 +20,66 @@ end
 
 ## Usage
 
-### Token-based Chunking
-
-```elixir
-{:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
-{:ok, chunks} = Chunx.Chunker.Token.chunk("Your text here", tokenizer, chunk_size: 512)
-```
-
-### Word-based Chunking
+All chunkers require a tokenizer. They accept a `Tokenizers.Tokenizer` or a
+custom adapter implementing the `Chunx.Tokenizer` behaviour.
 
 ```elixir
-{:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
-{:ok, chunks} = Chunx.Chunker.Word.chunk("Your text here", tokenizer, chunk_size: 512)
-```
-
-### Sentence-based Chunking
+alias Chunx.Chunker.Token
 
-```elixir
 {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
-{:ok, chunks} = Chunx.Chunker.Sentence.chunk("Your text here", tokenizer)
+{:ok, chunks} = Token.chunk("Text to split", tokenizer, chunk_size: 128)
 ```
 
-### Semantic Chunking
+Each returned chunk contains its text, half-open byte offsets into the original
+text, and its content-token count. Sentence and Semantic return
+`Chunx.SentenceChunk` structs; the other chunkers return `Chunx.Chunk` structs.
 
-```elixir
-{:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
+### Chunkers
 
-# The embedding function must return a list of Nx.Tensor.t()
-embedding_fn = fn texts ->
-  # Your embedding function here
-end
+| Module | Splitting unit | Overlap |
+| --- | --- | --- |
+| `Chunx.Chunker.Token` | Token offsets | Token count or fraction |
+| `Chunx.Chunker.Word` | Whole words | Token count or fraction |
+| `Chunx.Chunker.Sentence` | Whole sentences | Whole sentences within a token budget |
+| `Chunx.Chunker.Recursive` | Configured structural levels, then tokens | None |
+| `Chunx.Chunker.Semantic` | Sentence-embedding similarity | None |
 
-{:ok, chunks} = Chunx.Chunker.Semantic.chunk("Your text here", tokenizer, embedding_fn)
-```
+See the [API documentation](https://hexdocs.pm/chunx/) for each module's options
+and size-limit exceptions.
 
-### Recursive Chunking
+Semantic chunking also requires a function that returns one `Nx.Tensor` for
+each input string:
 
 ```elixir
-{:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
-{:ok, chunks} = Chunx.Chunker.Recursive.chunk("Your text here", tokenizer)
-```
-
-Recursive chunking tries paragraphs, sentences, punctuation, whitespace, and
-finally token boundaries until every chunk fits within the configured size.
+alias Chunx.Chunker.Semantic
 
-## Configuration
+embedding_fun = &MyApp.Embeddings.embed/1
 
-Each chunking strategy accepts various options to customize the chunking behavior:
-
-- `chunk_size`: Maximum number of content tokens per chunk
-- `chunk_overlap`: Number or proportion of content tokens shared by consecutive chunks
-- `min_sentences_per_chunk`: Minimum number of sentences per sentence-based chunk
-- `min_sentences`: Minimum number of sentences per semantic chunk
-- `threshold`: Similarity threshold for semantic chunking
-- And more...
-
-See the documentation for each chunker module for detailed configuration options.
+{:ok, chunks} =
+  Semantic.chunk("Text to split", tokenizer, embedding_fun,
+    chunk_size: 128,
+    threshold: :auto
+  )
+```
 
 ## Testing
 
-```elixir
-# Run the test suite
+Run the regular suite:
+
+```bash
 mix test
 ```
 
-Real-model embedding integration tests are excluded by default because they
-download and run a Hugging Face model. Enable them explicitly with:
+Embedding integration tests use
+`sentence-transformers/all-MiniLM-L6-v2`. They download and run the model, so
+they are excluded by default:
 
 ```bash
-mix test --include integration
+mix test --only integration
 ```
 
-To run only the integration tests, use `mix test --only integration`.
-
-The default model is `sentence-transformers/all-MiniLM-L6-v2`. Override it with
-`CHUNX_EMBEDDING_MODEL`, provided the model is supported by Bumblebee's text
-embedding serving and `Tokenizers.Tokenizer.from_pretrained/1`.
+Use `mix test --include integration` to run both suites together.
 
 ## License
 
-[MIT License](LICENSE)
+[MIT](LICENSE)

examples/demo.exs

@@ -3,43 +3,29 @@ Mix.install([
 ])
 
 defmodule Demo do
-  def run do
-    {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
-    
-    text = """
-    The process of text chunking in RAG applications represents a delicate balance between competing requirements. On one side, we have the need for semantic coherence – ensuring that each chunk maintains meaningful context that can be understood and processed independently. On the other, we must optimize for information density, ensuring that each chunk carries sufficient signal without excessive noise that might impede retrieval accuracy. In this post, we explore the challenges of text chunking in RAG applications and propose a novel approach that leverages recent advances in transformer-based language models to achieve a more effective balance between these competing requirements.
-    
-    # Heading 1
-    This is a paragraph with some **bold text** and _italic text_.
-    
-    ## Heading 2
-    - Bullet point 1
-    - Bullet point 2 with `inline code`
-    """
+  @text """
+  Text chunking divides a document into smaller sections. A tokenizer measures
+  their size. Different chunkers preserve different boundaries.
 
-    IO.puts("=== Token-based Chunking ===")
-    {:ok, token_chunks} = Chunx.Chunker.Token.chunk(text, tokenizer, chunk_size: 50, chunk_overlap: 10)
-    Enum.each(Enum.with_index(token_chunks), fn {chunk, i} ->
-      IO.puts("Chunk #{i} (Tokens: #{chunk.token_count}):\n#{chunk.text}\n")
-    end)
+  Sentence chunking keeps sentences together. Recursive chunking first tries
+  document structure, then falls back to smaller boundaries.
+  """
 
-    IO.puts("\n=== Word-based Chunking ===")
-    {:ok, word_chunks} = Chunx.Chunker.Word.chunk(text, tokenizer, chunk_size: 50, chunk_overlap: 10)
-    Enum.each(Enum.with_index(word_chunks), fn {chunk, i} ->
-      IO.puts("Chunk #{i} (Tokens: #{chunk.token_count}):\n#{chunk.text}\n")
-    end)
+  def run do
+    {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
 
-    IO.puts("\n=== Sentence-based Chunking ===")
-    {:ok, sentence_chunks} = Chunx.Chunker.Sentence.chunk(text, tokenizer, chunk_size: 50, chunk_overlap: 10)
-    Enum.each(Enum.with_index(sentence_chunks), fn {chunk, i} ->
-      IO.puts("Chunk #{i} (Tokens: #{chunk.token_count}):\n#{chunk.text}\n")
-    end)
+    show("Token", Chunx.Chunker.Token.chunk(@text, tokenizer, chunk_size: 20))
+    show("Word", Chunx.Chunker.Word.chunk(@text, tokenizer, chunk_size: 20))
+    show("Sentence", Chunx.Chunker.Sentence.chunk(@text, tokenizer, chunk_size: 20))
+    show("Recursive", Chunx.Chunker.Recursive.chunk(@text, tokenizer, chunk_size: 20))
+  end
 
-    IO.puts("\n=== Recursive Chunking ===")
-    {:ok, recursive_chunks} = Chunx.Chunker.Recursive.chunk(text, tokenizer, chunk_size: 50)
+  defp show(name, {:ok, chunks}) do
+    IO.puts("\n#{name}")
 
-    Enum.each(Enum.with_index(recursive_chunks), fn {chunk, i} ->
-      IO.puts("Chunk #{i} (Tokens: #{chunk.token_count}):\n#{chunk.text}\n")
+    Enum.each(chunks, fn chunk ->
+      IO.puts("[#{chunk.start_byte}, #{chunk.end_byte}) (#{chunk.token_count} tokens)")
+      IO.puts(chunk.text)
     end)
   end
 end

lib/chunx.ex

@@ -1,12 +1,11 @@
 defmodule Chunx do
   @moduledoc """
-  Chunk text with different chunking strategies.
+  Text chunking strategies.
 
-  Available chunking strategies:
-  - `Chunx.Chunker.Token` - Splits text into overlapping chunks based on token count
-  - `Chunx.Chunker.Word` - Splits text into overlapping chunks based on word boundaries
-  - `Chunx.Chunker.Sentence` - Splits text into overlapping chunks based on sentence boundaries
-  - `Chunx.Chunker.Semantic` - Splits text into overlapping chunks based on semantic similarity
-  - `Chunx.Chunker.Recursive` - Recursively splits text using structural boundaries
+    * `Chunx.Chunker.Token` splits at token offsets.
+    * `Chunx.Chunker.Word` keeps words intact.
+    * `Chunx.Chunker.Sentence` keeps sentences intact and supports overlap.
+    * `Chunx.Chunker.Recursive` tries structural boundaries before tokens.
+    * `Chunx.Chunker.Semantic` splits where sentence similarity decreases.
   """
 end

lib/chunx/chunk.ex

@@ -22,6 +22,7 @@ defmodule Chunx.Chunk do
           embedding: Nx.Tensor.t() | nil
         }
 
+  @doc "Creates a chunk. Byte offsets use a half-open range."
   @spec new(String.t(), non_neg_integer(), non_neg_integer(), pos_integer(), Nx.Tensor.t() | nil) ::
           t()
   def new(text, start_byte, end_byte, token_count, embedding \\ nil)

lib/chunx/chunker.ex

@@ -1,6 +1,6 @@
 defmodule Chunx.Chunker do
   @moduledoc """
-  Defines the interface for text chunking strategies.
+  Defines the shared callback types for chunkers.
   """
 
   alias Chunx.{Chunk, SentenceChunk, Tokenizer}
@@ -9,9 +9,10 @@ defmodule Chunx.Chunker do
   @type chunk_result :: {:ok, [Chunk.t()] | [SentenceChunk.t()]} | {:error, term()}
 
   @doc """
-  Splits text into chunks using the given tokenizer. Semantic chunkers receive
-  an embedding function as the third argument and may receive options as a
-  fourth argument.
+  Splits text using a tokenizer.
+
+  Semantic chunkers take an embedding function instead of options as the third
+  argument and accept options as a fourth argument.
 
   ## Parameters
     * `text` - The text to chunk

lib/chunx/chunker/recursive.ex

@@ -1,6 +1,6 @@
 defmodule Chunx.Chunker.Recursive do
   @moduledoc """
-  Implements recursive text chunking from coarse document boundaries to tokens.
+  Splits text through an ordered sequence of structural boundaries.
 
   Text is split using each configured level in order. Segments that still exceed
   `:chunk_size` are passed to the next level, while adjacent segments are merged
@@ -41,20 +41,21 @@ defmodule Chunx.Chunker.Recursive do
     :tokens
   ]
 
-  @type level :: [String.t()] | :whitespace | :tokens
+  @type level :: nonempty_list(String.t()) | :whitespace | :tokens
   @type chunk_opts :: [
           chunk_size: pos_integer(),
-          levels: [level()]
+          levels: nonempty_list(level())
         ]
 
   @default_opts [chunk_size: 512, levels: @default_levels]
 
   @doc """
-  Recursively splits text into chunks no larger than `:chunk_size` tokens.
+  Recursively splits text toward the `:chunk_size` target.
 
   ## Options
 
-    * `:chunk_size` - Maximum number of content tokens per chunk (default: 512).
+    * `:chunk_size` - Target maximum content-token count (default: 512). An
+      indivisible grapheme may exceed the target.
     * `:levels` - Ordered splitting levels. Each level is a non-empty list of
       delimiters, `:whitespace`, or `:tokens`. Token splitting is always used as
       a final fallback when custom levels are exhausted.