Chunx

Chunx splits text by tokens, words, sentences, document structure, or semantic similarity. It is an Elixir implementation inspired by Chonkie.

Installation

Add Chunx to mix.exs:

def deps do
  [
    {:chunx, "~> 0.2.0"}
  ]
end

Usage

All chunkers require a tokenizer. They accept a Tokenizers.Tokenizer or a custom adapter implementing the Chunx.Tokenizer behaviour.

alias Chunx.Chunker.Token

{:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
{:ok, chunks} = Token.chunk("Text to split", tokenizer, chunk_size: 128)

Each returned chunk contains its text, half-open byte offsets into the original text, and its content-token count. Sentence and Semantic return Chunx.SentenceChunk structs; the other chunkers return Chunx.Chunk structs.

Chunkers

Module	Splitting unit	Overlap
Chunx.Chunker.Token	Token offsets	Token count or fraction
Chunx.Chunker.Word	Whole words	Token count or fraction
Chunx.Chunker.Sentence	Whole sentences	Whole sentences within a token budget
Chunx.Chunker.Recursive	Configured structural levels, then tokens	None
Chunx.Chunker.Semantic	Sentence-embedding similarity	None

See the API documentation for each module's options and size-limit exceptions.

Semantic chunking also requires a function that returns one Nx.Tensor for each input string:

alias Chunx.Chunker.Semantic

embedding_fun = &MyApp.Embeddings.embed/1

{:ok, chunks} =
  Semantic.chunk("Text to split", tokenizer, embedding_fun,
    chunk_size: 128,
    threshold: :auto
  )

The repository also includes a non-semantic example.

Testing

Run the regular suite:

mix test

Embedding integration tests use sentence-transformers/all-MiniLM-L6-v2. They download and run the model, so they are excluded by default:

mix test --only integration

Use mix test --include integration to run both suites together.

License

MIT