Skip to content

index.md

Latest commit

 

History

History
105 lines (66 loc) · 3.54 KB

index.md

File metadata and controls

105 lines (66 loc) · 3.54 KB
Raw

zim index

Build a local SQLite search index for a ZIM file.

Overview

The index command creates a searchable SQLite database from ZIM archive contents. It extracts articles, builds an FTS5 full-text index, and creates semantic search embeddings with Model2Vec unless --no-semantic is passed.

Search Architecture

  • Full-text search: SQLite FTS5 on title, excerpt, and indexed article text
  • Semantic search: Model2Vec embeddings stored as compact vectors in SQLite
  • Vector search: Exact Go-side cosine ranking with bounded top-K memory
  • Parallel indexing: Uses all CPU cores by default

Usage

zim index [--force] [--no-semantic] <zim-file>

Options

Flag Description
--force Rebuild existing index without confirmation
--no-semantic Build full-text index only and skip ONNX embedding initialization

Prerequisites

No database service is required. The default index is a SQLite file stored in the ZIM library directory.

Semantic indexing uses the bundled Model2Vec/ONNX embedding path. Use --no-semantic when you only need keyword search or want the fastest first index.

Performance

Indexing Speed

  • Full-text only: fastest path; use --no-semantic
  • With embeddings: depends on ONNX/Model2Vec throughput
  • Parallel scaling: Near-linear with CPU cores

Database Size

Index size depends on indexed text and whether semantic vectors are stored.

Memory Usage

  • Embedding generation: Model2Vec ONNX model memory
  • Archive enumeration: streams the C namespace before A into a bounded worker queue instead of retaining directory-entry slices
  • SQLite indexing: local file writes, no database daemon
  • Recommended: 4GB+ RAM for smooth indexing

Examples

Index Wikipedia for full-text search

zim index simple-wiki

Index full-text only

zim index --no-semantic stackoverflow

Rebuild an existing index

zim index --force simple-wiki

Output

Indexing progress shows:

Indexing: 123456 / 500000 articles (24%) | 345 articles/sec

Upon completion, the index is immediately usable with zim search.

SIGINT and SIGTERM cancel scanning, embedding, and pending writes. A rebuild removes the old generation before replacement begins, and a canceled or failed run removes its partial replacement before the command exits.

Popularity rankings

Import a rankings TSV only after indexing the archive:

zim popularity simple-wiki ./rankings.tsv

Both the archive and rankings file are required. The TSV uses one underscored article title per line; the line number becomes its rank.

The importer trims each line, changes spaces to underscores, and ignores empty keys. Duplicate keys keep their smallest line number. A key matches either an article URL or its title with spaces changed to underscores. If both keys match one article, the article receives the smallest rank. The reported match count is the number of distinct ranking keys that matched at least one article.

The import streams bounded 250-row batches into temporary SQLite tables, then replaces all popularity ranks for the selected archive in one transaction. Parse, cancellation, or SQL errors roll back the update, so existing ranks remain intact.

See Also