Skip to content

search.md

Latest commit

 

History

History
412 lines (304 loc) · 9.2 KB

search.md

File metadata and controls

412 lines (304 loc) · 9.2 KB
Raw

Search Command

The search command finds articles in ZIM archive files using either full-text keyword search or semantic vector similarity search. It supports searching within a specific ZIM file or across all indexed library files.

Overview

The search command provides two powerful search modes:

  1. Full-Text Search (default) - Uses SQLite FTS5 for keyword matching with ranked results
  2. Semantic Search (--semantic) - Uses Model2Vec embeddings to find conceptually similar articles

Usage

Search Specific ZIM File

zim search <file.zim or name> <query>

Search All Library Files

zim search <query>

Command Options

Flag Short Description
--semantic -s Use vector similarity search instead of full-text
--json Output results in JSON format
--non-interactive Disable interactive selection mode

Search Modes

Full-Text Search (Default)

Full-text search uses SQLite FTS5 for fast, ranked keyword matching:

  • Searches both article titles and content excerpts
  • Uses AND logic for multi-word queries (machine learning matches articles with both terms)
  • Ranks results by relevance (title matches weighted higher than content matches)
  • Automatically handles special characters (hyphens, apostrophes become spaces)
  • Falls back to bounded native title similarity and prefix lookup only when the selected archive has no SQLite index
zim search wikipedia_en "machine learning"

Output:

Searching (full-text): machine learning
======================
Searching SQLite index... found 47 results

Found 47 results:

 1. Machine learning
 2. Deep learning
 3. Supervised learning
 4. Unsupervised learning
 5. Reinforcement learning
...

Enter number to read (1-20), 'm' for more, or 'q' to quit:

Semantic Search (--semantic)

Semantic search uses vector embeddings to find conceptually similar articles:

  • Generates 512-dimensional embeddings using Model2Vec
  • Compares article vectors using cosine distance
  • Finds related concepts even without exact keyword matches
  • Requires a semantic index created with zim index
zim search --semantic wikipedia_en "artificial intelligence"

Output:

Searching (semantic): artificial intelligence
==========================
Searching with semantic similarity... found 42 results

Found 42 results:

 1. Artificial intelligence
 2. Machine learning
 3. Neural network
 4. Deep learning
 5. Cognitive science
...

Enter number to read (1-20), 'm' for more, or 'q' to quit:

When to Use Each Mode

Use Full-Text When... Use Semantic When...
Searching for specific keywords Exploring related concepts
You know the exact terms Keywords don't return good results
Need fast, precise results Want broader, conceptual matches
Title/phrase matching Discovering connected topics

Search Features

Query Processing

The search query is automatically processed to improve matching:

  • Hyphens and apostrophes become spaces (Spider-Man → Spider Man)
  • Special characters are removed (C++ → C)
  • Multiple words use AND logic (quantum physics requires both terms)
  • Case-insensitive matching

Article Filtering

Search automatically filters out non-article resources:

  • Images (.svg, .png, .jpg, .gif, .webp, .ico)
  • Stylesheets and scripts (.css, .js)
  • Fonts (.woff, .woff2, .ttf)
  • Media files (.mp3, .mp4, .ogg, .webm)
  • Data files (.json, .xml)

Interactive Mode

In interactive mode (default), you can:

  • Enter a number (1-20) to read that article
  • Type 'm' to show more results (next 20)
  • Type 'q' or press Enter to quit

Non-interactive mode disables prompts and displays only results:

zim search --non-interactive wikipedia "python"

JSON Output

For automation and scripting, use --json:

zim search --json wikipedia "climate change"

Output:

{
  "query": "climate change",
  "results": [
    {
      "Title": "Climate change",
      "URL": "Climate_change"
    },
    {
      "Title": "Global warming",
      "URL": "Global_warming"
    }
  ],
  "count": 2
}

Examples

Example 1: Basic Keyword Search

zim search wikipedia_en "photosynthesis"
Searching (full-text): photosynthesis
=========================
Searching SQLite index... found 12 results

Found 12 results:

 1. Photosynthesis
 2. Carbon fixation
 3. Chlorophyll
 4. Calvin cycle
 5. Photosystem
...

Example 2: Semantic Concept Search

zim search -s wikipedia_en "how plants make food"
Searching (semantic): how plants make food
===========================================
Searching with semantic similarity... found 18 results

Found 18 results:

 1. Photosynthesis
 2. Plant nutrition
 3. Chloroplast
 4. Carbon fixation
 5. Autotroph
...

Example 3: Multi-Word Query

zim search wikipedia "quantum mechanics"

Searches for articles containing both "quantum" AND "mechanics":

Searching (full-text): quantum mechanics
===============================
Searching SQLite index... found 34 results

Found 34 results:

 1. Quantum mechanics
 2. Introduction to quantum mechanics
 3. History of quantum mechanics
 4. Quantum field theory
 5. Mathematical formulation of quantum mechanics
...

Example 4: Search All Library Files

zim search "Dionysus"

Searches across all indexed ZIM files:

Searching ALL library files for: Dionysus
=======================================

Found 15 results across 2 files:

wikipedia_en_all_maxi_2024:
  [1] Dionysus
  [2] Dionysian
  [3] Maenads

wikipedia_mythology:
  [4] Dionysus
  [5] Bacchus
  [6] Thyrsus
...

Example 5: JSON Output for Scripting

zim search --json wikipedia "algorithm"
{
  "query": "algorithm",
  "results": [
    {"Title": "Algorithm", "URL": "Algorithm"},
    {"Title": "Algorithmic efficiency", "URL": "Algorithmic_efficiency"},
    {"Title": "Sorting algorithm", "URL": "Sorting_algorithm"}
  ],
  "count": 3
}

Example 6: Non-Interactive Mode

zim search --non-interactive wikipedia "rust"
Searching (full-text): rust
==================
Searching SQLite index... found 8 results

Found 8 results:

 1. Rust
 2. Rust (programming language)
 3. Iron oxide
 4. Corrosion
 5. Rusting
...

Showing first 20 results. Use interactive mode to select and read articles.

Configuration

Search behavior can be configured via ~/.zim.yaml:

search:
  max_results: 50          # Maximum results to return
  non_interactive: false   # Disable interactive mode
  json_output: false       # Output in JSON format

Environment variables:

export ZIM_SEARCH_MAX_RESULTS=100
export ZIM_SEARCH_NON_INTERACTIVE=true
export ZIM_SEARCH_JSON_OUTPUT=true

Prerequisites

Local Index

Search requires a local SQLite index:

zim index <file>

Search Index

Before searching, create an index:

# Full-text search index
zim index wikipedia_en

# Full-text only, if you want to skip embeddings
zim index --no-semantic wikipedia_en

Indexing status:

zim info wikipedia_en

Look for Search Index: in the output.

Error Handling

Common Errors

All-library search is indexed-only. SQLite corruption, permission, and query failures are returned directly rather than being hidden by a native scan.

Error: database not initialized

Solution: Create the local index with zim index <file> or check index.path.

No index found for this ZIM file.
Run 'zim index <file>' to create a search index.

Solution: Create a search index with zim index <file>

SQLite search failed: no index for ZIM file

Solution: Check index.path, filesystem permissions, and whether the index exists

❌ No results found for 'query'

Solution: Try different search terms, use semantic search, or verify the index exists

Troubleshooting

  1. No results found

    • Check if index exists: zim info <file>
    • Try semantic search: zim search --semantic <file> <query>
    • Use broader search terms

  2. Too many resource files

    • Search automatically filters non-articles
    • If issue persists, check your query has meaningful keywords

  3. Slow search performance

    • Put the SQLite index on fast local storage
    • Check that exact vector ranking exists for vector search
    • Reduce max_results in config

Performance

Indexing Performance

  • Full-text: ~1000 articles/sec
  • Semantic: ~345 articles/sec (includes embedding generation)
  • Parallel indexing: Uses all CPU cores by default

Search Performance

  • Full-text: Typically <100ms for 50 results
  • Semantic: Typically <200ms for 50 results (with exact vector ranking)
  • Database size: ~0.8x of ZIM file size

Index Storage

Full-text indexes add minimal overhead. Semantic indexes with embeddings add significant storage due to 512-dimensional vectors per article.

Related Commands

  • zim index - Create search indexes
  • zim read - Read a specific article
  • zim list - List all articles in a ZIM file
  • zim serve - Start web server for browsing
  • zim info - Display ZIM file and index information