DocIngest

DocIngest is the open-source engine for turning documentation sites into searchable, MCP-accessible context for humans and coding agents.

It crawls docs, stores them as clean markdown, indexes them for search, and exposes the same corpus through a web UI, CLI, and MCP server. Use it to build a public docs index, self-host an internal corpus, or give coding agents fresher documentation context.

Quick Start • MCP + CLI • Screenshots • Setup Docs • Contributing

Status

What works today

• Index documentation sites from the web UI
• Browse and search indexed docs at docingest.com
• Open docs by domain, copy markdown, and download stored docs
• Re-index sources when upstream docs change
• Query docs from MCP-compatible coding tools
• Use the package as a lightweight CLI for quick lookup

Hosted corpus

• The live main deployment currently serves 1,635 latest documentation sites on docingest.com as of August 16, 2026
• DocIngest stores versioned snapshots per domain, so one docs site can have multiple historical versions behind the scenes
• The Git repository does not commit the full hosted corpus; the deployed service holds the actual indexed docs data

Known limitations

• Search and ranking work, but need deeper tuning
• Loading, empty, and success states need more polish
• Version-aware storage exists, but the product UX around versions is still early
• Not yet a mature enterprise docs platform with permissions, collaboration, and admin workflows

Screenshots

Homepage

Index a docs site

MCP setup guide

Quick Start

Prerequisites

• Node.js 18+ or Bun
• Firecrawl, hosted or self-hosted
• Redis for fast autocomplete/search

Redis is optional for tiny local tests, but recommended for anything serious.

Install

git clone https://github.com/Amal-David/docingest.git
cd docingest
npm install
cd server && npm install && cd ..

Configure

Copy .env.example to .env in the repo root and fill it in. Firecrawl is the only crawl backend, so set either FIRECRAWL_API_KEY (hosted) or FIRECRAWL_API_URL (self-hosted):

FIRECRAWL_API_KEY=fc-your-api-key-here
FIRECRAWL_API_URL=https://api.firecrawl.dev/v1
REACT_APP_API_URL=http://localhost:8001/api
REDIS_HOST=localhost
REDIS_PORT=6380

For local Docker with self-hosted Firecrawl:

FIRECRAWL_API_URL=http://localhost:3002/v1
REACT_APP_API_URL=http://localhost:8001/api
REDIS_HOST=localhost
REDIS_PORT=6380

For setup details, use these guides:

• Docker run modes
• Firecrawl setup
• Redis setup

Run

Choose the local services you want:

Run everything local:

docker compose --profile firecrawl --profile tools up -d

Run only Redis:

docker compose up -d redis

Run Redis and Firecrawl without the Redis UI:

docker compose --profile firecrawl up -d

Run Redis with the Redis UI:

docker compose --profile tools up -d

Run the app locally:

npm run dev

If port 8001 is already busy, use the alternate local API port:

npm run dev:local

Then open http://localhost:8000.

After indexing docs, build the Redis search index:

cd server
npm run build-index

MCP + CLI

Add DocIngest to Claude Code:

claude mcp add docingest -- npx -y @docingest/mcp-server

Use the same package as a CLI:

npx @docingest/mcp-server find react
npx @docingest/mcp-server read react.dev --topic hooks --max-tokens 5000
npx @docingest/mcp-server read react.dev --snapshot-id <immutable-snapshot-id>
npx @docingest/mcp-server search "server components" --limit 5

MCP tools:

• find-docs finds a library or docs domain
• read-docs fetches focused documentation content
• query-docs searches across indexed docs

For editor-specific config, see the MCP server README.

Corpus integrity upgrade

Existing V1/V2 records are preserved as evidence and are not silently promoted to public snapshot retrieval. Before deploying this release, run the backup-first audit, inspect its dry-run summary, then apply it deliberately:

npm run audit:integrity
npm run audit:integrity -- --apply

Snapshot quality backfill

Before the fix in #29, a snapshot was only approved when every accepted page cleared automatic approval, so a single short page could hold back an entire corpus and leave a stored domain answering "Documentation not found". The fix applies to new crawls; snapshots already on disk keep the verdict they were written with.

This one-time pass re-derives those verdicts from the markdown each snapshot was built from, so stored documentation is not stranded until someone re-crawls it. It only touches snapshots recorded as one-or-more-pages-need-review, the reason the old rule wrote — snapshots quarantined for challenge pages, error pages, or genuinely thin content are left alone. Dry-run is the default and the existing metadata is backed up before any write:

npm run backfill:quality
npm run backfill:quality -- --apply

An apply run stops on any domain that already has a backup from an earlier run, so a second pass cannot quietly overwrite the original boundary. The rest of the batch still proceeds. Review that backup, then re-run with --force to continue; the existing backup file is kept as it is.

npm run backfill:quality -- --apply --force

Setup Docs

Use these when you need more than the happy path:

• Redis setup for local/self-hosted Redis, indexing, and verification
• Firecrawl setup for hosted or self-hosted crawling
• Docker run modes for all-in-one or partial local services
• Nginx setup for production reverse proxy configuration
• Performance notes for speedups and next optimization work
• Reference for storage, API, deployment shape, and repo details

Tech Stack

• React + TypeScript + Tailwind CSS
• Node.js + Express + TypeScript
• Firecrawl for crawling
• Redis for autocomplete, full-text search, and cached docs
• File-based markdown storage

Contributing

Contributions are welcome, especially around crawling quality, search/ranking, MCP ergonomics, docs UX, and self-hosting.

• Issues
• Discussions

License

MIT