Local Ingestion Improvements - Path Optimization & Duplicate Prevention

[fabrice-cekome]

This pull request introduces a new feature: ingest_local_file, designed for seamless ingestion of local documentation into the RAG (Retrieval-Augmented Generation) system — ideal for internal knowledge bases.

Key Features

Local file ingestion

• New ingest_local_file function:

• Supports both single files and entire directories
• Recursively handles subfolders
• Automatically extracts code examples
• Generates contextual summaries for improved relevance

Smart path handling

• Truncates absolute paths to avoid unnecessary duplication
• Example:

• /project1/Knowledges/data.md and /project2/Knowledges/data.md are treated as the same resource
• Enhances storage efficiency and search accuracy

Robust cleanup and consistency

• Old entries are fully removed before re-ingestion
• Prevents duplicates due to changes (e.g., chunk size, file version)
• Ensures consistent indexing across document versions

Typical Use Cases

• Ingesting local technical documentation
• Incremental updates of project knowledge bases
• Shared documentation between projects with no duplication
• Version control and tracking of documentation changes

Usage in my projects

This feature ensures a single source of truth in the RAG across all projects, even when working with isolated or evolving documentation.

Typical use:

• Ingesting .txt files from context7.com, downloaded manually to avoid API limits
• Ingesting AI-generated documentation and guides specific to the project, updated at each stage of progress

This enables a constantly enriched and up-to-date RAG aligned with real project evolution.

[coleam00]

Nice work @fabrice-cekome! This is outside of the scope for what I want to do for the MCP server right now, but I appreciate this implementation!