Improved Document Indexing/ Search#149
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Improves resilience of the documentation indexing vector store by handling ChromaDB “panic” / corruption scenarios via health checks, backup creation, and restore logic.
Changes:
- Add ChromaDB initialization “health probe” that wipes and recreates the DB on failure.
- Add pre-write filesystem backup of the ChromaDB directory and restore helper used on write failures.
- Add tests covering corrupt DB recovery,
KeyboardInterruptpassthrough,_backup_path, and restore behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/holoviz_mcp/holoviz_mcp/data.py |
Adds ChromaDB health-check init, backup/restore helpers, and write-failure recovery path during indexing. |
tests/docs_mcp/test_data.py |
Adds integration-style tests for the new ChromaDB recovery and backup/restore behavior. |
…/full)
Implement document chunking for large markdown files and extend the
search() content parameter to support string modes ('chunk', 'truncated',
'full') alongside the existing boolean values for backward compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add extract_pascal_terms() with stopword filtering to capture single PascalCase component names (Scatter, Button, Tabulator) that were previously missed by extract_tech_terms(). Integrate a metadata boost query on source_path_stem into the search pipeline, creating a 3-tier merge: metadata boost > keyword pre-filter > semantic similarity. This fixes search failures for queries like "HoloViews Scatter" where project-name filtering removed all tech terms, leaving only semantic search which ranked reference guides too low. Also fixes pre-existing bug in test_get_reference_guide_multiple_projects (panel_material_ui → panel-material-ui) and updates todo.md with progress. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Major overhaul of the HoloViz MCP server
Documentation Indexing/ Search (
holoviz_mcp/data.py)Complete rewrite of the documentation indexing pipeline:
chunk(best-matching snippet),truncated(smart-truncated around query keywords), andfull(complete document) — lets callers trade off speed vs completenessTextInputorFlexBoxThreadPoolExecutor+asyncio.gather()for concurrent repo clone + document extraction across all repositories (thread-localMarkdownExporterfor nbconvert safety)index_hashes.json) enables skipping unchanged files before expensive content extraction. Re-runningupdate indexwith no changes completes in ~6s (down from ~54s for a single project, ~6min for all)holoviz-mcp update index --project panelonly processes the specified repo(s);--fullforces a complete rebuild ignoring cached hashes