Add PostgreSQL backend with pgvector semantic search#16
Merged
Conversation
Implements database introspection-based comparison between file schema (app/schema.sql) and embedded fallback schema in app/server.py. The test validates that both schemas remain synchronized by comparing resulting SQLite structures via sqlite_master queries, ensuring semantic equivalence while being resilient to formatting differences.
Introduces protocol-based architecture enabling support for multiple database backends (SQLite, PostgreSQL, Supabase) while maintaining clean separation of concerns and type safety. Key improvements: - StorageBackend protocol defines database-agnostic interface with 7 required methods - SQLiteBackend implementation with production-grade features (connection pooling, retry logic, circuit breaker) - Backend factory for runtime selection via STORAGE_BACKEND environment variable - All repositories now use StorageBackend interface instead of concrete implementations - Complete type safety with mypy and pyright validation - Comprehensive test coverage maintained (392 tests passing) Technical details: - Protocol-based design allows easy addition of new backends without modifying repositories - Repository pattern preserved with enhanced abstraction layer - All existing functionality maintained with zero breaking changes - Documentation updated to reflect new architecture This enables future PostgreSQL and Supabase support without touching repository code.
Resolves issue where SQLiteBackend was initialized three times during server startup, causing ~100-200ms unnecessary overhead and potential resource leaks. Backend is now created once at lifespan start and shared across initialization functions. Changes: - Create backend once at beginning of lifespan() and pass to init functions - Add optional backend parameter to init_database() for shared backend usage - Add optional backend parameter to apply_semantic_search_migration() - Remove redundant _ensure_backend() call preventing potential 4th initialization - Maintain backward compatibility for tests calling init functions directly Performance: 67% reduction in initializations (3 → 1), ~100-200ms faster startup Impact: Prevents resource leaks, cleaner lifecycle management, improved type safety Tests: All pass with full backward compatibility maintained
Resolves FastMCP warning about duplicate tool registration and improves API naming consistency by renaming semantic_search_tool to semantic_search_context. Changes: - Add @mcp.tool() decorator to function definition (line 1340) - Remove duplicate manual registration via mcp.add_tool() - Rename semantic_search_tool to semantic_search_context for API consistency - Update all log messages to reference new tool name Impact: Eliminates server startup warning, achieves 100% API naming consistency across all 8 MCP tools.
Updates all documentation references from semantic_search_tool to semantic_search_context following the API naming consistency improvement. Changes: - Update README.md API reference - Update semantic-search.md usage guide, examples, and headings
Updates test file comments to reference semantic_search_context following the tool rename.
Fixes integration test deadlock by implementing persistent backend lifecycle in initialized_server fixture and updating test mocks to use repository layer. Changes: - Create persistent backend before yielding to tests in initialized_server fixture - Add timeout protection (5 seconds) to backend shutdown in cleanup - Update test mocks from backend mocking to repository method mocking - Fix hasattr checks for proper backend existence validation Impact: Integration tests complete in 3.28s (was hanging indefinitely), all 15 tests pass.
Fixes critical test suite hang caused by asyncio primitives being created in __init__() before event loop exists. Primitives now created in initialize() method ensuring correct event loop binding. Changes: - Move Semaphore, Queue, Lock, Event creation from __init__ to initialize() - Add Optional type hints for all asyncio primitives - Add type guard assertions for safe access throughout the class - Fix __del__() to safely access optional shutdown_complete event - Simplify test assertions in test_resource_warnings.py Impact: Full test suite completes in 50s (was hanging indefinitely at test_deduplication.py), all 409 tests collected and 392 pass without any hangs.
Implements missing integration tests achieving 100% MCP tool coverage (8/8 tools). Tests include proper conditional logic for optional semantic search feature. Changes: - Add test_update_context with 5 sub-tests for selective field updates - Add test_semantic_search_context with availability detection and graceful skip - Update run_all_tests method to include both new tests - Update module docstring to reflect 8 tools coverage Coverage: 8/8 MCP tools tested (100%), up from 6/8 Tests: 9/9 passed with proper conditional handling for optional features Conditional logic: semantic_search_context tested when available, skipped when disabled
Complete implementation enabling dual-database architecture (SQLite + PostgreSQL) with full feature parity including semantic search via pgvector. Backend Infrastructure: - PostgreSQLBackend: 545-line async implementation using asyncpg connection pooling - PostgreSQL schema: 107 lines with JSONB indexes, BYTEA storage, and pgvector extension - StorageBackend protocol enhanced: Union[Sync, Async] callable support for dual backends - Backend factory: Runtime selection via STORAGE_BACKEND environment variable Repository Layer (30 methods adapted): - All 5 repositories support both SQLite and PostgreSQL backends - SQL dialect abstraction via helper methods: _placeholder(), _placeholders(), _json_extract() - ContextRepository: Dual-path implementation for 9 core methods - TagRepository, ImageRepository, StatisticsRepository, EmbeddingRepository: Full dual-backend support - MetadataQueryBuilder: Backend-aware query generation with proper parameter binding Semantic Search Enhancement: - pgvector integration for PostgreSQL/Supabase backends - EmbeddingRepository: 7 methods supporting both sqlite-vec and pgvector - PostgreSQL migration: add_semantic_search_postgresql.sql with vector type and HNSW indexes - Dual-backend semantic search fully operational Testing Infrastructure: - Parametrized test fixtures: test_backend_all yields both SQLite and PostgreSQL - Backend comparison tests: 8 test methods × 2 backends = 16 tests - PostgreSQL test isolation: Environment-based configuration, no hardcoded credentials - ResourceWarning elimination: Session fixture uses TCP socket check, extensive async cleanup Type Safety and Quality: - mypy: 0 errors (eliminated 35 asyncpg warnings with asyncpg-stubs) - pyright: 0 errors, 0 warnings - Type stubs: asyncpg-stubs, custom pgvector stubs - Complete type hints with proper Union types for dual-backend callables Performance and Configuration: - PostgreSQL settings: 15 configuration fields (connection, pool, SSL, Supabase) - Environment variable support: POSTGRESQL_HOST, POSTGRESQL_USER, POSTGRESQL_PASSWORD, etc. - Connection pooling: asyncpg.Pool (min=2, max=20, configurable) - Circuit breaker and retry logic: Production-grade fault tolerance Tests: 408 passed, 17 skipped, 0 warnings (ResourceWarnings eliminated) Quality: All pre-commit hooks pass (Ruff, Mypy, Pyright) Backends: Both SQLite and PostgreSQL fully functional and production-ready
Fixes mypy import-not-found error in CI for pgvector.asyncpg module. The pgvector package is part of optional semantic-search dependencies, and mypy needs to ignore missing imports when it's not installed.
alex-feel
force-pushed
the
alex-feel-dev
branch
from
375ddea to
1f1600e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.