Standardize VectorStore.search_vectors() output schema across all backend implementations #845

Summary

Different vector database backends return search results in inconsistent formats. The current VectorStore abstraction does not normalize these responses, forcing downstream consumers to implement backend-specific parsing logic.

Description

Each backend exposes different field names and scoring semantics.

Current inconsistencies include:

• Scores

  • Some backends return cosine similarity (higher is better).
  • Others return L2 distance (lower is better).

• Metadata

  • In-memory backend returns metadata.
  • Qdrant returns payload.
  • Milvus may omit metadata entirely.

• Vectors

  • Some backends return vector.
  • Others return values.
  • Some omit vectors unless explicitly requested.

As a result, downstream components (such as DecisionEmbeddingPipeline) must guess which fields exist and implement multiple fallback paths, increasing complexity and the risk of silent bugs.

Suggested Fix

Introduce a standardized search result schema for every backend.

Example:

@dataclass
class SearchResult:
    id: str
    score: float
    metadata: dict
    vector: Optional[np.ndarray]

(or an equivalent TypedDict).

Require every backend wrapper (FAISS, Qdrant, Milvus, Pinecone, etc.) to convert its native client response into this schema before returning results.

Additionally:

• Normalize score into a consistent similarity metric (for example, 0.0–1.0).
• Always expose metadata through the metadata field.
• Always expose vectors through the vector field (using None when unavailable).

Benefits

• Eliminates backend-specific parsing.
• Simplifies downstream pipelines.
• Reduces conditional logic throughout the codebase.
• Provides a stable API regardless of the underlying vector database.