Skip to content

Conversation

@pane2004
Copy link

@pane2004 pane2004 commented Dec 28, 2025

Adds an LLM-based pointwise reranker to the RAG pipeline, significantly improving retrieval quality.

Key Features

  • Pointwise scoring: Each passage is scored 1-10 based on relevance to the query using a detailed grading rubric
  • Parallel batch processing: Passages are distributed round-robin across N batches and scored concurrently using asyncio.gather(): https://fin.ai/research/using-llms-as-a-reranker-for-rag-a-practical-guide
  • Score thresholding: Passages scoring below 5 are automatically filtered out, reducing noise in results
  • Graceful fallback: On timeout or LLM failure, falls back to original embedding scores

Why Reranking Matters

The demo shows a dramatic improvement - vector search alone ranked the two most relevant passages at positions 13 and 14, while irrelevant passages with keyword overlap ranked 1-3.
After LLM reranking:

  • High-relevance passages → 1 and 2 with 10/10 scores
  • All irrelevant passages → filtered out completely

Files Changed

New files:

  • src/flare_ai_kit/rag/vector/reranker/ - New reranker module
    • base.py - Abstract BaseReranker class
    • gemini_reranker.py - GeminiPointwiseReranker implementation
    • prompts.py - Grading rubric and prompt templates
  • examples/08_rag_reranker_demo.py - Full RAG + reranker demo
  • tests/unit/rag/vector/reranker/ - 19 unit tests

Modified files:

  • src/flare_ai_kit/rag/vector/settings.py - Added RerankerSettings
  • src/flare_ai_kit/rag/vector/factory.py - Integrated reranker into VectorRAGPipeline
  • src/flare_ai_kit/rag/vector/api.py - Exported new classes
  • src/flare_ai_kit/common/exceptions.py - Added RerankerError, RerankerParseError

Test Plan

  • All 19 unit tests pass
  • Ruff formatting and linting pass
  • Pyright type checking passes
  • Manual testing with demo script shows correct reranking behavior

Closes #10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VectorRAG] Implement Re-ranking Step in RAG Pipeline

1 participant