[FEATURE REQUEST] Migrate RAG Implementation to Datapizza-AI

[merendamattia]

Description

Refactor the current RAG (Retrieval-Augmented Generation) implementation to use Datapizza-AI's native RAG capabilities instead of the current custom implementation. This will provide better integration with the existing agent framework and ensure consistency across the entire project.

Motivation

• Consistency: The project already uses Datapizza-AI agents (ChatbotAgent, FinancialAdvisorAgent) throughout the codebase. Migrating RAG to Datapizza-AI ensures a unified technology stack
• Maintainability: Using a single framework reduces complexity and makes the codebase easier to maintain and understand
• Feature Parity: Datapizza-AI's RAG likely includes features and optimizations that our custom implementation may lack
• Reduced Technical Debt: Eliminates the need to maintain a custom RAG solution alongside the framework
• Better Integration: Datapizza-AI's RAG should work seamlessly with the agent's memory, state management, and other features
• Performance: Framework-native implementations are typically optimized and battle-tested

Suggested implementation

1. Evaluate Datapizza-AI RAG Capabilities

   • Review Datapizza-AI documentation for RAG features
   • Test RAG query building and retrieval functionality
   • Compare performance with current custom implementation

2. Migrate Asset Retrieval

   • Replace src/retrieval/asset_retriever.py logic with Datapizza-AI RAG methods
   • Update the RAGAssetRetriever class to use Datapizza-AI's retrieval API
   • Test with existing ETF dataset in dataset/ETFs/

3. Integrate with Financial Advisor Agent

   • Update FinancialAdvisorAgent to use Datapizza-AI's RAG for portfolio generation
   • Ensure RAG queries are built and executed through the framework
   • Maintain current functionality while using framework methods

4. Update RAG Query Building

   • Migrate rag_query_builder_prompt usage to Datapizza-AI's RAG query methods
   • Ensure prompt templates work with the framework's RAG implementation

5. Testing & Validation

   • Verify portfolio generation still works correctly
   • Test RAG retrieval accuracy and relevance
   • Ensure performance is comparable or improved
   • Update unit tests to reflect new implementation

Technical Details

• Current RAG implementation: src/retrieval/asset_retriever.py
• Usage points: src/core/financial_advisor.py (lines with _rag_retriever)
• Dataset: dataset/ETFs/ (bond/ and stocks/)
• Integration point: FinancialAdvisorAgent.generate_balanced_portfolio()

Further content

• Document any API differences between custom and Datapizza-AI RAG implementations
• Create migration guide for developers working on RAG-related features
• Consider deprecating and removing src/retrieval/ directory once migration is complete
• Update project documentation and README to reflect the unified approach
• Consider adding Datapizza-AI RAG configuration to .env and environment setup docs
• May provide performance metrics comparison before/after migration