🧠 NewsMate: A RAG-Based News Chatbot

📌 Objective

NewsMate is an intelligent chatbot that provides real-time, conversational answers based on the latest news articles. It leverages Retrieval-Augmented Generation (RAG) to ground LLM responses in fresh, factual news content sourced from RSS feeds.

🛠️ Tech Stack

• Frontend: React.js with Vite
• Backend: Node.js with Express
• Embeddings: @xenova/transformers (MiniLM)
• Vector Store: Qdrant (Cloud-hosted)
• Database/Cache: Redis (Session management)
• Language Model: Gemini API
• Scheduler: Custom Node.js cron-like job

🔍 Key Features & Architecture

• RSS Feed Ingestion

  • Regularly fetches articles from multiple RSS sources (e.g., NYTimes).
  • Extracts title, link, and content.

• Embedding Generation

  • Uses @xenova/transformers to create dense vector representations of article content.
  • Ensures semantic similarity can be measured during retrieval.

• Deduplication

  • Each article’s link is hashed using SHA-256 and formatted into a UUID-like string.
  • This unique ID prevents duplicate insertion into the vector store.

• Vector Storage (Qdrant)

  • Embeddings are upserted into the news_articles collection.
  • Qdrant is queried later for semantically similar chunks during a chatbot session.

• Chatbot (RAG)

  • User question triggers a semantic search in Qdrant.
  • Top-k relevant context chunks are injected into the Gemini API prompt.
  • The model generates context-grounded answers.

• Session Management

  • Redis is used to track ongoing conversations and maintain continuity.

⚠️ Difficulties Faced & Resolutions

Problem	Cause	Resolution
"Bad Request: Invalid ID" from Qdrant	Direct use of URLs as point IDs (invalid format)	Introduced hashing (SHA-256) of links into UUID-like strings
Duplicate Data Despite No Uploads	Same link repeatedly inserted without proper deduplication logic	Added pre-check using qdrant.retrieve() to skip already embedded articles
ReferenceError: stats is not defined	Misuse of Qdrant collection metadata in ensureCollectionExists	Removed incorrect stats reference and replaced with proper collection existence check
Embedding Failures	Some articles had malformed or insufficient content	Added validation and fallback logic per article to skip bad entries
Silent Failures / Poor Logging	Errors weren't specific or granular	Improved error logs and debug messaging for each critical operation (embedding, upsert, fetch)

✅ Outcomes

• ✅ Successfully built a full-stack news chatbot using RAG.
• ✅ Embedded 50+ articles into Qdrant and enabled semantic search.
• ✅ Integrated Gemini for high-quality LLM responses.
• ✅ Achieved auto-updating context through periodic embedding refreshes.

📘 Future Improvements

• Add user authentication and persistent chat history.
• Integrate summarization and source citation.
• Scale to handle multi-lingual feeds.