Sentibot - Daily Financial News Summariser

Sentibot is a daily financial news intelligence bot that runs locally on a Windows laptop. It ingests articles from a set of financial RSS feeds, processes them through a multi-stage NLP pipeline using open-source models, and delivers a structured daily briefing to a private Discord channel as formatted markdown messages. A separate observability channel receives the run log and a system benchmark chart.

• Disclaimer: This project is for informational and educational purposes only and does not constitute financial advice. Data may be delayed, incomplete, inaccurate, or generated by an AI model.

---

Flow Diagram

---

Infrastructure and Model Decisions

Sentibot is designed to run on consumer hardware. The development machine is a laptop with 4 GB VRAM (NVIDIA RTX 3050). This constraint shaped every model and architecture decision in the project, since the majority of large models can't fit in 4 GB of VRAM and still leave headroom for the attention matrices of a long prompt, so the pipeline uses several small, specialised models rather than one large general-purpose one.

PyTorch is configured to use CUDA so all GPU-bound models run on the laptop's GPU rather than CPU, which gives a significant throughput improvement for batched inference tasks like sentiment classification.

Models Used

Model	Task	Device	Why
ProsusAI/finbert	Sentiment classification	GPU	Fine-tuned on financial text; far more accurate than general-purpose sentiment models on headlines. Runs efficiently in batched GPU inference across many titles at once
all-MiniLM-L6-v2	Title embeddings (deduplication + clustering)	CPU	Tiny (~90 MB), fast, produces high-quality 384-dimensional sentence embeddings. No VRAM cost, runs in parallel with GPU work.
cross-encoder/nli-MiniLM2-L6-H768	Zero-shot topic classification	CPU	Small cross-encoder that handles natural language inference, used to score headlines against topic labels without any fine-tuning.
Qwen/Qwen2.5-1.5B-Instruct	Fact extraction, summarisation, cluster naming, ticker extraction	GPU (4-bit INT4)	Smallest instruction-tuned model that produces coherent structured output. Loaded in 4-bit NF4 quantisation via bitsandbytes, which fits comfortably within the 3 GB VRAM budget.

---

Pipeline

Step 1 - Ingest

feedparser fetches every configured RSS feed. For each entry, newspaper3k downloads the full article body. If the page is paywalled or JavaScript-rendered and returns less than 100 characters of body text, the RSS summary snippet is used as a fallback. Each article is standardised into a dict with title, text, url, feed, and char_count.

Step 2 - Deduplicate

All article titles are embedded in one batch using all-MiniLM-L6-v2. Pairwise cosine similarity is computed and any article with similarity ≥ 0.85 to an earlier article is discarded. This threshold catches reworded versions of the same story across different feeds without being so aggressive that genuinely different articles are dropped. This step reduces unnecessary LLM inference calls downstream.

Step 3 - Filter

Titles are scored by the MiniLM zero-shot classifier against ten candidate labels: the five relevant topics (finance, business, geopolitics, technology, economics) and five irrelevant ones (sports, entertainment, lifestyle, health, weather). Rather than requiring any single label to clear a high threshold, which fails for headlines that split probability across several relevant labels simultaneously, the scores across all relevant labels are summed into a relevant mass score. Articles whose combined relevant mass is below MIN_RELEVANT_MASS = 0.35 are discarded.

This stage runs entirely on CPU before any GPU work begins, which is a deliberate ordering decision, as cheap filtering reduces the number of expensive Qwen inference calls needed.

Step 4 - Group by Topic

Articles are clustered into topic groups using AgglomerativeClustering with cosine distance on the title embeddings computed in Step 2. The number of clusters is determined dynamically using the heuristic min(max_groups, sqrt(n / 2)), capped at 7.

Agglomerative clustering was chosen over k-means for two reasons. First, it uses cosine distance natively, which is the correct metric for embedding space. Second, it is a hierarchical algorithm that merges the two most similar clusters at each step, which naturally produces tighter, more coherent topic groups than k-means, which partitions by centroid proximity and can produce uneven clusters with outliers.

Each cluster is named by passing a sample of its titles to Qwen, which returns a descriptive headline. The headline is then re-classified by the topic classifier - clusters whose generated name does not score above the relevance threshold on any relevant label are discarded before any further GPU processing. This second relevance gate stops off-topic clusters (lifestyle, sports, etc. that survived Step 3 as a group) from consuming LLM inference time.

Step 5 - Summarise Each Group

For each article in each group, a single Qwen call extracts both facts and tickers simultaneously:

FACTS:
- key fact one
- key fact two
TICKERS: AAPL, MSFT

Combining these into one call halves the number of inference steps compared to running separate fact-extraction and ticker-extraction calls. Tickers accumulate into a global deduplicated set across all groups.

Group summaries use map-reduce summarisation:

• Map - articles are split into chunks of 5. Each chunk's extracted facts are passed to Qwen for a 3–4 sentence summary. Using facts rather than raw article text keeps each chunk prompt small and signal-dense.
• Reduce - if the group has more than one chunk, the chunk summaries are passed to Qwen for a final 4–5 sentence synthesis.

This approach removes the need for a hard context-window cap and handles arbitrarily large groups without CUDA OOM errors.

Group-level sentiment is computed by running FinBERT in a single batched GPU call across all titles in the group. Individual scores are mapped to +1 (positive), 0 (neutral), −1 (negative) and averaged. FinBERT runs on titles rather than full article text because it has a hard 512-token limit and titles are the most signal-dense part of a financial news article.

Step 6 - Stock Metrics

The global ticker set is deduplicated and each ticker is validated against yfinance - any ticker that returns no price data is discarded as a hallucination. For valid tickers, yfinance fetches current price, day change %, P/E ratio, 52-week high/low, analyst mean target price, and sector.

Step 7 - Fear & Greed Index

A single HTTP GET to production.dataviz.cnn.io/index/fearandgreed/graphdata. Full browser-like headers are required - CNN's CDN returns a 418 error to requests that lack a realistic User-Agent and Referer.

Step 8 - Build Report

The report is formatted as Discord markdown and split into chunks of at most 2000 characters. Sections are kept intact where possible; oversized sections are split on line boundaries, never mid-sentence. Structure:

1. Header - date and Fear & Greed index
2. Ingestion summary - per-feed counts and deduplicated articles removed
3. One section per topic group - sentiment, article count, summary, tickers
4. Market Outlook - per-ticker price, fundamentals, and sentiment

Step 9 - Deliver

• Report markdown chunks → DISCORD_SUMMARY_CHANNEL_ID
• Run log + benchmark PDF → DISCORD_OBSERVABILITY_CHANNEL_ID

The benchmark PDF is saved locally to reports/ and contains four subplots (CPU %, RAM GB, GPU %, VRAM GB) over time with vertical dashed lines marking each pipeline stage boundary.

---

Project Structure

sentibot/
├── sentibot.py          # main pipeline class
├── system_monitor.py    # background resource polling and benchmark chart
├── config.py            # paths, constants, logger setup, rss feeds
├── utils.py         # llm prompt builder functions
├── requirements.txt
├── run.bat              # windows cmd entry point (used by task scheduler)
├── reports/             # daily json snapshots and benchmark pdfs
├── daily-execs/         # timestamped run logs from run.sh / run.bat
└── media/
    ├── sentibot-flow-diagram.jpg
    └── summarise-group-flow-diagram.jpg

---

Setup

1. Prerequisites

• Python 3.11 or 3.12 (3.13 has compatibility issues with some CUDA packages)
• NVIDIA GPU with CUDA 12.6 drivers installed
• Git for Windows (if running on Windows)

2. Clone and create virtual environment

git clone https://github.com/MattiaDiProfio/Sentibot.git
cd sentibot
python -m venv venv

3. Install dependencies

Install PyTorch with CUDA support first, then the rest:

# windows (cmd or powershell)
venv\Scripts\activate
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt

# mac / linux
source venv/bin/activate
pip install torch torchvision torchaudio
pip install -r requirements.txt

4. Discord setup

1. Go to discord.com/developers and create a new application.
2. Under Bot, enable Message Content Intent and copy the bot token.
3. Under OAuth2 → URL Generator, select bot scope and Send Messages + Attach Files permissions. Open the generated URL to invite the bot to your server.
4. Create two private channels in your server - one for the daily report, one for observability. Right-click each channel → Copy Channel ID (requires Developer Mode enabled in Discord settings).

5. Environment variables

Create a .env file in the project root:

DISCORD_TOKEN=your_bot_token_here
DISCORD_SUMMARY_CHANNEL_ID=your_summary_channel_id
DISCORD_OBSERVABILITY_CHANNEL_ID=your_observability_channel_id

6. Models

Models are downloaded automatically from Hugging Face on first run. Ensure you have ~4 GB of free disk space. If you're behind a proxy or on a restricted network, set HF_HOME to a writable directory before running.

---

Running

Manually

run.bat

Automatically on Windows - Task Scheduler

1. Open Task Scheduler → Create Basic Task
2. Set the trigger to Daily at your preferred time (e.g. 07:00)
3. Set the action to Start a Program:
   • Program: cmd.exe
   • Arguments: /c run.bat
   • Start in: <path to your repo>
4. Under Conditions, uncheck "Start only if the computer is on AC power"
5. Under Settings, uncheck "Stop the task if it runs longer than 3 days" (the pipeline takes ~5-15 minutes depending on article volume)

---

Configuration

All tuneable constants live in config.py:

Constant	Default	Description
MIN_RELEVANT_MASS	0.35	Minimum summed classifier score across relevant topic labels for an article to pass the filter
MIN_TOPIC_SCORE	0.35	Minimum top-label score for a cluster name to pass the post-cluster relevance check
DISCORD_MAX_LEN	2000	Maximum characters per Discord message
RSS_FEEDS	see config	Dict of feed name → URL
RELEVANT_TOPICS	finance, business, geopolitics, technology, economics	Set of topic labels considered relevant