End-to-end data platform that turns Telegram posts from Ethiopian medical, pharmaceutical, and cosmetics businesses into a queryable analytics warehouse — scraping, dbt modeling, YOLO-based image enrichment, a FastAPI service, and Dagster orchestration.
Telegram → Telethon scraper → JSON data lake → Postgres (raw)
→ dbt (staging → star schema) → FastAPI → consumers
Images → YOLOv8 detection → fct_image_detections
Orchestrated end-to-end by Dagster on a daily schedule.
| Layer | Tool |
|---|---|
| Extraction | Telethon (Telegram API) |
| Storage | PostgreSQL |
| Transformation | dbt (star schema, 20 tests) |
| Enrichment | YOLOv8-nano (Ultralytics) |
| API | FastAPI + Pydantic + SQLAlchemy |
| Orchestration | Dagster |
fct_messages (grain: one Telegram message) joined to dim_channels and dim_dates, with fct_image_detections as a satellite fact for image-enrichment results.
# 1. Database
createdb medical_warehouse
# 2. Data — live scrape (needs my.telegram.org credentials) or sample data
python3 src/scraper.py # live
python3 src/load_raw_to_postgres.py
# 3. Image enrichment
python3 src/yolo_detect.py
python3 src/load_detections_to_postgres.py
# 4. Transform
cd medical_warehouse && dbt run && dbt test && dbt docs generate
# 5. Serve
uvicorn api.main:app --reload # docs at /docs
# 6. Orchestrate
dagster dev -f pipeline.py # UI at :3000| Method | Path | Description |
|---|---|---|
| GET | /api/reports/top-products |
Most frequently mentioned terms across channels |
| GET | /api/channels/{channel_name}/activity |
Posting activity for a channel |
| GET | /api/search/messages |
Keyword search across message text |
| GET | /api/reports/visual-content |
Image-usage stats by channel |
src/ scraper, loaders, YOLO detection
medical_warehouse/ dbt project (staging + marts + tests)
api/ FastAPI service
notebooks/ analysis.ipynb — warehouse queries, charts, dbt test summary
orchestration/pipeline.py Dagster job definition