rm useless functions

Author: Chaimaekit

.github/workflows/build.yml

@@ -4,8 +4,8 @@ on:
   push:
     branches:
       - build
-  # schedule:
-  #   - cron: '*/10 * * * *'
+  schedule:
+    - cron: '*/10 * * * *'
 
 jobs:
   deploy:
@@ -54,7 +54,7 @@ jobs:
         env:
           CLOUD_ID: ${{ secrets.CLOUD_ID }}
           ELASTIC_PASSWORD: ${{ secrets.ELASTIC_PASSWORD }}
-        run: python elastic_script.py
+        run: python elastic/elastic_script.py
 
       - name: Run FastAPI and test endpoint
         env:

README.md

@@ -1,53 +1,76 @@
 # Evently – Event Scraper + FastAPI + Elasticsearch + GitHub Actions Cron
 
-Evently is a project that scrapes event data, indexes it into Elasticsearch, and serves it through a FastAPI backend with a front-end UI.  
+Evently is a platform that scrapes event data from multiple Moroccan sources, enriches each event with nearby bus and tramway transport info, indexes everything into Elasticsearch, and serves it through a FastAPI backend with a frontend UI.  
 All scraping and indexing are automated using GitHub Actions on a scheduled cron job.
 
-
 ---
 
-## 📌 Features :
+## 📌 Features
 
 ### 🔍 Scraping
-- Collects events from multiple sources
+- Collects events in parallel from multiple sources (Casaevents, Eventbrite, Events.ma, Guichet)
+- Parallel scraping for faster data collection
+
+### 🚌 Transport Integration
+- Each event is automatically enriched with nearby **Casabus** and **Tramway** lines at indexing time
+- Uses a **KDTree spatial index** for near-instant transport lookups
+- Transport data is precomputed and stored directly in the event document — no per-request calculation
 
 ### 🧠 Elasticsearch
-- Events are indexed
-- Prevents duplicates using hashed unique IDs
+- Events are indexed with duplicate prevention using hashed unique IDs (MD5)
+- Full-text search with filters by city, category, and date
+- Aggregations for stats, filters, trending, and recommendations
 
 ### ⚡ FastAPI Backend
 
-Endpoints:
-- `/` – serves the frontend  
-- `/events` – paginated events  
-- `/indexing` – manually trigger indexing  
-- `/stream` – test server-sent event streaming  
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/` | Frontend homepage |
+| GET | `/events?page=X&size=Y` | Paginated events |
+| GET | `/event/{event_id}` | Single event detail |
+| GET | `/search` | Search with filters (city, category, date) |
+| GET | `/transport/{event_id}` | Bus and tramway lines near an event |
+| GET | `/filters` | Available cities and categories |
+| GET | `/stats` | Total events, upcoming, cities, categories |
+| GET | `/trending` | Trending upcoming events |
+| GET | `/recommendations` | Similar events by category and city |
+| GET | `/calendar` | Events grouped by date for a given month |
+| POST | `/favorites/{event_id}` | Add event to favorites |
+| DELETE | `/favorites/{event_id}` | Remove event from favorites |
+| GET | `/favorites` | Get favorite events |
+| POST | `/export` | Export events as JSON or CSV |
+| GET | `/reindex` | Manually trigger full reindex |
+| GET | `/health` | Elasticsearch health check |
 
 ### 🎨 Frontend
-- Dynamic event loading from API  
-- Pagination support  
+- Dynamic event loading from API
+- Pagination, search, and filter support
 
 ### 🔄 GitHub Actions Automation
-A scheduled workflow runs:
-- All scrapers  
-- Elasticsearch indexing script  
-- Based on a cron schedule  
+A scheduled workflow runs all scrapers and Elasticsearch indexing automatically on a cron schedule.
 
 ---
 
 ## 🗂️ Project Structure
+
 ```
 Events_Project/
 │
 ├── elastic/
-│   ├── __init__.py           # exports global 'es' instance
-│   └── elastic_client.py     # creates Elasticsearch client
+│   ├── elastic_script.py     # indexing + transport enrichment
+│   └── elastic_client.py     # Elasticsearch client setup
+│
+├── main.py                   # FastAPI routes + lifespan startup
+│
+├── static/
+│   ├── script.js
+│   └── style.css
+│
+├── templates/
+│   └── index.html            # Frontend UI
 │
-├── main.py                   # FastAPI routes
-├── elastic_script.py         # indexing script
-├── index.html                # Frontend UI
 ├── requirements.txt
-├── .env                      # Environment variables
+├── .env                      # Environment variables (not committed)
 │
 ├── .github/
 │   └── workflows/
@@ -56,88 +79,96 @@ Events_Project/
 ├── Dockerfile
 ├── docker-compose.yml
 │
-├── scrape/                   # All scrapers
+├── scrape/                   # Scrapers (run in parallel)
 │   ├── casaevents.py
 │   ├── eventbrit.py
-│   ├── events.ma
+│   ├── eventsma.py
 │   └── guichet.py
 │
 ├── struct_events/
-│   └── models.py             # Events model
-
+│   ├── get_coords.py         # Geocoding utility
+│   └── models.py             # Events Pydantic model
+│
+├── transport/
+│   ├── bus.py                # KDTree transport lookup
+│   ├── cleaned_lines/
+│   └── overpass/
 ```
 
 ---
 
-
-
 ## ⚙️ Installation & Setup
 
+### 1️⃣ Pull the Docker image OR Clone the repo
 
-### 1️⃣ Pull the Docker image
 ```bash
 docker pull chaimaaeljerrar/scraping-image:latest
 ```
-
-
-### 2️⃣ Create your .env file
-```
-ELASTIC_PASSWORD=(your elasic password)
-CLOUD_ID=(your elastic cloud id)
+OR
+```bash
+git clone <repo_url>
 ```
 
+### 2️⃣ (Optional) Create a `.env` file
+
+Only needed if connecting to Elastic Cloud or if security is enabled:
 
-### 3️⃣ Install dependencies
 ```
-docker run -d -p 8000:8000 --env-file .env chaimaaeljerrar/scraping-image:latest
+ELASTIC_PASSWORD=your_password_here
+CLOUD_ID=your_cloud_id_here
 ```
 
+> When running locally with Docker Compose, Elasticsearch security is disabled by default — no `.env` is required.
 
-### 4️⃣ Access the frontend
-```
-👉 http://localhost:8000
-🔌 Elasticsearch Connection
-```
+### 3️⃣ Build and run the container
 
+```bash
+docker compose up
+```
 
+The app will:
+1. Wait for Elasticsearch to be healthy
+2. Automatically start scraping and indexing events in the background
+3. Serve the frontend at `http://localhost:8000`
 
-### 🤖 GitHub Actions – Scheduled Scraping
-This automatically indexes new events every 10mins.
+### 4️⃣ Access the app
 
+```
+http://localhost:8000
+```
 
+---
 
-### 🖥️ Frontend – Usage
-The frontend fetches events
-Pagination is fully managed on the client side.
+## 🧪 Testing Locally
 
+```bash
+# Check the UI
+http://localhost:8000
 
+# Check events API
+http://localhost:8000/events
 
-### 🔧 API Endpoints
-| Method | Endpoint                | Description                       |
-|--------|--------------------------|-----------------------------------|
-| GET    | `/`                      | Frontend homepage                 |
-| GET    | `/events?page=X&size=Y`  | Paginated events                  |
-| GET    | `/indexing`              | Manually trigger event indexing   |
-| GET    | `/stream`                | Server-sent events (debug)        |
+# Check health
+http://localhost:8000/health
 
+# Manually trigger reindex
+http://localhost:8000/reindex
+```
 
+---
 
+## 🔧 How Transport Works
 
-### 🧪 Testing locally
-```
-Run FastAPI
-Call: http://localhost:8000/events
-If you see JSON, everything works.
-Use the frontend to browse events visually.
-```
+At indexing time, each event's coordinates are used to query a **KDTree** built from all bus and tramway stop coordinates. The nearest lines (within 0.5km) are stored directly on the event document in Elasticsearch.
 
+This means `/transport/{event_id}` is just a simple document read — no spatial computation happens at request time.
 
+---
 
+## 📝 Future Improvements
 
-### 📝 Future Improvements
-```
-1️⃣ Add categories & filters
-2️⃣ Add search by city / date
-3️⃣ Add infinite scrolling
-4️⃣ Add AI-based event deduplication
-```
+1. Add infinite scrolling to the frontend
+2. Store favorites in Elasticsearch instead of in-memory
+3. Add AI-based event deduplication
+4. Add user authentication
+5. Deploy to cloud with security enabled

elastic/__init__.py

@@ -1,14 +1,15 @@
-from elasticsearch import Elasticsearch, helpers
+from elasticsearch import helpers
 from into_db.connection import get_db_events
 from scrape.casaevents import get_casa_events
 from scrape.eventbrit import get_event_brit
 from scrape.eventsma import get_events_ma
 from scrape.guichet import get_guichet
 import hashlib
-from elastic.elastic_client import get_es_client
+from elastic_client import get_es_client
 import logging
 import os
 from concurrent.futures import ThreadPoolExecutor, as_completed
+from transport.bus import get_transport_info
 
 
 
@@ -21,7 +22,6 @@
 
 
 def normalize_coordinates(coords):
-    """Safely normalize coordinates to [lon, lat] list or return empty list."""
     if coords is None:
         return []
     if isinstance(coords, list):
@@ -35,7 +35,6 @@ def normalize_coordinates(coords):
 
 def indexing():
     try:
-        logger.info("starting the indexing process ------")
         logger.info("testing Elasticsearch connection--------")
         if not es.ping():
             logger.error("cant connect to Elasticsearch !!!")
@@ -77,6 +76,7 @@ def indexing():
             doc_id = hashlib.md5(
                 (str(event.get("name", "")) + str(event.get("date", "")) + str(event.get("city", ""))).encode()
             ).hexdigest()
+            transport = get_transport_info(normalize_coordinates(event.get("coordinates")))
 
             doc = {
                 "id": event.get("id", ""),
@@ -87,6 +87,8 @@ def indexing():
                 "city": event.get("city", ""),
                 "place": event.get("place", ""),
                 "coordinates": normalize_coordinates(event.get("coordinates")),
+                "bus_lines": transport.get("bus_lines", []),
+                "tramway_lines": transport.get("tramway_lines", []),
                 "location": event.get("city", "") + " " + event.get("place", ""),
                 "producer": event.get("producer", ""),
                 "category": event.get("category", []),