|
1 | 1 | # IBEX-Scraper-API |
2 | 2 |
|
3 | | -A secure ASP.NET Core Web API that automatically scrapes **day-ahead prices from the Free Electricity Market (Day Ahead segment) on IBEX** and stores them in a structured Azure SQL database for use by external applications. |
| 3 | +A lightweight **ASP.NET Core Web API** that automatically scrapes **day-ahead electricity prices from IBEX (Independent Bulgarian Energy Exchange)** and stores them in a remote SQL Server database for use by external applications. |
| 4 | + |
| 5 | +**Live API:** [link](http://ibex-scraper.somee.com/swagger/index.html) *(HTTP only – free hosting plan limitation)* |
| 6 | + |
| 7 | +--- |
4 | 8 |
|
5 | | -Access it at [link](http://ibex-scraper.somee.com/swagger/index.html) |
6 | 9 | ## ⚡ Purpose |
7 | 10 |
|
8 | 11 | This API was built to **decouple IBEX access from dependent systems** by providing a secure, reliable, and filterable data backend. |
9 | 12 |
|
10 | 13 | It powers apps like [SunNext](https://github.com/JacksonJS12/SunNext) by allowing them to: |
11 | | -- Access daily price data without scraping IBEX directly |
12 | | -- Query historical electricity prices by date/hour |
13 | | -- Build custom dashboards and energy trading strategies |
| 14 | + |
| 15 | +* Access daily price data without scraping IBEX directly |
| 16 | +* Query historical electricity prices by date/hour |
| 17 | +* Build custom dashboards and energy trading strategies |
| 18 | + |
| 19 | +--- |
14 | 20 |
|
15 | 21 | ## 📊 What It Does |
16 | 22 |
|
17 | | -- Scrapes **IBEX Day Ahead segment** prices for the next day |
18 | | -- Extracts hourly energy prices (EUR/MWh) |
19 | | -- Saves results into an Azure SQL `MarketPrices` table with: |
20 | | - - `Date` (e.g. `day/month/year`) |
21 | | - - `Hour` (1–24 using Central European Time – CET Time Zone) |
22 | | - - `PricePerMWh` (BGN/MWh) |
| 23 | +* Scrapes **IBEX Day Ahead** prices for the next day |
| 24 | +* Extracts hourly energy prices (BGN/MWh) |
| 25 | +* Saves results into a `MarketPrices` table with: |
| 26 | + |
| 27 | + * `Date` (format: `dd/MM/yyyy`) |
| 28 | + * `Hour` (1–24 using EET/EEST timezone) |
| 29 | + * `PricePerMWh` (BGN/MWh) |
| 30 | + |
| 31 | +--- |
23 | 32 |
|
24 | 33 | ## 🕒 Scheduling |
25 | 34 |
|
26 | | -- Powered by an **Azure Logic App** that triggers daily at **14:00 BG time (UTC+3)** |
27 | | -- Calls the `POST /api/Scrape/scrape-and-save` endpoint |
28 | | -- Only stores new data (prevents duplicate entries for the same day) |
29 | | - |
30 | | - <center><img width="394" height="356" alt="image" src="https://github.com/user-attachments/assets/897ffe35-0acf-4eba-b825-cbbb05ec982d" /></center> |
| 35 | +* Prices are updated daily after **14:00 BG time (UTC+3)** when IBEX publishes the next day’s market prices |
| 36 | +* Data scraping is triggered by calling: |
| 37 | + |
| 38 | + ``` |
| 39 | + POST /api/Scrape/scrape-and-save |
| 40 | + ``` |
| 41 | +* Only stores **new** data (prevents duplicate entries for the same date/hour) |
| 42 | + |
| 43 | +--- |
31 | 44 |
|
32 | 45 | ## 🧱 Tech Stack |
33 | 46 |
|
34 | | -- **Backend:** ASP.NET Core Web API (.NET 8) |
35 | | -- **Scraper:** AngleSharp |
36 | | -- **Database:** Azure SQL |
37 | | -- **Scheduler:** Azure Logic Apps |
38 | | -- **Deployment:** Azure App Service |
39 | | -- **CI/CD:** GitHub Actions |
40 | | - |
41 | | -📁 Project Structure |
42 | | -IBEX-Scraper-API/ <br/> |
43 | | -├── Controllers/ <br/> |
44 | | -│ └── ScrapeController.cs <br/> |
45 | | -├── Services/ <br/> |
46 | | -│ └── IbexScraper/ <br/> |
47 | | -│ └── IbexScraperService.cs <br/> |
48 | | -│ └── IIbexScraperService.cs <br/> |
49 | | -├── Data/ <br/> |
50 | | -│ └── Migrations/ <br/> |
51 | | -│ └── Models/ <br/> |
52 | | -│ └── MarketPrice.cs <br/> |
53 | | -│ └── AppDbContext.cs <br/> |
54 | | -│ └── AppDbContextFactory.cs <br/> |
55 | | -└── Program.cs <br/> |
| 47 | +* **Backend:** ASP.NET Core Web API (.NET 8) |
| 48 | +* **Scraper:** AngleSharp |
| 49 | +* **Database:** MS SQL (Somee.com hosting) |
| 50 | +* **Deployment:** FTP to Somee.com (Free Hosting Plan, HTTP-only) |
| 51 | + |
| 52 | +--- |
| 53 | + |
| 54 | +## 🌐 Deployment Notes (Somee Free Plan) |
| 55 | + |
| 56 | +* Hosted on **Somee.com** free plan |
| 57 | +* **HTTP only** – no HTTPS due to free hosting limitations |
| 58 | +* API runs at: [http://ibex-scraper.somee.com](http://ibex-scraper.somee.com) |
| 59 | +* Connection string stored in `appsettings.Production.json` on the server |
| 60 | +* Deployment via FTP to `/www.ibex-scraper.somee.com` |
| 61 | + |
| 62 | +--- |
| 63 | + |
| 64 | +## 📁 Project Structure |
| 65 | + |
| 66 | +``` |
| 67 | +IBEX-Scraper-API/ |
| 68 | +├── Controllers/ |
| 69 | +│ └── ScrapeController.cs |
| 70 | +├── Services/ |
| 71 | +│ └── IbexScraper/ |
| 72 | +│ ├── IbexScraperService.cs |
| 73 | +│ └── IIbexScraperService.cs |
| 74 | +├── Data/ |
| 75 | +│ ├── Migrations/ |
| 76 | +│ ├── Models/ |
| 77 | +│ │ └── MarketPrice.cs |
| 78 | +│ ├── AppDbContext.cs |
| 79 | +│ └── AppDbContextFactory.cs |
| 80 | +└── Program.cs |
| 81 | +``` |
56 | 82 |
|
0 commit comments