ScrapeMonkey

ScrapeMonkey - is a project designed to scrape product information from online food stores written in Go. Currently, it supports scraping products from Kosik.cz, but the plan is to add additional stores in the future.

Features

Efficient Scraping: Extracts product data such as names, prices, descriptions, and more.

Modular Design: Built with a flexible architecture to add new stores with minimal effort.

Getting Started

Prerequisites

Make sure you have the following installed:

• Go (version 1.23 or later)

• installed make (optional)

• Internet connection

Usage

There are predefined commands using makefile, you can use them to build and run the project. If you don't have make installed, you can run the commands manually.

Environment

Copy .env.example to .env which describes the variables that are needed. The server refuses to boot if .env is missing.

Console view

Build the project

tbd

Run the project

make con

Run the project in debug mode

make con-debug

API server

make server

Pass overrides via flags, e.g. make server ARGS="-port=9090 -log-level=debug".

Benchmark

Run the benchmark

make bench

View the benchmark results

make bench-stat

Clean the benchmark results

make bench-clean

Future Plans

• Add scraping support for more online stores.

• Implement advanced filtering and sorting for scraped data.

• Add more types of user interfaces

• In a distant future, add ai to choose products and analyze the best store to buy from based on the user's prompt.