mkm-watchdog

It automatically and regularly scrapes predefined cardmarket.com listing pages, and sends notifications to a configured Telegram bot when new listings are found, with basic data.
Whenever a new listing appears on an article page, you will be notified, on mobile and on desktop.

Quick start

• Make sure Golang is installed on your machine.
• Clone the repository
• Set up the cardmarket urls in the config.toml file (more details below)
• Set up your Telegram credentials in the .env file (more details below)
• Run make build in order to build the executable.
• Run make run to launch the program.

Config.toml

Search URLs

To add a new url to scrape, add the following lines into the config.toml file:

[[searches]]
url = "copied URL"

Add as much as you want:

[[searches]]
url = "https://www.cardmarket.com/en/Magic/Products/Singles/Alpha/Black-Lotus"

[[searches]]
url = "https://www.cardmarket.com/en/Pokemon/Products/Singles/Neo-Genesis/Pikachu-NG70"

Other parameters:

• delay: period, in seconds, between two scraping loops. Keep it reasonably high.

Important notes:

If an URLs returns a product with more than 50 article results, only the first 50 will be scraped since the scraper does not handle the click on the "Show more results" button. One possible workaround is to use URLs with active filters instead, such as languages or cards conditions filters.

Telegram and .env

• First, you need to create a Telegram account.
• Then, for the following steps, you need to download and use the desktop version.

You can create a new Telegram bot via this link.

• Send the /newbot command to BotFather, and follow the steps to create a new bot. Once the bot is created, you will receive a token.
• Set the TELEGRAM_TOKEN variable in the .env file with your token.

TELEGRAM_TOKEN="1222533313:AAFwNd_HsPtpxBy35vEaZoFzUUB74v5mBpW"

Then, you need to find your chat ID.

• Paste the following link in your browser. Replace <Telegram-token> with the Telegram token.

https://api.telegram.org/bot<Telegram-token>/getUpdates?offset=0

• Send a message to your bot in the Telegram application. The message text can be anything. Your chat history must include at least one message to get your chat ID.
• Refresh your browser.
• Identify the numerical chat ID by finding the id inside the chat JSON object. In the example below, the chat ID is 123456789.

{  
   "ok":true,
   "result":[  
      {  
         "update_id":987654321,
         "message":{  
            "message_id":2,
            "from":{  
               "id":123456789,
               "first_name":"Mushroom",
               "last_name":"Kap"
            },
            "chat":{  
               "id":123456789,
               "first_name":"Mushroom",
               "last_name":"Kap",
               "type":"private"
            },
            "date":1487183963,
            "text":"hi"
         }
      }
   ]
}

• Set the TELEGRAM_CHAT_ID variable in the .env file with this value.

TELEGRAM_CHAT_ID=123456789