Fix MobyGames scraper

Author: Gemba

docs/CHANGELOG.md

@@ -22,6 +22,8 @@ humans](https://keepachangelog.com).
   is now optional. It gets added automagically when no other query keyword is
   applied.
 - Updated: `mamemap.csv` for MAME 0.275
+- Fixed: (#150) Enable MobyGames scraper usage with personal API key. Thanks,
+  @risalt and to the constructive support at MobyGames.
 - Fixed: (#122) Updated ZX-Spectrum scraper module. It uses now the site
   `zxinfo.dk` and a web API. The API provides at least the same information as
   the defunct search on `worldofspectrum.org`. Check out the added `--query=`

docs/SCRAPINGMODULES.md

@@ -144,10 +144,9 @@ The database also supports many non-Amiga platforms, but there's no doubt that A
 - Shortname: _`mobygames`_
 - Type: _Online_
 - Website: _[www.mobygames.com](https://www.mobygames.com)_
-- Type: _File name search based_
+- Type: _File name_ or _Moby Games ID_ search based
 - User credential support: _None required_
-- API request limit: _1 request per 10 seconds_
-- Rom limit per run: _35_
+- API request limit: _1 request per 5 seconds (Hobbyist subscription)_
 - Thread limit: _1_
 - Platform support: _[List](https://www.mobygames.com/browse/games)_ or see `mobygames_platforms.json` sibling to your `config.ini`
 - Media support: _`cover`, `screenshot`_
@@ -157,9 +156,7 @@ The database also supports many non-Amiga platforms, but there's no doubt that A
 
 MobyGames. What can I say. If you haven't heard about this database before you've been missing out. It's one of the best and oldest games databases on the internet. You'll probably come across references to MobyGames on other sites when searching for retro games. There's a reason for that - it's that good.
 
-There's a caveat to the module as it has quite strong restrictions for the number of requests that are allowed at any given time. This restriction is global for the entire Skyscraper user base, which means that it might quit on you if other users are currently scraping from it. For this reason it has been strongly limited inside of Skyscraper by forcing a maximum number of rom scrapings per run.
-
-Please use this module sparingly. And only ever use it to scrape those last few roms you can't get any data for using any of the other sources.
+There's a caveat to the module as it requires a subscription to get an API key, but you get well curated game information especially for hard to find titles. Examine the possible options: https://www.mobygames.com/api/subscribe/. Once you have obtained an API key add it to the [userCreds](CONFIGINI.md#usercreds) configuration (without any colon) in the [mobygames] INI section.
 
 ### IGDB
 
@@ -168,7 +165,7 @@ Please use this module sparingly. And only ever use it to scrape those last few
 - Shortname: _`igdb`_
 - Type: _Online_
 - Website: _[www.igdb.com](https://www.igdb.com)_
-- Type: _File name search based_
+- Type: _File name_ or _IGDB Game Id_ search based
 - User credential support: _Yes, free private API client-id and secret-key required! Read more below_
 - API request limit: _A maximum of 4 requests per seconds is allowed_
 - Thread limit: _4 (each being limited to 1 request per second)_
@@ -202,7 +199,7 @@ Substitute CLIENTID and SECRETKEY with your own details. And that's it, you shou
 - Shortname:  _`zxinfo`_, _`worldofspectrum`_, _`wos`_
 - Type: _Online_
 - Website: _[zxinfo.dk](https://zxinfo.dk)_
-- Type: _File name search based_
+- Type: _File name search_, _Game Id search_ or _Game hash search_
 - User credential support: _None required_
 - API request limit: _None_
 - Thread limit: _None_
@@ -249,7 +246,7 @@ Read a thorough description of this module [here](IMPORT.md).
 - Shortname: _`gamebase`_
 - Type: _Local_
 - Website: _[about the format](https://www.bu22.com/wiki/home)_
-- Type: _Exact filename, title or CRC match, for filename and title wildcards * and ? can be applied anywhere_
+- Type: _filename, title or CRC match, for filename and title wildcards '*' and '?' can be applied anywhere_
 - User credential support: _None required_
 - API request limit: _None_
 - Thread limit: 1

src/mobygames.cpp

@@ -31,6 +31,7 @@
 
 #include <QJsonArray>
 #include <QRandomGenerator>
+#include <QStringBuilder>
 #include <cstdio>
 
 static inline QMap<QString, QString> mobyRegionMap() {
@@ -56,7 +57,7 @@ static inline QMap<QString, QString> mobyRegionMap() {
 MobyGames::MobyGames(Settings *config, QSharedPointer<NetManager> manager)
     : AbstractScraper(config, manager, MatchType::MATCH_MANY) {
     connect(&limitTimer, &QTimer::timeout, &limiter, &QEventLoop::quit);
-    limitTimer.setInterval(10000); // 10 second request limit
+    limitTimer.setInterval(5000); // 5 second request limit (Hobbyist API)
     limitTimer.setSingleShot(false);
     limitTimer.start();
 
@@ -82,24 +83,15 @@ void MobyGames::getSearchResults(QList<GameEntry> &gameEntries,
 
     printf("Waiting as advised by MobyGames api restrictions...\n");
     limiter.exec();
-    bool ok;
-    int queryGameId = searchName.toInt(&ok);
-    QString req;
-    if (ok) {
-        req = QString(searchUrlPre + "?api_key=" +
-                      StrTools::unMagic(
-                          "175;229;170;189;188;202;211;117;164;165;185;209;164;"
-                          "234;180;155;199;209;224;231;193;190;173;175") +
-                      "&id=" + QString::number(queryGameId));
+    QString req = QString(searchUrlPre % "?api_key=" % config->password);
+    bool isMobyGameId;
+    int queryGameId = searchName.toInt(&isMobyGameId);
+    if (isMobyGameId) {
+        req = req % "&id=" % QString::number(queryGameId);
     } else {
-        req = QString(searchUrlPre + "?api_key=" +
-                      StrTools::unMagic(
-                          "175;229;170;189;188;202;211;117;164;165;185;209;164;"
-                          "234;180;155;199;209;224;231;193;190;173;175") +
-                      "&title=" + searchName +
-                      (platformId == -1
-                           ? ""
-                           : "&platform=" + QString::number(platformId)));
+        req = req % "&title=" % searchName %
+              (platformId == -1 ? ""
+                                : "&platform=" + QString::number(platformId));
         queryGameId = 0;
     }
     qDebug() << "Request: " << req;
@@ -113,10 +105,8 @@ void MobyGames::getSearchResults(QList<GameEntry> &gameEntries,
     }
 
     if (jsonDoc.object()["code"].toInt() == 429) {
-        printf(
-            "\033[1;31mToo many requests! This is probably because some other "
-            "Skyscraper user is currently using the 'mobygames' module. Please "
-            "wait a while and try again.\n\nNow quitting...\033[0m\n");
+        printf("\033[1;31mToo many requests! Please wait a while and try "
+               "again.\n\nNow quitting...\033[0m\n");
         reqRemaining = 0;
     }
 
@@ -135,11 +125,9 @@ void MobyGames::getSearchResults(QList<GameEntry> &gameEntries,
         while (!jsonPlatforms.isEmpty()) {
             QJsonObject jsonPlatform = jsonPlatforms.first().toObject();
             int gamePlafId = jsonPlatform["platform_id"].toInt();
-            game.url = searchUrlPre + "/" + game.id + "/platforms/" +
-                       QString::number(gamePlafId) + "?api_key=" +
-                       StrTools::unMagic(
-                           "175;229;170;189;188;202;211;117;164;165;185;209;"
-                           "164;234;180;155;199;209;224;231;193;190;173;175");
+            game.url = searchUrlPre % "/" % game.id % "/platforms/" %
+                       QString::number(gamePlafId) % "?api_key=" %
+                       config->password;
             game.releaseDate = jsonPlatform["first_release_date"].toString();
             game.platform = jsonPlatform["platform_name"].toString();
             bool matchPlafId = gamePlafId == platformId;
@@ -281,7 +269,7 @@ void MobyGames::getCover(GameEntry &game) {
     fflush(stdout);
     limiter.exec();
     QString req = QString(
-        game.url.left(game.url.indexOf("?api_key=")) + "/covers" +
+        game.url.left(game.url.indexOf("?api_key=")) % "/covers" %
         game.url.mid(game.url.indexOf("?api_key="),
                      game.url.length() - game.url.indexOf("?api_key=")));
     qDebug() << "Covers request" << req;
@@ -372,7 +360,7 @@ void MobyGames::getScreenshot(GameEntry &game) {
     fflush(stdout);
     limiter.exec();
     netComm->request(
-        game.url.left(game.url.indexOf("?api_key=")) + "/screenshots" +
+        game.url.left(game.url.indexOf("?api_key=")) % "/screenshots" %
         game.url.mid(game.url.indexOf("?api_key="),
                      game.url.length() - game.url.indexOf("?api_key=")));
     q.exec();

src/skyscraper.cpp

@@ -943,7 +943,8 @@ void Skyscraper::loadConfig(const QCommandLineParser &parser) {
     }
 
     if (config.importFolder.isEmpty()) {
-        config.importFolder = Config::getSkyFolder(Config::SkyFolderType::IMPORT);
+        config.importFolder =
+            Config::getSkyFolder(Config::SkyFolderType::IMPORT);
     }
     // If platform subfolder exists for import path, use it
     QDir importFolder(config.importFolder);
@@ -1040,11 +1041,14 @@ void Skyscraper::loadConfig(const QCommandLineParser &parser) {
         config.refresh = true;
     }
 
-    if (!config.userCreds.isEmpty() && config.userCreds.contains(":")) {
+    if (!config.userCreds.isEmpty()) {
         QList<QString> userCreds = config.userCreds.split(":");
         if (userCreds.length() == 2) {
             config.user = userCreds.at(0);
             config.password = userCreds.at(1);
+        } else if (userCreds.length() == 1) {
+            // API key
+            config.password = userCreds.at(0);
         }
     }
 
@@ -1129,13 +1133,17 @@ void Skyscraper::prepareScraping() {
         prepareIgdb(netComm, q);
     } else if (config.scraper == "mobygames" && config.threads != 1) {
         printf(
-            "\033[1;33mForcing 1 thread to accomodate limits in MobyGames "
+            "\033[1;33mForcing one thread to accomodate limits in MobyGames "
             "scraping module. Also be aware that MobyGames has a request limit "
-            "of 360 requests per hour for the entire Skyscraper user base. So "
-            "if someone else is currently using it, it will quit.\033[0m\n\n");
-        // Don't change these! This limit was set by request from Mobygames
+            "of 720 requests per hour for a Hobbyist subscription.\033[0m\n\n");
         config.threads = 1;
-        config.romLimit = 35;
+        if (config.password.isEmpty()) {
+            printf("The MobyGames scraping module requires an API key to "
+                   "work. Read more about that here: "
+                   "'https://gemba.github.io/skyscraper/"
+                   "SCRAPINGMODULES#mobygames'\n");
+            exit(1);
+        }
     } else if (config.scraper == "screenscraper") {
         prepareScreenscraper(netComm, q);
     } else if (config.scraper == "gamebase") {