Skip to content

Commit 9b04437

Browse files
authored
Update to TGDB and MobyGames (#186)
- Mobygames API v2 and scraper doc update - fix to catch empty API key for Mobygames - fix edge case on tgdb scraper - update map files (JSON files) for api scrapers - added more libpng warnings to mute - moved readJson() to parent class
1 parent 674c80b commit 9b04437

25 files changed

Lines changed: 506 additions & 298 deletions

docs/CACHE.md

Lines changed: 17 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -54,17 +54,20 @@ The database consists of resource entries connected to a unique id. The id is ca
5454

5555
#### Resource Types
5656

57+
These resource types contain the provided game data from a scraping
58+
source.
59+
5760
##### title
5861

59-
A game title
62+
The game title
6063

6164
##### platform
6265

63-
A game platform
66+
The game platform, may differ from provided platform with `-p`.
6467

6568
##### description
6669

67-
A game description
70+
The game description
6871

6972
##### publisher
7073

@@ -96,32 +99,35 @@ Game rating, real number between 0 and 1
9699

97100
##### cover
98101

99-
A cover image filename for a game (file exists in `covers` subfolder)
102+
Cover image filename for a game (file stored below `covers` subfolder)
100103

101104
##### screenshot
102105

103-
A screenshot image filename for a game (file exists in `screenshots` subfolder)
106+
Screenshot image filename for a game (file stored below `screenshots` subfolder)
104107

105108
##### wheel
106109

107-
A wheel (logo) image filename for a game (file exists in `wheels` subfolder)
110+
Wheel (logo) image filename for a game (file stored below `wheels` subfolder)
108111

109112
##### marquee
110113

111-
A marquee image filename for a game (file exists in `marquees` subfolder)
114+
Marquee image filename for a game (file stored below `marquees` subfolder)
112115

113116
##### texture
114117

115-
A texture image (disc/cartridge) filename for a game (file exists in `textures` subfolder)
118+
Texture image (disc/cartridge) filename for a game (file stored below `textures`
119+
subfolder)
116120

117121
##### video
118122

119-
A video file filename for a game (file exists in `videos` subfolder)
123+
Video file filename for a game (file stored below `videos` subfolder)
120124

121125
##### manual
122126

123-
(Since v3.12) A manual (PDF) file filename for a game (file exists in `manuals` subfolder)
127+
Since v3.12: Manual (PDF) file filename for a game (file stored below `manuals`
128+
subfolder)
124129

125130
##### fanart
126131

127-
(Since v3.18) A background image displayed in some frontends (e.g. Batocera) and themes for a game (file exists in `fanarts` subfolder)
132+
Since v3.18: Background image displayed in some frontends (e.g. Batocera) and
133+
themes for a game (file stored below `fanarts` subfolder)

docs/PLATFORMS.md

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ maintain local changes to the `platforms_idmap.csv` in a separate file with a
5858
`peas.json` overwrites the one from `peas.json`. The same applies for the
5959
platforms ID-mapping (see next section).
6060

61-
!!! tip
61+
!!! tip "Avoid Duplication"
6262

6363
If you need a specific folder name for a platform (on your setup or due to an
6464
EmulationStation theme) use a symbolic link (for example `megadrive` (=folder)
@@ -72,16 +72,21 @@ The second file is used for to instruct scrapers to lookup games by the numeric
7272
platform identifier the scraping site uses for queries. The file
7373
`platforms_idmap.csv` defines exact platform id for the web APIs of
7474
Screenscraper, MobyGames and the GamesDB. It is a CSV file which maps the
75-
platform handle (e.g. `megadrive`) to the respective platform id of the scraping
75+
platform handle (e.g. `c64`) to the respective platform id of the scraping
7676
site (selected with the CLI option `-s`):
7777

7878
```csv
7979
folder,screenscraper_id,mobygames_id,tgdb_id
8080
[...]
81-
megadrive,1,16,36
81+
c64,66,27,40
8282
[...]
8383
```
8484

85+
!!! tip "The Games DB"
86+
87+
The game data at TGDB is in rare case in different platforms to be found. Prominent example is Sega's Genesis respecitve Mega Drive. For these edgecases you may find the platform ids in the `platforms_idmap.csv` separated with an `|`. This means all platform ids will be tried to find a match in left-to-right order of the definition.
88+
89+
8590
You can display the number with their platform name on each of the three
8691
scraping sites in a more readable format with the script
8792
`peas_and_idmap_verify.py`. Find the script sibling to the Skyscraper
@@ -90,13 +95,15 @@ site):
9095

9196
```
9297
[...]
93-
├── megadrive
94-
│ ├── ScrS 1: Megadrive
95-
│ ├── Moby 16: Genesis
96-
│ └── TGDB 36: Sega Mega Drive
98+
├── c64
99+
│ ├── ScrS 66: Commodore 64
100+
│ ├── Moby 27: Commodore 64
101+
│ └── TGDB 40: Commodore 64
97102
[...]
98103
```
99104

105+
After the platform id is the textual representation the scraper source uses.
106+
100107
## How to Change Platform Aliases or Gamefile Extensions
101108

102109
Follow instructions in this section if you only want to overwrite platform alias
@@ -166,7 +173,7 @@ If you have multiple platforms defined in your local file make sure the platform
166173
blocks are separated by a comma `,`.
167174

168175

169-
!!! tip
176+
!!! tip "Case-sensitivity in EmulationStation Configuration"
170177

171178
On RetroPie you may also have to edit `~/.emulationstation/es_systems.cfg` and
172179
add `.bin` and `.BIN`.

docs/SCRAPINGMODULES.md

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,17 +18,18 @@ over a table cell to display the scraper module as tooltip:
1818
| [GameBase](#gamebase-db) | ✓ {title='GameBase'} | ✓ {title='GameBase'} |   {title='GameBase'} | ✓ {title='GameBase'} | ✓ {title='GameBase'} | ✓ {title='GameBase'} | ✓ {title='GameBase'} | ✓ {title='GameBase'} | ✓ ² {title='GameBase'} | ✓ {title='GameBase'} | ✓ {title='GameBase'} |   {title='GameBase'} |   {title='GameBase'} |   {title='GameBase'} |   {title='GameBase'} |   {title='GameBase'} |   {title='GameBase'} |
1919
| [Internet Game DB (IGDB)](#igdb) | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} | ✓ {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |   {title='Internet Game DB (IGDB)'} |
2020
| [File Import](#custom-resource-import) | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} | ✓ {title='File Import'} |
21-
| [MobyGames](#mobygames) | ✓ {title='MobyGames'} | {title='MobyGames'} | ✓ {title='MobyGames'} | {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} | {title='MobyGames'} | {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |
21+
| [MobyGames](#mobygames) | ✓ {title='MobyGames'} | ✓ ⁴ {title='MobyGames'} | ✓ {title='MobyGames'} | See ⁴ {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} | See ⁴ {title='MobyGames'} | See ⁴ {title='MobyGames'} | ✓ {title='MobyGames'} | ✓ {title='MobyGames'} | See ⁴ {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |   {title='MobyGames'} |
2222
| [OpenRetro](#openretro) | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} |   {title='OpenRetro'} | ✓ {title='OpenRetro'} | ✓ {title='OpenRetro'} |   {title='OpenRetro'} |   {title='OpenRetro'} | ✓ {title='OpenRetro'} |   {title='OpenRetro'} |   {title='OpenRetro'} |   {title='OpenRetro'} |
2323
| [ScreenScraper](#screenscraper) | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} | ✓ {title='ScreenScraper'} |
2424
| [The Games DB](#thegamesdb-tgdb) | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} |   {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} |   {title='The Games DB'} | ✓ {title='The Games DB'} | ✓ {title='The Games DB'} |   {title='The Games DB'} |   {title='The Games DB'} | ✓ {title='The Games DB'} |
2525
| [ZXInfo](#zxinfo-formerly-world-of-spectrum) | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} |   {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ ³ {title='ZXInfo'} | ✓ {title='ZXInfo'} | ✓ {title='ZXInfo'} |   {title='ZXInfo'} |   {title='ZXInfo'} |   {title='ZXInfo'} |   {title='ZXInfo'} |   {title='ZXInfo'} |   {title='ZXInfo'} |
26-
| Coverage | 10/10 | 10/10 | 8/10 | 10/10 | 9/10 | 10/10 | 10/10 | 8/10 | 8/10 | 10/10 | 10/10 | 4/10 | 4/10 | 5/10 | 3/10 | 2/10 | |
26+
| Coverage | 10/10 | 10/10 | 8/10 | 9/10 | 9/10 | 10/10 | 10/10 | 7/10 | 7/10 | 10/10 | 10/10 | 4/10 | 4/10 | 5/10 | 3/10 | 2/10 | 3/10 |
2727

2828
**Remarks**:
2929
¹ Skyscraper uses ArcadeDB's Flyer and as a failsafe the Title screen, as Arcade games usually where not sold in a box
3030
² GameBase provides only an adult flag, thus it is either 18 or no age rating
31-
³ The source zxinfo.dk provides only an x-rated flag, thus it is either 18 or no age rating
31+
³ The source zxinfo.dk provides only an x-rated flag, thus it is either 18 or no age rating
32+
⁴ Release date will contain the first release date worldwde with Hobbyist API subscription. Age Recommendation, Rating, Max. Players, Video and release date per platform require an APIv2 Bronze subscription or higher. Skyscraper supporting anything else than a Hobbyist subscription is very unlikely.
3233

3334
### Recognized Keywords in Query
3435

@@ -39,11 +40,11 @@ over a table cell to display the scraper module as tooltip:
3940
| gamebase | Game filename, Game title and Game CRC (automatically detected). Except for CRC, globbing patterns (`*` and `'?`) can be used. |
4041
| igdb | Title or use id=... to query by IGDB game ID |
4142
| import | No query supported |
42-
| mobygames | Title or numeric MobyGames ID (see _Identifiers_ section their website / on game details) |
43+
| mobygames | Title or numeric MobyGames ID (see _Moby ID:_ right below the title when displaying a game on the website) |
4344
| openretro | Only title |
4445
| screenscraper | romnom=, crc=, md5=, sha1=; see [Screenscraper documentation](https://www.screenscraper.fr/webapi2.php?alpha=0&numpage=0#jeuInfos) for description |
4546
| thegamesdb, tgdb | Only title |
46-
| zxinfo (worldofspectrum) | Title, game Id (id=...) or game filehash (MD5 or SHA512) |
47+
| zxinfo (worldofspectrum) | Title, game Id (id=...) or game filehash (MD5 or SHA512) |
4748

4849
!!! tip "Aliases for Game Filenames"
4950

@@ -143,7 +144,7 @@ The database also supports many non-Amiga platforms, but there's no doubt that A
143144
- Shortname: _`mobygames`_
144145
- Type: _Online_
145146
- Website: _[www.mobygames.com](https://www.mobygames.com)_
146-
- Type: _File name_ or _Moby Games ID_ search based
147+
- Type: _File name_ or _MobyGames ID_ search based
147148
- User credential support: _None required_
148149
- API request limit: _1 request per 5 seconds (Hobbyist subscription)_
149150
- Thread limit: _1_
@@ -153,9 +154,23 @@ The database also supports many non-Amiga platforms, but there's no doubt that A
153154
- `Skyscraper -p fba -s mobygames <SINGLE FILE TO SCRAPE>`
154155
- `Skyscraper -p fba -s mobygames --startat <FILE TO START AT> --endat <FILE TO END AT>`
155156

156-
MobyGames. What can I say. If you haven't heard about this database before you've been missing out. It's one of the best and oldest games databases on the internet. You'll probably come across references to MobyGames on other sites when searching for retro games. There's a reason for that - it's that good.
157+
MobyGames APIv2 imposes more limits than APIv1. Not only you will need a payed
158+
subscription (to get an API key), but even with the entry-level (=Hobbyist)
159+
subscription you cannot scrape the same data as with APIv1. These are the
160+
limitations:
157161

158-
There's a caveat to the module as it requires a subscription to get an API key, but you get well curated game information especially for hard to find titles. Examine the possible options: https://www.mobygames.com/api/subscribe/. Once you have obtained an API key add it to the [userCreds](CONFIGINI.md#usercreds) configuration (without any colon) in the [mobygames] INI section.
162+
- Release date will contain only the first release date worldwde with Hobbyist API
163+
subscription.
164+
- Age Recommendation, Rating, Maximum of Players, Video and release date per
165+
platform require an APIv2 Bronze subscription or higher.
166+
167+
Skyscraper supporting anything else than a Hobbyist subscription is very
168+
unlikely. It is saddening to see the service of MobyGames degrading after the
169+
acquisition by Atari SA.
170+
171+
However, once you have obtained an API key (starting with `moby_...`) add it to
172+
the [userCreds](CONFIGINI.md#usercreds) configuration (without any colon) in the
173+
`[mobygames]` INI-file section.
159174

160175
### IGDB
161176

mobygames_platforms.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
"84": "Browser",
4343
"231": "Bubble",
4444
"154": "Camputers Lynx",
45+
"332": "Casio FP-1000/1100",
4546
"124": "Casio Loopy",
4647
"306": "Casio Programmable Calculator",
4748
"125": "Casio PV-1000",
@@ -186,6 +187,7 @@
186187
"44": "Nintendo DS",
187188
"87": "Nintendo DSi",
188189
"203": "Nintendo Switch",
190+
"333": "Nintendo Switch 2",
189191
"266": "North Star",
190192
"244": "Noval 760",
191193
"116": "Nuon",
@@ -306,7 +308,7 @@
306308
"82": "Wii",
307309
"132": "Wii U",
308310
"3": "Windows",
309-
"5": "Windows 3.x",
311+
"5": "Windows 16-bit",
310312
"140": "Windows Apps",
311313
"66": "Windows Mobile",
312314
"98": "Windows Phone",

peas.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1092,6 +1092,7 @@
10921092
"genesis",
10931093
"mega drive",
10941094
"sega mega drive",
1095+
"sega genesis",
10951096
"sega mega drive/genesis"
10961097
],
10971098
"formats": [

platforms_idmap.csv

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ mame-advmame,75,143,23
5151
mame-libretro,75,143,23
5252
mame-mame4all,75,143,23
5353
mastersystem,2,26,35
54-
megadrive,1,16,36
54+
megadrive,1,16,36|18
5555
moto,141,147,-1
5656
msx,113,57,4929
5757
n64,14,9,3
@@ -122,7 +122,7 @@ naomi,56,-1,23
122122
neogeocd,70,54,4956
123123
palm,219,65,-1
124124
pcenginecd,114,45,4955
125-
plus4,99,115,5007
125+
plus4,99,115,5007|5006
126126
ps3,59,81,12
127127
ps4,60,141,4919
128128
psvita,62,105,39

0 commit comments

Comments
 (0)