Support for backcover/boxback (#197)

- Online: Screenscraper.fr and TGDB
- Local: esgamelist, import
- Updated documentation

Other changes:

- rewrote copy of gamelist media (QFlags)
- IGDB improvements and token lifetime fix

Minor changes:

- fix tests
- more compact table layout in mkdocs
- overwrite static Skyscraper config files on startup only when different from source

Author: Gemba

.whitesource

@@ -1,8 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <artwork>
-  <!-- copies the resourcetypes without any artwork applied to the respective mediafolder -->
-  <output type="screenshot"/>
-  <output type="marquee" resource="wheel" />
-  <output type="cover" />
-  <output type="wheel" />
+  <!-- 
+    No artwork is applied and the media files are copied as-is for the gamelist.
+    See ARTWORK.md or artwork.xml.example* for options.
+   -->
 </artwork>

batocera-artwork.xml

@@ -115,4 +115,17 @@
     <source>esgamelist</source>
     <source>screenscraper</source>
   </order>
+  <order type="fanart">
+    <source>import</source>
+    <source>esgamelist</source>
+    <source>screenscraper</source>
+    <source>thegamesdb</source>
+    <source>igdb</source>
+  </order>
+  <order type="backcover">
+    <source>import</source>
+    <source>esgamelist</source>
+    <source>screenscraper</source>
+    <source>thegamesdb</source>
+  </order>
 </priorities>

cache/priorities.xml.example

@@ -79,7 +79,7 @@ The developer of a game
 
 ##### players
 
-How many players are supported by a game
+How many players are supported by a game (simultaneous or alternating)
 
 ##### tags
 
@@ -111,7 +111,9 @@ Wheel (logo) image filename for a game (file stored below `wheels` subfolder)
 
 ##### marquee
 
-Marquee image filename for a game (file stored below `marquees` subfolder)
+Marquee image filename for a game (file stored below `marquees` subfolder).
+Historically this artwork contained the game title and the manufacturer. It
+was shown above the screen of an arcade machine.
 
 ##### texture
 
@@ -131,3 +133,8 @@ subfolder)
 
 Since v3.18: Background image displayed in some frontends (e.g. Batocera) and
 themes for a game (file stored below `fanarts` subfolder)
+
+##### backcover
+
+Since v3.18: Image of the back side of a game packaging, aka cover back side or
+box back (file stored below `backcovers` subfolder)

docs/CACHE.md

@@ -3,16 +3,33 @@
 This page summarizes the changes of each Skyscraper release, a [changlog for
 humans](https://keepachangelog.com).
 
-### Version 3.18.0 (2025-TBA)
+### Version 3.18.0 (2025-11-23)
+
+Opened [Github Discussions](https://github.com/Gemba/skyscraper/discussions),
+the place to put ideas or to table Skyscraper puzzles.
+
+#### Frontends and Scraping
 
 - Added: Support for [Batocera gamelists](FRONTENDS.md#batocera) incl. fanart
-  output
-- Added: Support for fanart gamelist output for ES-DE and some [EmulationStation
-  variants](CONFIGINI.md#gamelistvariants)
-- Added: Fanart scraping with Screenscraper-, TGDB- and Import-scraper. See flag
-  and config option [`fanarts`](CLIHELP.md#fanarts)
-- Added: Support for ES-DE miximages output (aka Skyscraper artwork) with [flag
-  miximages](CLIHELP.md#miximages)
+  and boxback output, thanks for external testing and feedback, @RandomNinjaAtk.
+- Added: Support for fanart and backcover gamelist output for ES-DE and some
+  [EmulationStation variants](CONFIGINI.md#gamelistvariants)
+- Added: Fanart scraping with Screenscraper-, The Games DB-, IGDB- and
+  Import-scraper. See flag and config option [`fanarts`](CLIHELP.md#fanarts)
+- Added: Support for ES-DE miximages output (aka Skyscraper artwork) with [`--flag
+  miximages`](CLIHELP.md#miximages)
+- Added: Scraping of backcover (aka. boxback) from Screenscraper and The Games
+  DB and output for Batocera and ES-DE. See flag and config INI option
+  [`backcovers`](CLIHELP.md#backcovers)
+- Updated: The Games DB scraper for Sega Genesis also tries Sega Megadrive for
+  matches, if you made changes to `platforms_idmap.csv`, see [the platform
+  doc](PLATFORMS.md#transferring-local-platform-changes) to move it to another
+  file, to benefit from this update.
+- Fixed: Wrong IGDB token expire time calculation, which resulted in games not
+  to be found at their site in some scenarios.
+
+#### CLI and Options
+
 - Added: Option to force Screenscraper scrapes to use the stem of the game
   filename instead of derived parameters. See
   [`--searchstem`](CLIHELP.md#-searchstem-extension) and the last question in
@@ -25,7 +42,12 @@ humans](https://keepachangelog.com).
   to `--flags videos`. However, in the config file accept only plural as before.
 - Added: Flag [`--buildinfo`](CLIHELP.md#--buildinfo), comes in handy when
   reporting an issue.
-- Updated: Option `theInFront` covers also the indefinite article 'a'.
+- Updated: The prefix "~/" in path-like configuration options will be expanded
+  to [QDir:homePath()](https://doc.qt.io/qt-6/qdir.html#homePath).
+- Updated: Option `theInFront` now covers also the indefinite article 'a'.
+
+#### Varia
+
 - Updated: macOS installation instructions to use Qt6
 - Updated: Docker uses Ubuntu 24.04 and Qt6
 - Updated: Documentation, added usage level for configuration options. See
@@ -35,6 +57,7 @@ humans](https://keepachangelog.com).
 - Fixed: Various edge cases remediated, esp. #167 and #169, thanks to all
   reporters!
 
+
 ### Version 3.17.0 (2025-05-04)
 
 - Changed: Handling of relative path in configuration options adapted to enable

docs/CHANGELOG.md

@@ -452,6 +452,10 @@ From Skyscraper 3.5.0 all command-line options that change the scraping behaviou
 
 To enable multiple flags separate them by commas (eg. `--flags FLAG1,FLAG2`) or apply `--flags` option multiple times.
 
+#### backcovers
+
+By default Skyscraper doesn't scrape and cache game backcovers resources because not all scraping sites provide this data and also only some frontends support a boxback display. You can enable it by using this flag. Consider setting this in [`config.ini`](CONFIGINI.md#backcovers) instead.
+
 #### fanarts
 
 By default Skyscraper doesn't scrape and cache game fanart resources because not all scraping sites provide this data and also only some frontends support fanart display. You can enable it by using this flag. Consider setting this in [`config.ini`](CONFIGINI.md#fanarts) instead.
@@ -478,7 +482,7 @@ This is an epheremal flag and has only an effect when used with the ES-DE fronte
 
 !!! warning "Epheremal Flag for ES-DE"
 
-    Once the miximage of ES-DE feature has been released, this flag may be removed again from Skyscraper.
+    Once the miximage of ES-DE feature has been released, this flag may be removed again from Skyscraper and the logic will be adapted to match the ES-DE behaviour.
 
 #### nobrackets
 
@@ -552,6 +556,10 @@ This flag is _only_ relevant when generating a game list (by leaving out the `-s
 
 Only relevant when generating an EmulationStation, a Retrobat or a Pegasus game list, with the `-f` option. Emulationstation is the default frontend when the `-f` option is left out. The `relative` flag forces the rom and any media paths (if they are the same as the input folder) inside the game list to be relative to the rom input folder. Consider setting this in [`config.ini`](CONFIGINI.md#relativepaths) instead.
 
+#### skipexistingbackcovers
+
+When generating gamelists, skip copying backcovers that already exist in the media output folder.
+
 #### skipexistingcovers
 
 When generating gamelists, skip processing covers that already exist in the media output folder.
@@ -620,13 +628,12 @@ Some scraping modules use file checksums to identify the game in their databases
 
 #### videos
 
-By default Skyscraper doesn't scrape and cache video resources because of the significant disk space required to save them. You can enable videos using this flag. Consider setting this in [`config.ini`](CONFIGINI.md#videos) instead.
+By default Skyscraper doesn't scrape and cache video resources because of the significant disk space required to save them. You can enable videos using this flag, also explicitly set this flag to output videos for the frontend/gamelist. Consider setting this in [`config.ini`](CONFIGINI.md#videos) instead.
 
 **Example(s)**
 
 ```
-Skyscraper -p amiga --flags forcefilename,nosubdirs,skipexistingwheels
-Skyscraper -p nes --flags videos,nomarquees
+Skyscraper -p nes --flags videos
 ```
 
 ### --gamelistfilename <FILENAME>
@@ -763,9 +770,11 @@ Skyscraper has a resource cache which works just like the browser cache in Firef
 
 You can force all data to be refetched from the servers by setting this option, effectively updating the cached data with new data from the source.
 
+If you provide one or more game filenames on the command line, the refresh flag is automatically set.
+
 !!! note
 
-     _Only_ use this option if you know data has changed for several roms at the source. Otherwise you are hammering the servers for no reason.
+     _Only_ use this option if you know data has changed for several roms at the source. Otherwise you are hammering online servers for no reason.
 
 **Example(s)**
 

docs/CLIHELP.md

@@ -73,6 +73,7 @@ This is an alphabetical index of all configuration options their usage level and
 | [addExtensions](CONFIGINI.md#addextensions)                 | Advanced       |    Y     |       Y        |                |               |
 | [addFolders](CONFIGINI.md#addfolders)                       | Expert         |          |                |       Y        |               |
 | [artworkXml](CONFIGINI.md#artworkxml)                       | Advanced       |    Y     |       Y        |       Y        |               |
+| [backcovers](CONFIGINI.md#backcovers)                       | Basic          |    Y     |                |                |               |
 | [brackets](CONFIGINI.md#brackets)                           | Basic          |    Y     |       Y        |       Y        |               |
 | [cacheCovers](CONFIGINI.md#cachecovers)                     | Basic          |    Y     |       Y        |                |       Y       |
 | [cacheFolder](CONFIGINI.md#cachefolder)                     | Basic          |    Y     |       Y        |                |               |
@@ -1098,6 +1099,23 @@ Allowed in sections: Only for frontends `[emulationstation]`, `[esde]` or `[retr
 
 ---
 
+#### backcovers
+
+By default Skyscraper doesn't scrape and cache game backcover resources because
+not all scraping sites provide this data and also only some frontends support
+the display of the cover backside. If enabled Skyscraper will collect game
+manuals for the scraping modules that provide this data. For the frontends ES-DE
+and Batocera no further option must be set to enable the output of the boxback
+in the gamelist and into the appropriate folder during gamelist creation.
+You may want to review and [adjust the cache
+priorities](CACHE.md#resource-and-scraping-module-priorities) for the fanart
+selection when data from several scraping modules is present in your cache.
+
+Default value: false  
+Allowed in sections: `[main]`
+
+---
+
 #### fanarts
 
 By default Skyscraper doesn't scrape and cache game fanart resources because not
@@ -1107,10 +1125,13 @@ modules that provide this data. For the frontends ES-DE and Batocera no further
 option must be set to enable the output of fanart in the gamelist and into the
 appropriate folder during gamelist creation. For other EmulationStation forks
 where themes support the display of fanart, see also option
-[gameListVariants](CONFIGINI.md#gamelistvariants).
+[gameListVariants](CONFIGINI.md#gamelistvariants).  
+You may want to review and [adjust the cache
+priorities](CACHE.md#resource-and-scraping-module-priorities) for the fanart
+selection when data from several scraping modules is present in your cache.
 
 Default value: false  
-Allowed in sections: `[main]`, `[<PLATFORM>]`
+Allowed in sections: `[main]`
 
 ---
 
@@ -1124,6 +1145,9 @@ and Batocera no further option must be set to enable the output of the PDF
 manuals to the appropriate folder during gamelist creation. For other
 EmulationStation forks which support PDF manual display, see also option
 [gameListVariants](CONFIGINI.md#gamelistvariants).
+You may want to review and [adjust the cache
+priorities](CACHE.md#resource-and-scraping-module-priorities) for the fanart
+selection when data from several scraping modules is present in your cache.
 
 Default value: false  
 Allowed in sections: `[main]`, `[<PLATFORM>]`

docs/CONFIGINI.md

@@ -83,3 +83,11 @@ And now for the questions (with answers)...
     Yes, from Skyscraper 3.18 onwards. The cause is that Screenscraper is the only scraping module which by default passes the extension to the server during the scraping information along with other information (checksums, filesize).  
     When scraping a set of games use the [`--searchstem`](CLIHELP.md#-searchstem-extension) option, i.e. `--searchstem '*.<customext>'`, that way Skyscraper will explicitly query only for the filename stem (=filename without file extension) of the game file. Example: Let's assume you are using `*.desktop` as custom game file extension, then the command would be `Skyscraper -s screenscraper -p <platform> --searchstem '*.desktop'`. You may also use the config file [counterpart](CONFIGINI.md#searchstem). On top you may add `--verbosity 3` to investigate what search term is presented to the Screenscraper server.  
     However, when querying for a single game with [`--query`](CLIHELP.md#-query-string) you don't have to specify this custom extension.
+
+??? Question "I have been enrolled on IGDB, but whenever I try to scrape I get a HTTP 401 (unauthorized). How to resolve this?"
+
+    It may be caused by a system without a realtime clock (like Raspberry Pi) or on systems with a highly unsynchronized clock. As the IGDB token lifetime is calculated on the system time, this might be the culprit. You may remove the file `~/.skyscraper/igdbToken.dat` and try again to scrape with IGDB.
+
+??? Question "I have scraped manuals/fanarts/backcovers from different sources, but in the gamelist/frontend I only get the most recent scraped. How can I correct this?"
+
+    Most likely the `priorities.xml` file is missing these entries for the platform. You can find an example for manual, fanart and backcover cache types and their priority of selection at the [cache documentation](CACHE.md#resource-and-scraping module-priorities) and at the end of the the `~/.skyscraper/priorities.xml.example` file. Once you have added these entries and adjusted them your preferences you will get the media outputted in that order instead of the most recent scraped media.