Batch of fanart/Batocera support

- Scraping for fanart in import, TGDB and Screenscraper
- Copy media files with Batocera filename convention
- Enable output of manuals automatically when frontend ESDE (manual must be present in cache)
- Enable output of manuals,fanart automatically when frontend Batocera (manual must be present in cache)
- Flat artwork file (batocera-artwork.xml) for compositor (see also config.ini.example)
- Set fanart and manual flags when gamelistvariants of ES are used
- Import module automagically imports fanart, manuals and videos when provided in import folder
- Add Batocera usage in FRONTENDS.md doc

Author: Gemba

VERSION.ini

@@ -1 +1 @@
-VERSION="3.17.5"
+VERSION="3.18.0-dev"

batocera-artwork.xml

@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Just the screenshot without any decorations or scaling -->
+<artwork>
+  <output type="screenshot"/>
+</artwork>

config.ini.example

@@ -187,3 +187,6 @@
 cacheRefresh="true"
 ; https://github.com/RetroPie/RetroPie-Setup/pull/3704
 cacheScreenshots="false"
+
+[batocera]
+artworkXml="batocera-artwork.xml"

docs/CACHE.md

@@ -35,7 +35,7 @@ Preferred way of editing the cache is via the various [`--cache`](CLIHELP.md#-ca
     If you decide to add your own files to the subfolders, you risk them being deleted by Skyscraper later on if it is run with one of the cache cleanup command line options. You've been warned!
 
 
-**Other cool stuff you CAN DO**: Each subfolder in the `/home/<USER>/.skyscraper/cache/` folder is self-contained and can be copied to other Skyscraper installations at your convenience. Just copy the folder itself over to some other computer that has Skyscraper 1.6.0 or later installed, and you can make use of the data when generating game lists. If you add it at a non-default location, set the custom folder with `-d <FOLDER>`.
+**Other cool stuff you CAN DO**: Each subfolder in the `/home/<USER>/.skyscraper/cache/` folder is self-contained and can be copied to other Skyscraper installations at your convenience. Just copy the folder itself over to some other computer that has Skyscraper v1.6 or later installed, and you can make use of the data when generating game lists. If you add it at a non-default location, set the custom folder with `-d <FOLDER>`.
 
 ### Resource Cache Format
 
@@ -52,10 +52,6 @@ The database consists of resource entries connected to a unique id. The id is ca
    timestamp="<UNIX TIMESTAMP IN MSECS>">Resource data</resource>
 ```
 
-!!! note
-
-    Pre-3.3.0 versions of Skyscraper used `sha1` as the name of the unique id key. Later versions use `id`.
-
 #### Resource Types
 
 ##### title
@@ -124,4 +120,8 @@ A video file filename for a game (file exists in `videos` subfolder)
 
 ##### manual
 
-A manual (PDF) file filename for a game (file exists in `manuals` subfolder)
+(Since v3.12) A manual (PDF) file filename for a game (file exists in `manuals` subfolder)
+
+##### fanart
+
+(Since v3.18) A background image displayed in some frontends (e.g. Batocera) and themes for a game (file exists in `fanarts` subfolder)

docs/CLIHELP.md

@@ -448,6 +448,10 @@ From Skyscraper 3.5.0 all command-line options that change the scraping behaviou
 
 To enable multiple flags separate them by commas (eg. `--flags FLAG1,FLAG2`) or apply `--flags` option multiple times.
 
+#### fanart
+
+By default Skyscraper doesn't scrape and cache game fanart resources because not all scraping sites provide this data and also only some frontends support fanart display. You can enable it by using this flag. Consider setting this in [`config.ini`](CONFIGINI.md#fanart) instead.
+
 #### forcefilename
 
 This flag forces Skyscraper to use the filename (excluding extension) instead of the cached titles when generating a game list. Consider setting this in [`config.ini`](CONFIGINI.md#forcefilename) instead.
@@ -540,6 +544,10 @@ Only relevant when generating an EmulationStation, a Retrobat or a Pegasus game
 
 When generating gamelists, skip processing covers that already exist in the media output folder.
 
+#### skipexistingfanart
+
+When generating gamelists, skip copying fanart files that already exist in the media output folder.
+
 #### skipexistingmanuals
 
 When generating gamelists, skip copying manuals that already exist in the media output folder.

docs/CONFIGINI.md

@@ -81,6 +81,7 @@ This is an alphabetical index of all configuration options their usage level and
 | [excludeFrom](CONFIGINI.md#excludefrom)                     | Advanced       |    Y     |       Y        |                |               |
 | [excludePattern](CONFIGINI.md#excludepattern)               | Advanced       |    Y     |       Y        |       Y        |               |
 | [extensions](CONFIGINI.md#extensions)                       | Expert         |          |       Y        |                |               |
+| [fanart](CONFIGINI.md#fanart)                               | Basic          |    Y     |                |                |               |
 | [forceFilename](CONFIGINI.md#forcefilename)                 | Advanced       |    Y     |       Y        |       Y        |               |
 | [frontend](CONFIGINI.md#frontend)                           | Basic          |    Y     |                |                |               |
 | [gameBaseFile](CONFIGINI.md#gamebasefile)                   | Expert         |          |       Y        |                |               |
@@ -290,7 +291,8 @@ Cleans up some misformatting in scraped description:
 2. Multiple spaces between sentences are reduced to one space
 3. Bulletpoint beginning with \* or ● are replaced with a dash
 4. Stylized ellipsis (… Unicode:`…`) is replaced with three dot characters
-5. Multiple exclamation marks are reduced to one, unless for game titles are explicitly typed like that, like 'Super Punch-Out!!'.
+5. Multiple exclamation marks are reduced to one, unless for game titles are
+   explicitly typed like that, like 'Super Punch-Out!!'.
 
 !!! quote
 
@@ -424,7 +426,7 @@ Allowed in sections: `[main]`, `[<PLATFORM>]`, `[<FRONTEND>]`
 
 #### videos
 
-By default Skyscraper doesn't scrape and cache video resources because of the significant disk space required to save them. You can enable videos using this option.
+By default Skyscraper doesn't scrape and cache video resources because of the significant disk space required to save them. You can enable videos using this option. If your frontend supports video display also explicitly set this option to true. See also the option to [symlink video files](#symlink) instead of copying, if space is a premium.
 
 Default value: `false`  
 Allowed in sections: `[main]`, `[<PLATFORM>]`, `[<FRONTEND>]`, `[<SCRAPER>]`
@@ -1060,9 +1062,32 @@ Allowed in sections: Only for frontends `[emulationstation]`, `[esde]` or `[retr
 
 ---
 
+#### fanart
+
+By default Skyscraper doesn't scrape and cache game fanart resources because not
+all scraping sites provide this data and also only some frontends support fanart
+display. If enabled Skyscraper will collect game manuals for the scraping
+modules that provide this data. For frontend Batocera no further option must be
+set to enable the output of fanart in the gamelist and into the appropriate
+folder during gamelist creation. For other EmulationStation forks where themes
+support the display of fanart, see also option
+[gameListVariants](CONFIGINI.md#gamelistvariants).
+
+Default value: false  
+Allowed in sections: `[main]`, `[<PLATFORM>]`
+
+---
+
 #### manuals
 
-By default Skyscraper doesn't scrape and cache game manuals resources because not all scraping sites provide this data and also only some frontends support PDF display of these game manuals. If enabled Skyscraper will collect game manuals for the scraping modules that provide this data. For frontend ES-DE no further option must be set to enable the output of the PDF manuals to the appropriate folder. For other EmulationStation forks see also option [gameListVariants](CONFIGINI.md#gamelistvariants).
+By default Skyscraper doesn't scrape and cache game manuals resources because
+not all scraping sites provide this data and also only some frontends support
+PDF display of these game manuals. If enabled Skyscraper will collect game
+manuals for the scraping modules that provide this data. For frontend ES-DE and
+Batocera no further option must be set to enable the output of the PDF manuals
+to the appropriate folder during gamelist creation. For other EmulationStation
+forks which support PDF manual display, see also option
+[gameListVariants](CONFIGINI.md#gamelistvariants).
 
 Default value: false  
 Allowed in sections: `[main]`, `[<PLATFORM>]`

docs/FRONTENDS.md

@@ -106,6 +106,116 @@ file will be preserved: For these on top is the `folderlink` element is preserve
 
 This is modeled after EmualtionStation as it uses it with slight differences.
 
+### Batocera
+
+Over time the fork of EmulationStation in Batocera has diverted stark from the
+initial EmulationStation. Thus, the gamelist format is somewhat different, but
+the textual scraping elements are still the same. Most notably Batocera
+EmulationStation stores a lot of additional information in the gamelist, and the
+set of information may differ for each game.  
+Skyscraper aims to cover the most frequent used elements for scraping, currently
+fanart, manuals and videos. See the [scraping
+modules](SCRAPINGMODULES.md#capabilities-of-scrapers) for support of these
+mediafiles.
+
+This is the complete set of scraping binary data supported by Skyscraper:
+
+| Batocera Gamelist XML-Element | Skyscraper support |
+| :---------------------------- | :----------------: |
+| boxart                        |         ✓          |
+| fanart                        |         ✓          |
+| image (in game Screenshot)    |         ✓          |
+| manual                        |         ✓          |
+| marquee                       |         ✓          |
+| video                         |         ✓          |
+| wheel                         |         ✓          |
+
+ROMs are expected to be in the input folder `/userdata/roms/<PLATFORM>` for
+every `<PLATFORM>` you scrape.
+
+-   Default game list location: `/userdata/roms/<PLATFORM>`
+-   Default game list filename: `gamelist.xml`
+-   Default media file location: `/userdata/roms/<PLATFORM>/{images,videos,manuals}`
+
+#### Metadata preservation
+
+These extra elements are preserved.
+
+| Batocera Gamelist XML-Element | When present in Gamelist  |
+| :---------------------------- | :-----------------------: |
+| `bezel`                       | Preserved                 |
+| `boxback`                     | Preserved                 |
+| `cartridge`                   | Preserved                 |
+| `magazine`                    | Preserved                 |
+| `map`                         | Preserved                 |
+| `mix`                         | Preserved                 |
+| `music`                       | Preserved                 |
+| `thumbnail`                   | Preserved                 |
+| `titleshot`                   | Preserved                 |
+
+Also all other non scrapable elements are preserved (Batocera EmulationStation
+adds those on occasion) like: `cheevosHash`, `cheevosId`, `scrap`, ...
+
+#### Usage of Skyscraper for Batocera
+
+It is possible to compile Skyscraper on Batocera supported systems but you will
+need the build-toolchain (at least GCC, make and Qt). Additionally, the
+configuration files must be installed in a defined location (see the `PREFIX`
+option in the top-level `README.md`, if you are keen to compile Skyscraper on
+Batocera).
+
+A more convienient way is to use Skyscraper from you Desktop system: Below you
+can find a step-by-step guide for Linux systems, macOS should be similar.
+Windows desktop users can use SMB shares and can adapt the following steps.
+
+1. Mount the root folder of Batocera in your Desktop system. Let the mountpoint
+   be `/home/mylogin/bato_sshfs` in this example:
+   ```bash
+   mount -t sshfs root@<batocera-IP-address>:/ /home/mylogin/bato_sshfs
+   ```
+2. Change to the platform folder (in this case `snes`) you want to scrape:
+   ```bash
+   cd /home/mylogin/bato_sshfs/userdata/roms/snes
+   ```
+3. Then run on your Desktop system to scrape data from screenscraper and put it
+   into the Skyscraper cache on your Desktop system:
+   ```bash
+   Skyscraper -s screenscraper -p snes --flags fanart,manuals,videos -i "$(pwd)"
+   -g "$(pwd)" -o "$(pwd)"
+   ```
+   You can also set the values for [input-](CONFIGINI.md#inputfolder) (`-i`),
+   [gamelist-](CONFIGINI.md#gamelistfolder) (`-g`) und
+   [media-folder](CONFIGINI.md#mediafolder) (`-o`) permanently in the Skyscraper
+   config file.
+4. Afterwards run the gamelist creation and media file deployment from
+   cache:
+   ```bash
+   Skyscraper -f batocera -p snes --flags videos -i "$(pwd)" -g "$(pwd)" -o
+   "$(pwd)"
+   ```
+5. Do a SSH login to your Batocera system and restart the EmulationStation
+   ```bash
+   batocera-es-swissknife --restart && emulationstation-standalone
+   ```
+   Leave the SSH shell open to reload EmulationStation with ++ctrl+c++ for
+   subsequent gamelist reloads. You can also restart the whole system, however
+   it takes longer.
+6. Navigate to the system/platform you just scraped to review the scraping
+   result.
+
+!!! tip "New to Skyscraper?"
+
+    For starters I suggest to enable (set `true`) these settings:
+    [`unattended`](CONFIGINI.md#unattend),
+    [`unattendedSktip`](CONFIGINI.md#unattendSkip) and most importantly
+    [`gameListBackup`](CONFIGINI.md#gamelistbackup). If you have precious media
+    files which you may want to keep make also a manual backup of the media folders
+    of. Also you will have to add your credentials for Screenscraper for example in
+    the [config file of Skyscraper](CONFIGINI.md#usercreds).
+
+For general advise on SSH usage see the [Batocera
+documentation](https://wiki.batocera.org/security).
+
 ### Attract-Mode
 
 -   Default game list location: `/home/<USER>/.attract/romlists`
@@ -140,4 +250,4 @@ You need to add the individual platform rom directories to Pegasus (if they are
 
 #### Metadata preservation
 
-Skyscraper will preserve any metadata key-value pairs added to the header and / or individual game list entries.
+Skyscraper will preserve any metadata key-value pairs added to the header and / or individual game list entries.

docs/IMPORT.md

@@ -10,13 +10,13 @@ The following describes how to import your own custom textual, artwork and / or
 
     Be sure to also check the [`--cache edit` option](CLIHELP.md#-cache-editnewtype).
 
-### Images, Videos and Game Manuals
+### Images, Videos, Fanart and Game Manuals
 
 To import videos or images into the resource cache, use the following procedure:
 
 -   Name your image or video file with the _exact_ base name of the rom you wish to connect it to. Example: `Bubble Bobble.nes` will import images with a filename of `Bubble Bobble.jpg` or `Bubble Bobble.png` or other well-known image formats. As long as the base name is an _exact_ match. Same goes for video files. I recommend only making use of well-known video formats since Skyscraper imports them directly without conversion (unless you [convert them](CONFIGINI.md#videoconvertcommand)), so they need to be supported directly by the frontend you plan to use.
 -   Game manuals are expected to use PDF format and have the extension `.pdf`. The base name must match the ROM file, thus the game manual of the example is `Bubble Bobble.pdf`.
--   Place all of your images, videos or game manuals in the `/home/<USER>/.skyscraper/import/<PLATFORM>/screenshots`, `covers`, `wheels`, `marquees`, `videos` or `manuals` folders.
+-   Place all of your images, fanart, videos or game manuals in the `/home/<USER>/.skyscraper/import/<PLATFORM>/screenshots`, `covers`, `wheels`, `marquees`, `fanarts`, `videos` or `manuals` folders.
 -   Now run Skyscraper with `Skyscraper -p <PLATFORM> -s import`. If you named your files correctly, they will now be imported. Look for the green 'YES' in the output at the rom(s) you've placed files for. This will tell you if it succeeded or not.
 -   The data is now imported into the resource cache. To make use of it read the section [How to actually use the data?](#how-to-actually-use-the-data) below.
 

src/abstractfrontend.h

@@ -40,7 +40,7 @@ class AbstractFrontend : public QObject {
 public:
     AbstractFrontend();
     virtual ~AbstractFrontend();
-    void setConfig(Settings *config);
+    virtual void setConfig(Settings *config);
     virtual void checkReqs(){};
     virtual void assembleList(QString &, QList<GameEntry> &){};
     virtual void skipExisting(QList<GameEntry> &, QSharedPointer<Queue>){};
@@ -58,6 +58,7 @@ class AbstractFrontend : public QObject {
     virtual QString getTexturesFolder() { return QString(); };
     virtual QString getVideosFolder() { return QString(); };
     virtual QString getManualsFolder() { return QString(); };
+    virtual QString getFanartsFolder() { return QString(); };
     virtual void sortEntries(QList<GameEntry> &gameEntries);
 
 protected:

src/abstractscraper.cpp

@@ -154,6 +154,11 @@ void AbstractScraper::populateGameEntry(GameEntry &game) {
                 getManual(game);
             }
             break;
+        case GameEntry::Elem::FANART:
+            if (config->fanart) {
+                getFanart(game);
+            }
+            break;
         default:;
         }
     }