You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/CACHE.md
+21-4Lines changed: 21 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,17 +4,34 @@ Whenever you scrape any platform with any scraping module, Skyscraper caches eac
4
4
5
5
Think of the resource cache as the cache in an internet browser. Most of the data on webpages don't change much. By caching some of the data locally, it can load parts of a webpage from that data instead of loading it from the remote server. This is exactly how the Skyscraper cache works. It helps keep the online servers healthy by not hammering them whenever you need resources you already downloaded once. And it allows you to re-generate the frontend game lists if you add new games or perhaps want to change the style of the exported artwork.
6
6
7
-
**Default resource cache folder**
7
+
### Default resource cache folder
8
8
9
9
The default base folder for all of Skyscrapers' locally cached data is in the `/home/<USER>/.skyscraper/cache` folder. In this folder you'll find individual platform subfolders. Any of these are selfcontained and contains all of the cached data for that particular platform.
10
10
11
-
**Resource and scraping module priorities**
11
+
### Resource and scraping module priorities
12
12
13
-
There is ONE file that you can and should edit inside each of the `/home/<USER>/.skyscraper/cache/<PLATFORM>` folders. That file is called `priorities.xml` and decides the scraper priority of resources for each resource type. For instance, if you know that `thegamesdb` always provides the best `descriptions` for games, you'd add an `<order type="description">` node with a `<source>thegamesdb</source>` subnode. You can have multiple `<source>` nodes, Skyscraper will then prefer the topmost source when generating a game list. If the topmost isn't found it'll prioritize the next one and so forth. Any source that isn't listed with an `<order>` node will be prioritized using timestamps (newest wins) for when each resource was added to the cache. So you don't _have_ to add all of them.
13
+
There is ONE file that you can and should edit inside each of the `/home/<USER>/.skyscraper/cache/<PLATFORM>` folders. That file is called `priorities.xml` and decides the scraper priority of resources for each resource type. Here is the part of that file:
14
+
15
+
```xml
16
+
<?xml version="1.0" encoding="UTF-8"?>
17
+
<priorities>
18
+
[...]
19
+
<ordertype="description">
20
+
<source>import</source>
21
+
<source>esgamelist</source>
22
+
<source>mobygames</source>
23
+
<source>thegamesdb</source>
24
+
<source>screenscraper</source>
25
+
<source>openretro</source>
26
+
</order>
27
+
[...]
28
+
```
29
+
30
+
For instance, if you know that `thegamesdb` always provides the best `descriptions` for games, you'd add an `<order type="description">` node with a `<source>thegamesdb</source>` subnode. You can have multiple `<source>` nodes, Skyscraper will then prefer the topmost source when generating a game list. If the topmost isn't found it'll prioritize the next one and so forth. Any source that isn't listed with an `<order>` node will be prioritized using timestamps (newest wins) for when each resource was added to the cache. So you don't _have_ to add all of them.
14
31
15
32
Skyscraper provides the example file `/home/<USER>/.skyscraper/cache/priorities.xml.example`. Please don't edit this file manually, as it will be overwritten when you update Skyscraper. When a platform is scraped for the first time, it will automatically copy the example file to `/home/<USER>/.skyscraper/cache/<PLATFORM>/priorities.xml` unless it already exists. You can of course also copy the file yourself before scraping a platform. If you do so, be sure to remove the `.example` part of the filename so it's just called `priorities.xml`.
16
33
17
-
**Update locally cached data**
34
+
### Update locally cached data
18
35
19
36
If you wish to update / refresh the locally cached resources for a particular platform and scraping module, Skyscraper provides the `--refresh` option. If this flag is set on the command line, any data in the resource cache will be updated with the new incoming data.
0 commit comments