[Tracking] Initial Blizzard downloads not running at line rate - aka increasing slice size

[unspec]

Issue to track the reoccuring issue whereby initial Blizzard downloads do not run at max internet line rate.

We have seen, and can reproduce, the issue whereby downloading a Blizzard game for the first time (i.e currently uncached) is at slower than expected speed vs downloading the same game without using lancache.

[MathewBurnett]

Notes from our testing ... tl&dr is that the blizzard client some times behaves poorly with the chunking and can also give a 429 asking for a backoff. We came to the conclusion that 8m slices are a nice compromise.

Standard (mono)

approx 35+ Mbps
time 14m:28s

==> /cache/logs/error.log <==
2020/02/23 15:00:37 [error] 1679#1679: *1756 unexpected status code 429 in slice response while reading response header from upstream, client: 10.20.70.248, server: , request: "GET /tpr/hs/data/a2/ad/a2ad7088ab6d1ccfecf6374877b55c7f HTTP/1.1", subrequest: "/tpr/hs/data/a2/ad/a2ad7088ab6d1ccfecf6374877b55c7f", upstream: "http://137.221.64.3:80/tpr/hs/data/a2/ad/a2ad7088ab6d1ccfecf6374877b55c7f", host: "eu.cdn.blizzard.com"

No Cache

approx 65 Mbps (line speed)
time 8m:45

Generic

approx 35+ Mbps
time 14m:18 (cached: 1m)

Generic - Slice size 2m (kept 1m)

approx 42+ Mbps
time 11:34 (12.5)

Generic - Slice size 4m

approx 47+ Mbps
time 10:52

Generic - Slice size 8m

approx 55+ Mbps
time 9:45 (cached: 1m)

==> /cache/logs/error.log <==
2020/02/23 16:22:38 [error] 672#672: *133 unexpected status code 429 in slice response while reading response header from upstream, client: 10.20.70.248, server: , request: "GET /tpr/hs/data/6e/e0/6ee00cbad15beb01ea2073fa6c869018 HTTP/1.1", subrequest: "/tpr/hs/data/6e/e0/6ee00cbad15beb01ea2073fa6c869018", upstream: "http://137.221.64.2:80/tpr/hs/data/6e/e0/6ee00cbad15beb01ea2073fa6c869018", host: "eu.cdn.blizzard.com"

-> steam @ 60+
-> origin @ 60+`

[unspec]

It's apparent that a modest increase in slice size can help blizzard downloads in particular.

Please be aware that there are a number of potential downsides to increasing the slice size, that should be fully understood before making changes to it.

In particular in a multi-user environment (like most LAN parties) where more than one person may be trying to download the same game, and thus the same multi-gig files, the performance might be reduced. The usual behaviour is if somebody tries to download a slice already being downloaded by somebody else, the second person will be forced to wait until that slice has fully downloaded before being served with the slice. If you have a relatively slow internet connection (or its very highly utilised - like at most LAN parties) then the time to download a larger slice will be high and the resulting wait time could cause problems for the game client.

Ultimately the performance will be a trade off and factors such as number of internet connections & speed, utilisation, number of users, size of game, game client behaviour and cache server performance (cpu, ram & in particular disk I/O) will all influence whether or not increasing the slice size will help you. Testing / LAB performance may be very different to "real world" event performance with the same hardware / settings.

We are planning a PR that will expose slice size as an environment variable in a future version.

[Lepidopterist]

In addition to the above - changing the slice size will make most of the existing data in your cache unusable, and thus will have to be downloaded again. Old data will be purged off as space is required.

Anything that was originally downloaded that is smaller than your original slice size will still be valid.
Any 'end parts' of files, that would be identical with the old and new slice size will also be valid.

For example. If you had an 8.5MB file you originally downloaded with a 1m slice size, and you change the slice size to 4m, the first 8 slices will be invalidated, but the last 0.5MB slice will still be valid. If you changed the slice size to 6MB, all pieces would be invalid.

Hopefully that's clear?

We propose to add a hash check on Monolithic startup to identify whether various cache parameters that may invalidate the cache have changed since last run and throw an error. This will then force you to run with an override environment variable to proceed.

Our longer term plan will be to default to (most likely) an 8MB slice size.

[Lepidopterist]

And as an addendum, I would anticipate that changing the slice size will have very minimal impact on Steam and Uplay as they already use files that are mainly around 1MB or less. Blizzard and Origin are potentially the CDN's that will benefit the most, as they download large files which are sliced by LanCache.

[unspec]

Added in #101