Changed: Update Sewer56.Update with More Efficient File Size Check & User-Agent fix.

The launcher now consumes the updated Sewer56.Update library (4.1.0) which replaces GET-for-size probes with HEAD requests and sets a User-Agent on all HTTP traffic.  Launcher-side wiring:

- SharedHttpClient: every static HttpClient now sends the library's ApplicationUserAgent header.
- WebDownloadablePackage: GetNameAndSize uses HEAD instead of GET (no body abandoned).  DownloadAsync HttpClient also gets a User-Agent.
- Startup.cs (pack download): HttpClient now carries a User-Agent.
- App.xaml.cs: sets HttpEx.ApplicationUserAgent = "Reloaded-II/<version>" at the very start of OnStartup, before any HTTP request can be issued.

Package bumps:
  Sewer56.Update              4.0.2 -> 4.1.0
  Sewer56.Update.Resolvers.GitHub     1.5.2 -> 1.6.0
  Sewer56.Update.Resolvers.NuGet      1.4.1 -> 1.5.0
  Sewer56.Update.Resolvers.GameBanana 1.4.2 -> 1.5.0

Author: Sewer56

changelog-template.hbs

@@ -1,11 +1,11 @@
 [Read and Discuss in a Browser](https://github.com/Reloaded-Project/Reloaded-II/discussions/473).
 [Previous Changelog](https://github.com/Reloaded-Project/Reloaded-II/releases/tag/1.30.2).
 
-Just a tiny maintenance update. My health hasn't been the best lately- because
-I haven't been taking good care of myself. So this is all I can give for now in
-some spare time.
+Just a tiny maintenance update that took around ~12h of work over the weekend.
+My health hasn't been the best lately- because I haven't been taking good care of myself.
+So this is all I can give for now in some spare time.
 
-# 1.30.3: Startup Freeze Fix & Steam DRM Docs
+# 1.30.3: Startup Freeze Fix, Network Hammering Fix & Steam DRM Docs
 
 ## Fix Launcher Freezing on Startup by @drewmcelhany
 
@@ -31,8 +31,79 @@ Also hardened the surrounding code while at it:
 Fixes [#910](https://github.com/Reloaded-Project/Reloaded-II/issues/910).
 
 (Note: The actual cause is my NuGet server being slow as of late; while gobbling
-up gigabytes of RAM. I'm currently investigating why. Regardless, the launcher
-should never freeze over a slow server; so the fix applies universally.)
+up gigabytes of RAM. I've been investigating why.)
+
+## Fix Server Slowdown / Self-Inflicted DDoS by @Sewer56
+
+Found the cause of the slow server mentioned above.
+
+It was me; to be more exact; a multitude of different issues all piling up:
+
+- Launcher started full file downloads just to get their size
+- Launcher didn't send a `User-Agent` header
+- Server wrote to the database on every download
+- Server re-ran search queries that rarely change
+- Server let clients request unbounded result sets
+
+### Launcher Downloading Files to Get Their Size
+
+When checking for mod updates; the launcher needs the file size of each
+available update.
+
+To get it, the launcher sent a full `GET` request; which started downloading the file,
+and then immediately threw it away. Every user, on every launch, for every
+available update.
+
+Multiply that by enough users at once; and we've basically DDoS'd ourselves.
+Hence the slowdown, and the RAM gobbling.
+
+Things got worse when I added HTTPS support via a Caddy reverse proxy late last
+year; since abandoned downloads now kept a TLS connection open longer per
+request.
+
+The size check now issues a `HEAD` request; which gets the file size from headers
+alone- no body download.
+
+### Launcher Not Sending a `User-Agent`
+
+The launcher wasn't sending a `User-Agent` header on any of its HTTP requests.
+
+It now does; so I can distinguish scrapers more easily.
+
+The rest of the issues were server-side. The [server itself](https://github.com/Sewer56/BaGet-ReloadedII)
+was also not built for the kind of traffic Reloaded gets; so I fixed that too.
+Original BaGet was mostly for smaller deployments.
+
+### Batch Download Counts
+
+Every package download ran a database write before returning the file.
+
+These writes fought over the same database lock; queuing everything up.
+
+Download counts (which BaGet doesn't even properly track in the first place) are
+now buffered in memory and flushed every 5 seconds.
+
+In local testing this pushed throughput from ~1,100 to ~4,700 req/s; and cut
+worst-case latency from 640ms down to 37ms.
+
+### Cache Search and Autocomplete
+
+Search and autocomplete hit the database fresh on every request.
+These are now cached in memory.
+
+In local testing this pushed throughput from ~1,470 to ~4,640 req/s; and cut
+worst-case latency from 51ms down to 19ms. Results may be stale for up to 2
+minutes; but package listings don't change often enough to notice.
+
+### Cap Search Page Size & Fix Total Hit Count
+
+The search endpoint accepted any amount of user requested results; so a request
+like `take=100000` would pull a massive result set.
+
+The page size is now capped at 1000; well above what any legitimate client uses.
+
+Also fixed; the `totalHits` field was reporting the page size instead of the
+actual total number of matching packages.
 
 ## Document Steam DRM Debugging Issues by @nenkai & @Sewer56
 

source/Reloaded.Mod.Launcher.Lib/Startup.cs

@@ -163,6 +163,7 @@ private static void DownloadAndOpenPackAndExit(string downloadUrl)
             downloadUrl = downloadUrl.Substring(Constants.ReloadedPackProtocol.Length + 1);
 
         using var httpClient = new HttpClient();
+        httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(HttpEx.ApplicationUserAgent);
         var file = new MemoryStream(Task.Run(() => httpClient.GetByteArrayAsync(downloadUrl)).Result);
         var config = IoC.Get<LoaderConfig>();
         Actions.ShowInstallModPackDialog(new InstallModPackDialogViewModel(new ReloadedPackReader(file), config, new AggregateNugetRepository(config.NuGetFeeds)));

source/Reloaded.Mod.Launcher.Lib/Usings.cs

@@ -54,6 +54,7 @@
 global using Reloaded.Mod.Shared;
 global using SevenZip;
 global using Sewer56.Update;
+global using Sewer56.Update.Http;
 global using Sewer56.Update.Extractors.SevenZipSharp;
 global using Sewer56.Update.Misc;
 global using Sewer56.Update.Packaging.Interfaces;

source/Reloaded.Mod.Launcher/App.xaml.cs

@@ -2,6 +2,7 @@
 using Environment = Reloaded.Mod.Shared.Environment;
 using Path = System.IO.Path;
 using Paths = Reloaded.Mod.Loader.IO.Paths;
+using Sewer56.Update.Http;
 
 namespace Reloaded.Mod.Launcher;
 
@@ -17,6 +18,11 @@ public partial class App : Application
 
     private void OnStartup(object sender, StartupEventArgs e)
     {
+        // Identify our HTTP traffic server-side. Set before any HTTP request is made
+        // (command-line download handlers may issue requests below).
+        var ver = typeof(App).Assembly.GetName().Version!;
+        HttpEx.ApplicationUserAgent = $"Reloaded-II/{ver.Major}.{ver.Minor}.{ver.Build}";
+
         // Run update handler.
         if (Sewer56.Update.Hooks.Startup.HandleCommandLineArgs(GetCommandLineArgs()))
         {

source/Reloaded.Mod.Loader.Update.Packaging/Reloaded.Mod.Loader.Update.Packaging.csproj

@@ -26,11 +26,11 @@
 
   <ItemGroup>
     <PackageReference Include="DotNet.ReproducibleBuilds" Version="1.2.39" PrivateAssets="All" />
-    <PackageReference Include="Sewer56.Update" Version="4.0.2" />
+    <PackageReference Include="Sewer56.Update" Version="4.1.0" />
     <PackageReference Include="Sewer56.Update.Extractors.SevenZipSharp" Version="1.1.4" GeneratePathProperty="true" />
     <PackageReference Include="Sewer56.Update.Packaging" Version="3.0.1" />
-    <PackageReference Include="Sewer56.Update.Resolvers.NuGet" Version="1.4.1" />
-    <PackageReference Include="Sewer56.Update.Resolvers.GameBanana" Version="1.4.2" />
+    <PackageReference Include="Sewer56.Update.Resolvers.NuGet" Version="1.5.0" />
+    <PackageReference Include="Sewer56.Update.Resolvers.GameBanana" Version="1.5.0" />
     <PackageReference Include="System.Formats.Asn1" Version="10.0.0" />
   </ItemGroup>
 

source/Reloaded.Mod.Loader.Update/Providers/Web/WebDownloadablePackage.cs

@@ -111,6 +111,7 @@ public async Task<string> DownloadAsync(string packageFolder, IProgress<double>?
 
         // Start the modification download.
         using var httpClient = new HttpClient();
+        httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(HttpEx.ApplicationUserAgent);
         var downloadProgress = progressSlicer.Slice(0.9);
 
         await retryPolicy.ExecuteAsync(async () =>
@@ -159,10 +160,12 @@ public async static Task<string> CopyPackagesFromExtractFolderToTargetDir(string
 
     private async Task GetNameAndSize(Uri url)
     {
-        // Obtain the name of the file.
+        // Obtain the name and size of the file via HEAD (headers only, no body download).
         try
         {
             var fileReq = WebRequest.CreateHttp(url);
+            HttpEx.ApplyUserAgent(fileReq);
+            fileReq.Method = "HEAD";
             using var fileResp = await fileReq.GetResponseAsync();
 
             try

source/Reloaded.Mod.Loader.Update/Reloaded.Mod.Loader.Update.csproj

@@ -41,9 +41,9 @@
     </PackageReference>
     <PackageReference Include="RedistributableChecker" Version="0.2.3" />
     <PackageReference Include="ReverseMarkdown" Version="4.7.1" />
-    <PackageReference Include="Sewer56.Update" Version="4.0.2" />
+    <PackageReference Include="Sewer56.Update" Version="4.1.0" />
     <PackageReference Include="Sewer56.Update.Misc" Version="1.1.0" />
-    <PackageReference Include="Sewer56.Update.Resolvers.GitHub" Version="1.5.2" />
+    <PackageReference Include="Sewer56.Update.Resolvers.GitHub" Version="1.6.0" />
     <PackageReference Include="System.Formats.Asn1" Version="10.0.0" />
     <PackageReference Include="System.Net.Http" Version="4.3.4" />
     <PackageReference Include="System.Text.RegularExpressions" Version="4.3.1" />

source/Reloaded.Mod.Loader.Update/Usings.cs

@@ -48,6 +48,7 @@
 global using ReverseMarkdown;
 global using Sewer56.DeltaPatchGenerator.Lib.Utility;
 global using Sewer56.Update;
+global using Sewer56.Update.Http;
 global using Sewer56.Update.Extractors.SevenZipSharp;
 global using Sewer56.Update.Interfaces.Extensions;
 global using Sewer56.Update.Misc;

source/Reloaded.Mod.Loader.Update/Utilities/SharedHttpClient.cs

@@ -23,6 +23,7 @@ public static HttpClient CachedAndCompressed
             {
                 AutomaticDecompression = DecompressionMethods.All
             });
+            _cachedAndCompressed.DefaultRequestHeaders.UserAgent.ParseAdd(HttpEx.ApplicationUserAgent);
 
             return _cachedAndCompressed;
         }
@@ -39,6 +40,7 @@ public static HttpClient Cached
                 return _cached;
 
             _cached = AkavacheWebCacheStore.Instance.CreateClient();
+            _cached.DefaultRequestHeaders.UserAgent.ParseAdd(HttpEx.ApplicationUserAgent);
             return _cached;
         }
     }
@@ -57,6 +59,7 @@ public static HttpClient UncachedAndCompressed
             {
                 AutomaticDecompression = DecompressionMethods.All
             });
+            _uncachedAndCompressed.DefaultRequestHeaders.UserAgent.ParseAdd(HttpEx.ApplicationUserAgent);
 
             return _uncachedAndCompressed;
         }