Refactor xdelta to C module (#65)

* initial c module setup

* fix: builds

* feat: add build instructions to build a c module

* ci: checkout subprojects

* fix: build issues on windows and mac x86

* ci: try using other python 3.10

* refactor: better reference counting

* ci: change macos-13 to 15-intel

* feat(xdelta): implement cache for source buffer lookups

Author: imLinguin

.github/workflows/build.yaml

@@ -12,18 +12,23 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-24.04, ubuntu-24.04-arm, macos-13, macos-15, windows-2025]
+        os: [ubuntu-24.04, ubuntu-24.04-arm, macos-15-intel, macos-15, windows-2025]
     runs-on: ${{ matrix.os }}
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
+        with:
+          submodules: 'true'
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.9"
+          python-version: "3.10"
 
       - name: Install pyinstaller and dependencies
         run: pip3 install --upgrade pyinstaller requests
 
+      - name: Build C module
+        run: pip3 install -e . 
+
       - name: Set strip on Linux and Mac
         id: strip
         run: echo "option=--strip" >> $GITHUB_OUTPUT

.gitignore

@@ -2,4 +2,7 @@ build
 dist
 *.egg-info
 *.spec
-__pycache__
+__pycache__
+env
+*.so
+*.dll

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "xdelta3"]
+	path = xdelta3
+	url = git@github.com:jmacd/xdelta.git
+	branch = release3_0_apl

README.md

@@ -23,10 +23,13 @@ Heroic uses `$XDG_CONFIG_HOME/heroic/gog_store/auth.json`
 
 Here is the command to pull the source code
 
-```
+```bash
 git clone https://github.com/Heroic-Games-Launcher/heroic-gogdl
 cd heroic-gogdl
-./bin/gogdl --help
+python -m venv venv
+. venv/bin/activate
+pip install .
+gogdl --help
 ```
 
 If you have any questions ask on our [Discord](https://discord.com/invite/rHJ2uqdquK) or through GitHub issue
@@ -37,13 +40,14 @@ If you wish to test the gogdl in Heroic flatpak you likely need to build `gogdl`
 
 - Get pyinstaller
 
-```
+```bash
 pip install pyinstaller
 ```
 
 - Build the binary (assuming you are in heroic-gogdl direcory)
 
-```
+```bash
+pip install -e . # Ensure you build the C code to python module
 pyinstaller --onefile --name gogdl gogdl/cli.py
 ```
 

bin/gogdl

@@ -535,6 +535,7 @@ def interrupt_shutdown(self):
     def shutdown(self):
         self.logger.debug("Stopping progressbar")
         self.progress.completed = True
+        self.progress.join()
 
         # Clear speed queues
         for q in [self.download_speed_updates, self.writer_speed_updates]:

gogdl/dl/managers/task_executor.py

@@ -17,7 +17,7 @@
 from enum import Enum, auto
 from multiprocessing import Process, Queue
 from gogdl.dl.objects.generic import MemorySegment, TaskFlag, TerminateWorker
-from gogdl.xdelta import patcher
+import gogdl.xdelta3
 
 
 class FailReason(Enum):
@@ -357,8 +357,7 @@ def run(self):
                     patch = os.path.join(task.destination, task.patch_file)
                     patch = dl_utils.get_case_insensitive_name(patch)
                     target = task_path
-
-                    patcher.patch(source, patch, target, self.speed_queue)
+                    gogdl.xdelta3.patch(source, patch, target, self.speed_queue)
 
                 except Exception as e:
                     print("Patch failed", e)