Migrate from libmpq to mpqfs

[AJenbo]

mpqfs is a minimal MIT-licensed MPQ v1 reader/writer that replaces both libmpq and PKWare in a single dependency.

We use mpqfs from: https://github.com/diasurgical/mpqfs

Impact:

• DevilutionX is now 2,149 lines lighter (25 files changed, 184 insertions, 2,333 deletions, mpqfs is 2643 lines and libmpq is ~2500).
• The entire block-level SDL_RWops streaming implementation (~300 lines of manual seek/read/block management) is replaced by a single mpqfs_open_rwops() call.
• File-number plumbing removed throughout, mpqfs uses filenames directly, simplifying AssetRef, FindAsset, and all call sites.
• The callback-based PKWare implode()/explode() API is replaced with simple buffer-to-buffer calls.
• Thread-safe audio streaming is now a single function call instead of manual archive cloning + block offset table management.

P.s. despite the angry CI this does work on my machine ™️

I need some sleep before I can figure out the CI issue.

P.s.s. should I strip out the write feature or expand it and also replace that part of DevilutionX with mpqfs?

Update: Builds should work now, the intergration test hash is updated since the different compression algo produces a different but valid data stream.

[glebm]

Amazing! How is the performance?

libmpq is pretty fast and reuses the decompression buffer (per thread).

File-number plumbing removed throughout

Some of the file-number stuff was intentional to avoid computing the number multiple times, though I'm not sure how important that micro-optimization really is.

[AJenbo]

Amazing! How is the performance?

libmpq is pretty fast and reuses the decompression buffer (per thread).

File-number plumbing removed throughout

Some of the file-number stuff was intentional to avoid computing the number multiple times, though I'm not sure how important that micro-optimization really is.

Yeah from what I saw it should make very little difference, I could look into reintroducing it, but it would probably make the interface a lot less clean

[AJenbo]

@glebm applied some optimizations to mpqfs (reuse compression buffer) and things should be ready for review now.

[glebm]

Yeah from what I saw it should make very little difference, I could look into reintroducing it, but it would probably make the interface a lot less clean

I think it'd just require adding a has and open functions that accept a file number?

MPQFS_API bool mpqfs_has_file_hash(mpqfs_archive_t *archive, uint32_t hash);
MPQFS_API SDL_IOStream *mpqfs_open_io_from_hash(mpqfs_archive_t *archive, uint32_t hash);
// etc...

[AJenbo]

Alright I think I got the optimizations working

[AJenbo]

I'm a little unsure about disabeling compression support on OG Xbox, but hopefully it won't need it for the base game.

[glebm]

Can you please also update:

DevilutionX/tools/make_src_dist.py

Line 40 in 69e5632

"libmpq",

[glebm]

Looking at the writer implementation in mpqfs, it appears to keep all of the archive contents in RAM, which can increase peak RAM usage (save files can be > 1 MiB).

DevilutionX implementation is leaner -- it writes the files directly to the archive as they get added.

[AJenbo]

Yeah mpqfs write implementation is currently just good enough to produce valid saves, but not good enough to be used. I was wondering if I should just strip it out but ideally I would prefer to move all MPQ support to mpqfs rather then having an internal implementation for writing like we currently do.

I'll see about fixing compression on Xbox and making the write support ready to replace ours. Thanks for going over the code.

[glebm]

Makes sense. I also think that it might be cleaner to remove the SDL stuff from mpqfs.

This is because multiple SDL versions are supported, which might necessitate multiple mpqfs packages (mpqfs-sdl2, mpqfs-sdl3, mpqfs-no-sdl, etc), if mpqfs is ever used by projects other than DevilutionX.

[AJenbo]

Yeah I did consider that might not be ideal if it was to be used by a save game viewer. I didn't realize it could also affect how packaging works. The intent was to make it's interface to us as simple as possible.

[AJenbo]

Amazing! How is the performance?

CPU Results (spawn.mpq, Release build)

Operation	libmpq	mpqfs	Speedup
hash_string (×4)	0.080 µs	0.058 µs	1.37×
file_hash (triple)	0.054 µs	0.039 µs	1.38×
encrypt+decrypt 4K	2.339 µs	2.319 µs	1.01×
encrypt+decrypt 64K	37.629 µs	37.481 µs	1.00×
open+close	27.1 µs	26.5 µs	1.02×
file lookup	0.047 µs	0.037 µs	1.28×
file read	312 µs	313 µs	1.00×
bulk read (1028 files)	164 ms	165 ms	1.00×

Memory Results

Metric	libmpq	mpqfs	Improvement
Total on open	66.3 KiB	48.4 KiB	27% less
Per-file transient overhead	~91 KiB	~8 KiB	11× less
Bulk extraction peak RSS delta	1.37 MiB	776 KiB	44% less
Memory scaling (65K entries)	3200 KiB	2048 KiB	1.56×

[AJenbo]

• mpqfs now supports streaming data.
• mpqfs is now used for write operations as well as read.
• fix xbox compression
• mkdist has been updated
• Compression and decompression now use stack buffers (~8 KB) to avoid heap allocations on the hot path, falling back to heap for larger payloads.
• When a save file is fully rewritten it rotates a .tmp, this should be safer then what we did before.