diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 0a0bce9..ba2633c 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -53,7 +53,7 @@ jobs:
if: matrix.os == 'ubuntu-latest'
run: |
python tools/mpqref.py manifest src/test/resources/mpqs \
- --names src/main/resources/DefaultListfile.txt \
+ --names src/test/resources/DefaultListfile.txt \
-o /tmp/fixtures.tsv
diff -u src/test/resources/golden/fixtures.tsv /tmp/fixtures.tsv
@@ -65,8 +65,15 @@ jobs:
path: build/reports/tests/test
retention-days: 7
+ # Telemetry, not a gate. continue-on-error as well as fail-on-error:
+ # the latter only covers the upload, so an outage while the action fetches
+ # its own reporter binary still failed the job -- which it did, with a 504
+ # from coveralls.io, on a commit whose build, tests, reference checks and
+ # manifest check had all passed. A third-party reporting service being
+ # down is not a reason to call this build broken.
- name: Report coverage
if: matrix.os == 'ubuntu-latest'
+ continue-on-error: true
uses: coverallsapp/github-action@v2
with:
file: build/reports/jacoco/test/jacocoTestReport.xml
diff --git a/AUDIT.md b/AUDIT.md
index 6d3aaee..035e3e9 100644
--- a/AUDIT.md
+++ b/AUDIT.md
@@ -87,7 +87,9 @@ These are real bugs or hazards in behaviour that must not be carried into the ne
- **P2-3 Sector CRC flag (0x04000000).** Verify per-sector ADLER/CRC on read when flag present; option to emit on write. Wire into extraction pipeline, not bolted onto `MpqFile`.
- Done, in `MpqFileReader` on read and `MpqSectorWriter` on write, opt-in via `MpqWriteOptions.withSectorChecksums`. The checksums are Adler-32 **seeded with zero** -- not CRC32, and not standard Adler-32; see format notes 9 and 10, which also record what that distinction cost. Verification is on by default when reading and can be turned off to recover damaged archives.
- **P2-4 `(attributes)` write support** *(issue [#11](https://github.com/inwc3/JMPQ3/issues/11))*. Honour the attributes bytemask properly on read (today hardcodes crc+timestamp layout and has a suspicious `-1` in the entry count — AttributesFile.java:38); regenerate CRC32+FILETIME on write when requested. Remove the dead commented block in `JMpqEditor.close`. Historical context from the issue thread: generation was disabled because CRC32 differed from StormLib for some `.wav` files — root cause is likely multi-compression handling (first sector of ADPCM-compressed wavs is not ADPCM-compressed since it holds the wav header). Fix alongside P2-6 and pin with a StormLib-golden CRC test. Acceptance for the issue itself: load + close `war3.mpq`-style archives without dropping `(attributes)` in a way the game rejects.
- - Done. `MpqAttributes` reads every array the bytemask declares and accepts the entry counts StormLib tolerates; the unexplained `-1` is gone from the deprecated parser too. Generation is opt-in via `MpqWriteOptions.withAttributes`, with a pinnable timestamp so builds stay reproducible. The `.wav` CRC32 concern from the issue thread does not arise: the checksum is taken over decoded content, and the multi-compression ordering it depended on was fixed in P2-6.
+ - Done. `MpqAttributes` reads every array the bytemask declares and accepts the entry counts StormLib tolerates; the unexplained `-1` is gone from the deprecated parser too. Generation is opt-in via `MpqWriteOptions.withAttributes`, with a pinnable timestamp so builds stay reproducible.
+ - **Verified.** `tools/mpqref.py` now implements Huffman and ADPCM, transcribed from StormLib's `huff.cpp` and `adpcm.cpp` rather than from the Java, and decodes `wavTest.w3x`'s `ReviveNightElf.wav` to the same MD5 as JMPQ3 does: `6b131014f093fca5972bfc1a0477f1b1` over 144464 bytes. The file is a textbook instance of the issue thread's concern -- sector 0 is plain zlib because it carries the RIFF header, and the remaining 34 sectors are `0x41`, Huffman over ADPCM mono. The committed golden manifest now holds that digest, so `GoldenFileTests` checks the wav decode on every run instead of skipping it.
+ - Superseded note, kept for the reasoning: the argument used to be that the checksum is taken over decoded content, so it could only differ from StormLib if the decode differed, and the multi-compression ordering that would cause that was made table-driven in P2-6. The argument was right, but it was an argument.
- **P2-5a Tolerant header parsing for real-world (protected) maps** *(issue [#46](https://github.com/inwc3/JMPQ3/issues/46))*. 59/857 sampled maps fail with "Bad header size": `readHeaderSize` hard-rejects `headerSize < 32 || > 208` (JMpqEditor.java:443) even though the game itself ignores the field for v0 archives. Mirror StormLib's leniency: derive the effective header size from the format version, clamp/ignore garbage values, and treat other header fields defensively (this also removes most of the need for consumers to pass `FORCE_V0`). Acceptance: the Forest Defense sample from the issue opens and extracts.
- Done in Phase 0, as a side effect of modelling the header: `MpqHeader` repairs rather than rejects, following `ConvertMpqHeaderToFormat4`.
- **P2-5b Fake-header protection resilience — nice to have** *(issue [#47](https://github.com/inwc3/JMPQ3/issues/47))*. Some protected maps plant decoy `MPQ\x1A` headers so `searchHeader` either accepts a bogus one or gives up. Approach: on finding a candidate header, validate it (plausible table positions/sizes within file) and keep scanning on failure instead of committing to the first match. Best-effort only — full protected-map support is explicitly not a goal; skip if it destabilises normal parsing.
@@ -98,24 +100,34 @@ These are real bugs or hazards in behaviour that must not be carried into the ne
## Phase 3 — Code hygiene & dependencies
- **P3-1 Logging.** `logback-classic` + `logback.xml` ship in the library's runtime deps/resources and hijack consumers' logging config. Keep `slf4j-api` only; move logback + config to `testRuntimeOnly`. Remove `DebugHelper.appendData`'s `printStackTrace`.
+ - Done. `logback-classic` is `testRuntimeOnly`, no `logback.xml` ships, and `DebugHelper.appendData` raises `UncheckedIOException` instead of printing a stack trace.
- **P3-2 Dependency audit.** `commons-compress` used only for `SeekableInMemoryByteChannel` (trivially self-implemented) — and will be needed for BZIP2 (P2-6), so decide once. Evaluate replacing unmaintained `jzlib` with `java.util.zip` (`Deflater`/`Inflater`) — the hand-rolled `zlibStoreLevel0` in CompressionUtil duplicates jzlib level-0 anyway; benchmark before/after. `xz` currently unused (see P2-6).
+ - Done. `jzlib` removed: `java.util.zip.Deflater`/`Inflater` do the same job through the JDK's bundled zlib, which is native and maintained. Verified byte-identical output on the same inputs before removing the dependency, so archives are unchanged. `commons-compress` stays -- it is what decodes BZIP2 sectors, so the P2-6 question the audit wanted decided once is decided: keep. `xz` is now used, by LZMA.
+ - Also removed `ZlibStore`, whose only caller discarded its output. `compress` with no recompression built a zlib stream of stored blocks -- necessarily larger than its input -- so every sector paid for a copy and an Adler-32 to produce something the caller always rejected by its own "did it shrink" test. It returns `null` now and the caller stores raw, byte for byte as before. Its hand-rolled Adler-32 also carried the signed-overflow bug found in P2-3.
- **P3-3 Delete dead/duplicated code.** Commented-out `loadDefaultListFile` and attributes block; near-identical sector loops `extractCompressedBlock` vs `extractImplodedBlock` (MpqFile.java:94,150); triple keygen duplication (P1-4); `Either` union class → sealed interface or two-field record; unused `DefaultListfile.txt` decision (resource shipped but load path commented out).
+ - Done. `LinkedIdentityHashMap`, `GrowingBuffer`, `Either` and `ZlibStore` are gone; the commented-out listfile and attributes blocks with them. `DefaultListfile.txt` was decided rather than left: it is test-only, so it moved to `src/test/resources` and stopped adding a megabyte to the published jar.
- **P3-4 Java 25 modernisation.** Records for `Block`/header/bucket models, sealed types where useful, **pattern matching for switch + record patterns** (final since 21) for compression dispatch and per-version header handling, `SequencedCollection` for the ordered file maps, FFM `MemorySegment`/`Arena` in the read layer (P1-3). Replace the `ThreadLocal` `STORE_BUFFER` in CompressionUtil with per-call allocation or a `ScopedValue`. The Vector API stays **out** (still incubator in 25 — not acceptable for a library). Set toolchain and `options.release` to 25.
+ - Done across the phases: records for the header, entry and options models, sealed `Content` hierarchy, pattern-matched switches for compression dispatch, `SequencedMap` for the ordered name maps, FFM `MemorySegment`/`Arena` in the read layer, and the `ThreadLocal` scratch buffer gone. Toolchain and `options.release` are 25. The Vector API stays out, as the audit requires.
- **P3-5 Naming/typos sweep.** `getAllVaildBlocks`, "Invaild block position", `DegugHelperTests`, `FLAG_LMZA`, javadoc stubs like "the fc" / "the b" / auto-generated noise. Full javadoc on the new public API.
+ - Done. `getAllVaildBlocks` is deprecated in favour of `getAllValidBlocks`, kept for binary compatibility; `DegugHelperTests` renamed; `FLAG_LMZA` survives only as prose describing the old bug. Full javadoc on the new public API.
## Phase 4 — Tests & CI
- **P4-1 Golden-file round-trip suite.** For each supported version/flag combo: StormLib-generated fixture → open → extract-all → compare hashes; rebuild → reopen with both jmpq3 and (in CI, optionally) StormLib CLI → compare. Today `testRebuild`/`testRecompressBuild` assert nothing (MpqTests.java:161,215).
- **P4-2 Fix test infrastructure.** `getResource().getFile()` breaks on paths with spaces and inside jars — copy resources to a temp dir via streams. Tests currently mutate files inside `build/resources` and litter `out/` in the working dir. Remove `System.out.println`.
- **P4-3a Bound decompression output allocations** *(found during Phase 0 self-review)*. Header and table validation now rejects implausible table geometry, and the write path no longer preallocates from a header-supplied archive size. One vector remains: a block may declare an arbitrary `normalSize`, and each decompressor allocates its expected output size up front, so a small crafted archive can still force large per-sector allocations. Fixing it properly means having the codecs grow their output instead of preallocating, which wants the Jazzer fuzz harness to validate it — hence grouped here with P4-3 rather than done blind in Phase 0.
+- **P4-3b ADPCM/Huffman verification — done** *(found while auditing issue [#11](https://github.com/inwc3/JMPQ3/issues/11))*. The fixture turned out to already exist: `wavTest.w3x` carries a 144 KB wav whose sectors are `0x41`, Huffman over ADPCM mono. What was missing was a second opinion, so `tools/mpqcodecs.py` implements both codecs from StormLib's C, and both implementations agree byte for byte. PKWARE remains the one codec the reference cannot read; JMPQ3's writer never emits it, so round-trip coverage is unaffected.
- **P4-3 New coverage needed.** Concurrency (two archives in parallel — pins P0-2), locale API, encrypted-file round-trip (incl. ADJUSTED key), sector CRC, v2–v4 fixtures, malformed-archive rejection (pins P0-8), empty file, file > one sector exactly at boundary, listfile ordering determinism.
- **P4-4 CI.** Only `gradle-publish.yml` exists (runs on release). Add a build+test workflow on push/PR (Windows + Linux matrix — path handling differs), publish jacoco report. Consider migrating TestNG → JUnit 5 while tests are being reworked (low priority, do only if touching most tests anyway).
## Phase 5 — Docs & packaging
- **P5-1 README.** There is none. Cover: what/why, quick-start for new API, migration table `JMpqEditor` → new API, supported format matrix (read/write per version/feature), thread-safety statement.
+ - Done. Covers the format and feature matrix, the new API, integrity, thread safety and the limitations. `ReadmeExampleTests` compiles and runs every snippet in it, because the previous readme had drifted into claiming sparse and bzip2 were unsupported and that `(attributes)` could not be generated.
- **P5-2 Publishing coordinates.** `group 'systems.crigges'` vs publication `groupId 'inwc3'` inconsistency; version bump to 2.0.0 with the new API; verify jitpack.yml still matches the Java 25 toolchain.
+ - Done. `group` and the publication `groupId` agree on `org.inwc3`, the version is `2.0.0-SNAPSHOT`, and `jitpack.yml` documents why `openjdk17` is still correct there: it only has to run Gradle, and the Java 25 toolchain is resolved by foojay.
- **P5-3 Format notes doc.** Short `docs/mpq-format-notes.md` recording the spec interpretations chosen (with StormLib source references) — this is what makes it a *reference* library.
+ - Done, 15 sections with StormLib citations.
- **P5-4 CLI tool — optional** *(issue [#10](https://github.com/inwc3/JMPQ3/issues/10))*. Small separate module/jar (list/extract/insert/rebuild, listfile + input/output dir options) wrapping the new API, per the consensus in the issue thread to keep it out of the library artifact. Do last; drop if time-constrained.
---
diff --git a/Readme.md b/Readme.md
index 634eaca..3962f7a 100644
--- a/Readme.md
+++ b/Readme.md
@@ -1,152 +1,228 @@
-[](https://github.com/inwc3/JMPQ3/actions/workflows/build.yml) [](https://jitpack.io/#inwc3/JMPQ3) [](https://coveralls.io/github/inwc3/JMPQ3?branch=master) [](https://codebeat.co/projects/github-com-inwc3-jmpq3-master)
-
-# JMPQ3
-
-JMPQ3 is a small Java library for reading and modifying MPQ (MoPaQ) archives.
-Common file endings are `.mpq`, `.w3m`, and `.w3x`.
-
-MoPaQ is Blizzard's older proprietary archive format for game data. It is used by Warcraft III maps and was later replaced by CASC in newer Blizzard games.
-
-JMPQ3 is primarily tested with Warcraft III maps and MPQs. Archives from other games may work, but Warcraft III compatibility is the main target.
-
-For MPQ format background and a graphical editor, see Ladislav Zezula's MPQ tools:
-http://www.zezula.net/en/mpq/main.html
-
-## Requirements
-
-JMPQ3 2.0.0 and newer requires Java 25 or newer at runtime.
-JMPQ3 1.9.x requires Java 11.
-
-## Installation
-
-JMPQ3 is available through JitPack:
-https://jitpack.io/#inwc3/JMPQ3/
-
-Gradle:
-
-```gradle
-repositories {
- maven { url 'https://jitpack.io' }
-}
-
-dependencies {
- implementation 'com.github.inwc3:JMPQ3:2.0.0'
-}
-```
-
-You can also depend on a Git commit or branch through JitPack while testing unreleased changes.
-
-## Opening Archives
-
-Use try-with-resources so the editor is closed correctly. Writable archives are rebuilt when the editor is closed.
-
-```java
-import systems.crigges.jmpq3.JMpqEditor;
-import systems.crigges.jmpq3.MPQOpenOption;
-
-import java.io.File;
-import java.nio.charset.StandardCharsets;
-
-try (JMpqEditor mpq = new JMpqEditor(new File("MyMap.w3x"), MPQOpenOption.READ_ONLY, MPQOpenOption.FORCE_V0)) {
- if (mpq.hasFile("war3map.j")) {
- byte[] script = mpq.extractFileAsBytes("war3map.j");
- System.out.println(new String(script, StandardCharsets.UTF_8));
- }
-
- for (String fileName : mpq.getFileNames()) {
- System.out.println(fileName);
- }
-}
-```
-
-`MPQOpenOption.READ_ONLY` opens the archive without modifying it.
-
-`MPQOpenOption.FORCE_V0` reads the archive like Warcraft III does, ignoring newer optional MPQ metadata where needed. This is useful for Warcraft III maps and some intentionally odd or damaged archives.
-
-## Modifying Archives
-
-Open without `READ_ONLY` to allow writes. Changes are applied when `close()` runs.
-
-```java
-import systems.crigges.jmpq3.JMpqEditor;
-import systems.crigges.jmpq3.MPQOpenOption;
-
-import java.io.File;
-
-try (JMpqEditor mpq = new JMpqEditor(new File("MyMap.w3x"), MPQOpenOption.FORCE_V0)) {
- if (mpq.hasFile("war3map.j")) {
- mpq.deleteFile("war3map.j");
- }
-
- mpq.insertFile("war3map.j", new File("build/war3map.j"));
- mpq.insertByteArray("war3mapImported/readme.txt", "generated by JMPQ3".getBytes());
-}
-```
-
-`insertFile` stores the file path and reads the file when the archive is rebuilt on close. Keep that source file alive until the editor is closed. If the data is temporary, use `insertByteArray` instead.
-
-To overwrite an existing file directly:
-
-```java
-mpq.insertFile("war3map.j", new File("build/war3map.j"), true);
-mpq.insertByteArray("war3mapImported/readme.txt", bytes, true);
-```
-
-## Creating Archives From Scratch
-
-JMPQ3 can create a minimal empty archive and then rebuild it with inserted files. This is useful for Warcraft III "folder mode" maps, where a `.w3x` directory contains the files that should become a real MPQ archive.
-
-```java
-import systems.crigges.jmpq3.JMpqEditor;
-import systems.crigges.jmpq3.MPQOpenOption;
-
-import java.io.File;
-import java.nio.file.Files;
-import java.nio.file.Path;
-
-File outputMap = new File("build/MyMap.w3x");
-JMpqEditor.createEmptyArchive(outputMap);
-
-try (JMpqEditor mpq = new JMpqEditor(outputMap, MPQOpenOption.FORCE_V0)) {
- mpq.insertFile("war3map.w3i", new File("folderMap/war3map.w3i"));
- mpq.insertFile("war3map.j", new File("folderMap/war3map.j"));
- mpq.insertFile("war3mapImported/icon.blp", new File("folderMap/war3mapImported/icon.blp"));
-
- byte[] generatedScript = Files.readAllBytes(Path.of("build/war3map.j"));
- mpq.insertByteArray("war3map.j", generatedScript, true);
-}
-```
-
-You can also get the initial empty archive bytes without writing a file:
-
-```java
-byte[] emptyArchive = JMpqEditor.createEmptyArchive();
-```
-
-Empty directories are not represented in MPQ archives, so only insert regular files.
-
-## Extracting All Known Files
-
-MPQs do not always contain a complete list of filenames. `extractAllFiles` extracts known files when a usable `(listfile)` is available. Archives without a complete listfile may still contain files that can only be accessed if you know their exact path.
-
-```java
-try (JMpqEditor mpq = new JMpqEditor(new File("MyMap.w3x"), MPQOpenOption.READ_ONLY, MPQOpenOption.FORCE_V0)) {
- mpq.extractAllFiles(new File("extracted"));
-}
-```
-
-For writable archives with a missing or incomplete listfile, you can provide an external listfile before rebuilding:
-
-```java
-try (JMpqEditor mpq = new JMpqEditor(new File("MyMap.w3x"), MPQOpenOption.FORCE_V0)) {
- mpq.setExternalListfile(new File("listfile.txt"));
- mpq.insertByteArray("war3mapImported/generated.txt", "hello".getBytes());
-}
-```
-
-## Known Issues
-
-- Unsupported decompression algorithms: sparse and bzip2.
-- Supported compression is zlib/zopfli.
-- JMPQ3 does not currently build a valid `(attributes)` file. Warcraft III maps appear to work without it.
-- Empty directories are not stored in MPQ archives.
+[](https://github.com/inwc3/JMPQ3/actions/workflows/build.yml) [](https://jitpack.io/#inwc3/JMPQ3) [](https://coveralls.io/github/inwc3/JMPQ3?branch=master)
+
+# JMPQ3
+
+JMPQ3 is a Java library for reading and writing MPQ (MoPaQ) archives. Common
+file endings are `.mpq`, `.w3m`, and `.w3x`.
+
+MoPaQ is Blizzard's older proprietary archive format for game data. It is used
+by Warcraft III maps and was later replaced by CASC in newer Blizzard games.
+
+JMPQ3 is primarily tested against Warcraft III maps. Archives from other games
+may work, but Warcraft III compatibility is the main target.
+
+Format behaviour is validated against [StormLib](https://github.com/ladislav-zezula/StormLib)
+source rather than against circulating format documents. Where the format is
+genuinely ambiguous, the interpretation chosen is recorded with its citation in
+[`docs/mpq-format-notes.md`](docs/mpq-format-notes.md).
+
+## Requirements
+
+| JMPQ3 | Java |
+|---|---|
+| 2.0.0 and newer | 25 |
+| 1.9.x | 11 |
+
+## Installation
+
+Available through [JitPack](https://jitpack.io/#inwc3/JMPQ3/):
+
+```gradle
+repositories {
+ maven { url 'https://jitpack.io' }
+}
+
+dependencies {
+ implementation 'com.github.inwc3:JMPQ3:2.0.0'
+}
+```
+
+## Format support
+
+| Version | Read | Write |
+|---|---|---|
+| v0 (32-byte header) | yes | yes |
+| v1 (44-byte header, 64-bit offsets, hi-block table) | yes | yes |
+| v2 (68-byte header) | yes | no |
+| v3 (208-byte header, compressed tables, MD5 digests) | yes | no |
+
+Versions 2 and 3 are read through the classic hash and block tables, which
+StormLib writes alongside HET/BET. **HET/BET tables themselves are not
+implemented**, so an archive that omits the classic tables cannot be opened. See
+P2-2a in [`AUDIT.md`](AUDIT.md) for why that is deferred rather than guessed at.
+
+### Features
+
+| | Read | Write |
+|---|---|---|
+| Sector checksums (`SECTOR_CRC`) | verified by default | optional |
+| `(attributes)` | parsed per its own bytemask | optional, generated |
+| `(listfile)` | yes | generated |
+| Locale variants of one path | yes | yes |
+| User data header (`MPQ\x1B`) | parsed, payload readable | not written |
+| Encrypted files, including `MPQ_FILE_KEY_V2` | yes | internal files only |
+| Signature verification / signing | no | no |
+| Patch (PTCH) archives | no | no |
+
+### Compression
+
+Decompression: zlib, PKWARE implode, BZIP2, sparse, LZMA (v2+ only), Huffman +
+ADPCM mono/stereo, and the multi-codec combinations StormLib defines. Dispatch is
+table-driven from StormLib's `dcmp_table` — note that the type byte `0x12` means
+different things depending on format version, which is why it cannot be tested as
+a bitmask. See format note 2.
+
+Compression on write is deflate, with an optional [Zopfli](https://github.com/google/zopfli)
+mode for smaller output at much greater cost.
+
+## Quick start
+
+The 2.0 API separates reading from writing. `MpqArchive` opens an archive and
+never modifies it; `MpqArchiveWriter` builds one and writes it when you say so.
+
+```java
+import org.inwc3.jmpq.MpqArchive;
+import org.inwc3.jmpq.MpqOpenOptions;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+
+try (MpqArchive archive = MpqArchive.open(Path.of("MyMap.w3x"), MpqOpenOptions.warcraft3())) {
+ if (archive.contains("war3map.j")) {
+ byte[] script = archive.read("war3map.j");
+ System.out.println(new String(script, StandardCharsets.UTF_8));
+ }
+
+ for (String name : archive.names()) {
+ System.out.println(name);
+ }
+}
+```
+
+`MpqOpenOptions.warcraft3()` reads the archive the way Warcraft III does: format
+version 0 is forced, so a corrupted header size or version cannot stop the
+archive opening, and user data headers are ignored. `MpqOpenOptions.defaults()`
+trusts the header instead.
+
+### Writing
+
+Nothing is written until you ask. There is no rebuild-on-close.
+
+```java
+import org.inwc3.jmpq.MpqArchiveWriter;
+import org.inwc3.jmpq.MpqWriteOptions;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+MpqArchiveWriter.create(MpqWriteOptions.defaults())
+ .put("war3map.j", Files.readAllBytes(Path.of("build/war3map.j")))
+ .put("war3mapImported/readme.txt", "generated by JMPQ3".getBytes(StandardCharsets.UTF_8))
+ .save(Path.of("build/MyMap.w3x"));
+```
+
+To rebuild an existing archive, seed a writer from it. Files are copied with
+their stored bytes intact when the target keeps the source's sector size, and
+re-encoded otherwise — that is a correctness requirement, not an optimisation,
+because a sector offset table is expressed in the archive's sector size.
+
+```java
+import org.inwc3.jmpq.MpqArchive;
+import org.inwc3.jmpq.MpqArchiveWriter;
+import org.inwc3.jmpq.MpqOpenOptions;
+import org.inwc3.jmpq.MpqWriteOptions;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+Path path = Path.of("MyMap.w3x");
+byte[] newScript = Files.readAllBytes(Path.of("build/war3map.j"));
+
+try (MpqArchive source = MpqArchive.open(path, MpqOpenOptions.warcraft3())) {
+ MpqArchiveWriter writer = MpqArchiveWriter.from(source, MpqWriteOptions.defaults());
+ writer.remove("war3mapImported/old.blp");
+ writer.put("war3map.j", newScript);
+ writer.save(path.resolveSibling("Rebuilt.w3x"));
+}
+```
+
+### Options worth knowing
+
+```java
+MpqWriteOptions.defaults()
+ .withFormatVersion(1) // 0 or 1; 2 and above are read-only
+ .withSectorSizeShift(3) // 512 << shift; 3 gives the 4 KiB Warcraft III uses
+ .withListfile(false) // omit (listfile): not enumerable by name
+ .withPrefix(true) // keep bytes before the header, as Warcraft III maps have
+ .withSectorChecksums(true) // record an Adler-32 per sector
+ .withAttributes(true) // generate (attributes)
+ .withAttributesTimestamp(millis) // pin it, or the build is not reproducible
+ .withHashTableCapacity(0x10000) // explicit capacity
+ .withExtraBlockEntries(32); // spare block slots
+```
+
+An archive with no `(listfile)` cannot be enumerated by name. Ask what a rebuild
+would cost before doing one:
+
+```java
+if (archive.filesLostOnRebuild() > 0) {
+ // These blocks exist but nothing names them, so a rebuild cannot carry them.
+}
+```
+
+### Integrity
+
+Sector checksums are verified while decoding when an archive records them, and a
+mismatch fails the read rather than returning bytes known to be wrong. Turn it
+off to recover what is still intact from a damaged archive:
+
+```java
+MpqArchive.open(path, MpqOpenOptions.defaults().withSectorChecksumVerification(false));
+```
+
+A format version 3 archive may record MD5 digests of its own tables.
+`archive.integrity()` reports `VERIFIED`, `MISMATCHED` or `UNRECORDED`; a
+mismatch is reported rather than fatal, because the tables may still decode every
+file.
+
+## Thread safety
+
+An archive opened from a `Path` is confined to the thread that opened it, because
+its memory mapping is. Archives opened from a byte array are safe to read
+concurrently. Nothing in `MpqArchive` mutates the archive. A single
+`MpqArchiveWriter` belongs to one thread.
+
+## Migrating from 1.x
+
+`JMpqEditor` and the rest of `systems.crigges.jmpq3` still work and are
+deprecated, not removed. They are now a thin facade over the new core, so both
+paths decode identically.
+
+The coordinates changed: `org.inwc3:jmpq3`. See
+[`docs/migration-2.0.md`](docs/migration-2.0.md) for a method-by-method mapping
+and the behaviour differences worth knowing — the most important being that
+`close()` no longer rewrites the archive as a side effect.
+
+## Limitations
+
+- HET/BET tables are not implemented, so a v2–v4 archive without classic tables
+ cannot be opened.
+- Writing is limited to format versions 0 and 1.
+- No signature verification or signing, and no patch (PTCH) archive support.
+- Empty directories are not representable in MPQ, so only insert regular files.
+- Archives are built in memory, so an archive must fit in the heap.
+
+## Contributing
+
+`AUDIT.md` is the working task list, with each item's status and the reasoning
+behind anything deferred. `docs/mpq-format-notes.md` records format decisions
+with StormLib citations — if you change parsing behaviour, that is where the
+justification belongs.
+
+`tools/mpqref.py` is an independent MPQ reader written from StormLib source,
+sharing no code with the library. CI uses it to verify that archives JMPQ3 writes
+can be read by something other than JMPQ3. It has caught bugs that round-trip
+tests structurally cannot: when a reader and a writer share a misconception they
+agree with each other and with nothing else. Extend it when you add a format
+feature.
diff --git a/build.gradle b/build.gradle
index f9f863e..a677359 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,78 +1,85 @@
-plugins {
- id "idea"
- id "jacoco"
- id "java"
- id 'maven-publish'
-}
-
-group 'org.inwc3'
-version '2.0.0-SNAPSHOT'
-
-java {
- toolchain {
- languageVersion.set(JavaLanguageVersion.of(25))
- }
- withSourcesJar()
- withJavadocJar()
-}
-
-tasks.withType(JavaCompile).configureEach {
- options.release.set(25)
- options.encoding = 'UTF-8'
- options.compilerArgs << '-Xlint:all,-serial,-this-escape'
-}
-
-tasks.withType(Javadoc).configureEach {
- options.encoding = 'UTF-8'
- // The compat adapter is deprecated by design; do not fail the build on it.
- options.addStringOption('Xdoclint:none', '-quiet')
-}
-
-repositories {
- mavenCentral()
- maven { url = 'https://jitpack.io' }
-}
-
-jacoco {
- toolVersion = "0.8.13"
-}
-
-dependencies {
- implementation 'com.jcraft:jzlib:1.1.3'
- implementation group: 'org.apache.commons', name: 'commons-compress', version: '1.27.1'
- implementation 'com.github.eustas:CafeUndZopfli:5cdf283e67'
- implementation group: 'org.tukaani', name: 'xz', version: '1.9'
- implementation group: 'org.slf4j', name: 'slf4j-api', version: '2.0.16'
-
- // Logging backend is a test-only concern: a library must never impose one
- // on its consumers (P3-1).
- testRuntimeOnly group: 'ch.qos.logback', name: 'logback-classic', version: '1.5.18'
- testImplementation 'org.testng:testng:7.11.0'
-}
-
-test {
- useTestNG()
- // FFM mapped-segment access in the read layer.
- jvmArgs '--enable-native-access=ALL-UNNAMED'
- testLogging {
- events "failed"
- exceptionFormat "full"
- }
-}
-
-jacocoTestReport {
- reports {
- xml.required.set(true)
- }
-}
-
-publishing {
- publications {
- maven(MavenPublication) {
- groupId = 'org.inwc3'
- artifactId = 'jmpq3'
-
- from components.java
- }
- }
-}
+plugins {
+ id "idea"
+ id "jacoco"
+ id "java"
+ id 'maven-publish'
+}
+
+group 'org.inwc3'
+version '2.0.0-SNAPSHOT'
+
+java {
+ toolchain {
+ languageVersion.set(JavaLanguageVersion.of(25))
+ }
+ withSourcesJar()
+ withJavadocJar()
+}
+
+tasks.withType(JavaCompile).configureEach {
+ options.release.set(25)
+ options.encoding = 'UTF-8'
+ options.compilerArgs << '-Xlint:all,-serial,-this-escape'
+}
+
+tasks.withType(Javadoc).configureEach {
+ options.encoding = 'UTF-8'
+ // The compat adapter is deprecated by design; do not fail the build on it.
+ options.addStringOption('Xdoclint:none', '-quiet')
+}
+
+repositories {
+ mavenCentral()
+ maven { url = 'https://jitpack.io' }
+}
+
+jacoco {
+ toolVersion = "0.8.13"
+}
+
+dependencies {
+ implementation group: 'org.apache.commons', name: 'commons-compress', version: '1.27.1'
+ implementation 'com.github.eustas:CafeUndZopfli:5cdf283e67'
+ implementation group: 'org.tukaani', name: 'xz', version: '1.9'
+ implementation group: 'org.slf4j', name: 'slf4j-api', version: '2.0.16'
+
+ // Logging backend is a test-only concern: a library must never impose one
+ // on its consumers (P3-1).
+ testRuntimeOnly group: 'ch.qos.logback', name: 'logback-classic', version: '1.5.18'
+ testImplementation 'org.testng:testng:7.11.0'
+}
+
+test {
+ useTestNG()
+ // FFM mapped-segment access in the read layer.
+ jvmArgs '--enable-native-access=ALL-UNNAMED'
+ // Opt-in path to the third-party maps named in issues #46 and #47, which
+ // are not committed here. IssueSampleTests skips when it is unset.
+ // Run with: ./gradlew test -PissueSamples=/path/to/maps
+ // Accepts either form, since -D is the reflex: a Gradle project property
+ // (-PissueSamples) or a system property on the Gradle JVM forwarded to the
+ // test JVM (-Djmpq3.issueSamples).
+ systemProperty 'jmpq3.issueSamples',
+ findProperty('issueSamples') ?: System.getProperty('jmpq3.issueSamples', '')
+ testLogging {
+ events "failed"
+ exceptionFormat "full"
+ }
+}
+
+jacocoTestReport {
+ reports {
+ xml.required.set(true)
+ }
+}
+
+publishing {
+ publications {
+ maven(MavenPublication) {
+ groupId = 'org.inwc3'
+ artifactId = 'jmpq3'
+
+ from components.java
+ }
+ }
+}
diff --git a/src/main/java/systems/crigges/jmpq3/DebugHelper.java b/src/main/java/systems/crigges/jmpq3/DebugHelper.java
index f42d061..0d98c2f 100644
--- a/src/main/java/systems/crigges/jmpq3/DebugHelper.java
+++ b/src/main/java/systems/crigges/jmpq3/DebugHelper.java
@@ -6,13 +6,25 @@
public class DebugHelper {
protected static final char[] hexArray = "0123456789ABCDEF".toCharArray();
+ /** How many bytes a dump shows before giving up. */
+ private static final int MAX_BYTES = 500;
+
+ /**
+ * @param bytes bytes to render.
+ * @return space-separated hex, truncated to the first {@value #MAX_BYTES}
+ * bytes. This is a diagnostic aid, not a serialisation format.
+ */
public static String bytesToHex(byte[] bytes) {
- char[] hexChars = new char[bytes.length * 3];
- for (int j = 0; j < Math.min(bytes.length, 500); j++) {
- int v = bytes[j] & 0xFF;
- hexChars[(j * 3)] = hexArray[(v >>> 4)];
- hexChars[(j * 3 + 1)] = hexArray[(v & 0xF)];
- hexChars[(j * 3 + 2)] = ' ';
+ // Sized to what is actually rendered. Allocating for the full array and
+ // relying on trim() to remove the unwritten tail meant a multi-megabyte
+ // buffer to print half a kilobyte.
+ final int shown = Math.min(bytes.length, MAX_BYTES);
+ final char[] hexChars = new char[shown * 3];
+ for (int j = 0; j < shown; j++) {
+ final int v = bytes[j] & 0xFF;
+ hexChars[j * 3] = hexArray[v >>> 4];
+ hexChars[j * 3 + 1] = hexArray[v & 0xF];
+ hexChars[j * 3 + 2] = ' ';
}
return new String(hexChars).trim();
}
diff --git a/src/main/java/systems/crigges/jmpq3/compression/CompressionUtil.java b/src/main/java/systems/crigges/jmpq3/compression/CompressionUtil.java
index 8e18340..d523336 100644
--- a/src/main/java/systems/crigges/jmpq3/compression/CompressionUtil.java
+++ b/src/main/java/systems/crigges/jmpq3/compression/CompressionUtil.java
@@ -56,11 +56,19 @@ private CompressionUtil() {
* @param data raw sector content.
* @param recompress compression strategy.
* @return the compressed bytes without the leading
- * compression-type byte, which the caller prepends.
+ * compression-type byte, which the caller prepends, or {@code null}
+ * when no recompression was asked for and the caller should store
+ * the data as it is.
*/
public static byte[] compress(byte[] data, RecompressOptions recompress) {
if (!recompress.recompress) {
- return ZlibStore.storeLevel0(data);
+ // Nothing, and the caller stores the sector raw. This used to build
+ // a zlib stream of stored blocks, which is by construction larger
+ // than its input -- so every sector paid for a full copy and an
+ // Adler-32 to produce something the caller always discarded, because
+ // it applies a "did it actually shrink" test. The archives written
+ // are byte for byte identical without it.
+ return null;
}
if (recompress.useZopfli) {
return new ZopfliHelper().deflate(data, recompress.iterations);
diff --git a/src/main/java/systems/crigges/jmpq3/compression/JzLibHelper.java b/src/main/java/systems/crigges/jmpq3/compression/JzLibHelper.java
index e4052a1..879394f 100644
--- a/src/main/java/systems/crigges/jmpq3/compression/JzLibHelper.java
+++ b/src/main/java/systems/crigges/jmpq3/compression/JzLibHelper.java
@@ -1,27 +1,30 @@
package systems.crigges.jmpq3.compression;
-import com.jcraft.jzlib.Deflater;
-import com.jcraft.jzlib.GZIPException;
-import com.jcraft.jzlib.Inflater;
-import com.jcraft.jzlib.JZlib;
-
import java.util.Arrays;
+import java.util.zip.DataFormatException;
+import java.util.zip.Deflater;
+import java.util.zip.Inflater;
/**
- * Deflate/inflate via jzlib.
+ * Deflate and inflate for MPQ sectors.
+ *
+ * Why not jzlib any more (P3-2)
+ * This used to wrap {@code com.jcraft:jzlib}, a pure-Java zlib port last
+ * released in 2011. {@link Deflater} and {@link Inflater} do the same job
+ * through the JDK's bundled zlib, which is native and maintained, so the
+ * dependency bought nothing but a supply-chain risk and a slower codec on the
+ * hottest path in the library.
*
- * Thread safety: every call creates and ends its own
- * {@link Inflater}/{@link Deflater}. The previous implementation held them in
- * static fields together with a shared scratch array and documented itself as
- * "not thread-safe"; two archives compressed at the same time silently produced
- * corrupt sectors.
+ * The class name and signatures are unchanged because they are public API that
+ * tests and downstream code call.
+ *
+ *
Thread safety
+ * Every call creates and ends its own {@link Inflater}/{@link Deflater}. An
+ * older version held them in static fields with a shared scratch array and
+ * documented itself as "not thread-safe"; two archives compressed at the same
+ * time silently produced corrupt sectors.
*/
public final class JzLibHelper {
- /**
- * Whether level-0 output may omit the zlib header and Adler-32 trailer.
- * MPQ consumers expect the wrapper, so this stays off.
- */
- private static final boolean RAW_NOWRAP_FOR_LEVEL0 = false;
private JzLibHelper() {
}
@@ -37,6 +40,14 @@ public static byte[] inflate(byte[] bytes, int offset, int uncompSize) {
}
/**
+ * Inflates a zlib stream.
+ *
+ * A stream that ends early yields what it produced rather than an error:
+ * the caller compares the length against what the block table promised and
+ * reports the shortfall with the file's name attached, which is a better
+ * diagnostic than one from in here. Genuinely malformed data is a different
+ * matter and is thrown.
+ *
* @param bytes buffer holding the deflate stream.
* @param offset index of the first stream byte.
* @param length number of stream bytes available.
@@ -45,111 +56,65 @@ public static byte[] inflate(byte[] bytes, int offset, int uncompSize) {
* produced.
*/
public static byte[] inflate(byte[] bytes, int offset, int length, int uncompSize) {
+ if (uncompSize == 0) {
+ return new byte[0];
+ }
final byte[] out = new byte[uncompSize];
- final Inflater inf = new Inflater();
+ final Inflater inflater = new Inflater();
try {
- inf.init(); // default = zlib wrapper
- inf.setInput(bytes, offset, length, true);
+ inflater.setInput(bytes, offset, length);
int outPos = 0;
- while (outPos < uncompSize) {
- inf.setOutput(out, outPos, uncompSize - outPos);
- final int rc = inf.inflate(JZlib.Z_NO_FLUSH);
-
- if (rc == JZlib.Z_STREAM_END) {
- outPos = (int) inf.getTotalOut();
- break;
- }
- if (rc == JZlib.Z_OK || rc == JZlib.Z_BUF_ERROR) {
- outPos = (int) inf.getTotalOut();
-
- // No input left and no progress possible: stop instead of
- // spinning on a truncated stream.
- if (inf.avail_in == 0 && rc == JZlib.Z_BUF_ERROR) {
+ while (outPos < uncompSize && !inflater.finished()) {
+ final int produced = inflater.inflate(out, outPos, uncompSize - outPos);
+ if (produced == 0) {
+ // No progress and nothing left to feed it: the stream stops
+ // short of what the block table claimed.
+ if (inflater.needsInput() || inflater.needsDictionary()) {
break;
}
- continue;
}
- throw new IllegalStateException("inflate error: " + rc);
+ outPos += produced;
}
-
return outPos == uncompSize ? out : Arrays.copyOf(out, outPos);
+ } catch (DataFormatException e) {
+ throw new IllegalStateException("inflate error: " + e.getMessage(), e);
} finally {
- inf.end();
+ inflater.end();
}
}
/**
- * @param bytes data to compress.
- * @param strongDeflate {@code true} for maximum compression, {@code false}
- * for stored blocks only.
+ * @param bytes data to compress.
+ * @param strongDeflate {@code true} for maximum compression. {@code false}
+ * is no longer a meaningful request — see
+ * {@link CompressionUtil#compress} — and is treated as
+ * maximum compression rather than silently producing
+ * something larger than the input.
* @return the deflate stream.
*/
public static byte[] deflate(byte[] bytes, boolean strongDeflate) {
- final int level = strongDeflate ? JZlib.Z_BEST_COMPRESSION : JZlib.Z_NO_COMPRESSION;
- final boolean nowrap = !strongDeflate && RAW_NOWRAP_FOR_LEVEL0;
- final Deflater def = newDeflater(level, nowrap);
- byte[] comp = new byte[worstCaseZlibSize(bytes.length, !nowrap)];
-
+ final Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
try {
- def.setInput(bytes, 0, bytes.length, true);
- def.setOutput(comp, 0, comp.length);
-
- while (true) {
- final int rc = def.deflate(JZlib.Z_NO_FLUSH);
- if (rc == JZlib.Z_OK || rc == JZlib.Z_BUF_ERROR) {
- if (def.avail_in == 0) {
- break;
- }
- if (def.avail_out == 0) {
- comp = grow(comp);
- def.setOutput(comp, (int) def.getTotalOut(), comp.length - (int) def.getTotalOut());
- }
- continue;
+ deflater.setInput(bytes);
+ deflater.finish();
+
+ // Worst case for incompressible input: stored blocks, each covering
+ // at most 65535 bytes at a cost of 5 bytes, plus the zlib wrapper.
+ final int blocks = (int) (((long) bytes.length + 65534) / 65535);
+ byte[] out = new byte[bytes.length + blocks * 5 + 6 + 16];
+
+ int written = 0;
+ while (!deflater.finished()) {
+ if (written == out.length) {
+ out = Arrays.copyOf(out, out.length * 2);
}
- throw new IllegalStateException("deflate(Z_NO_FLUSH) error: " + rc);
+ written += deflater.deflate(out, written, out.length - written);
}
-
- while (true) {
- if (def.avail_out == 0) {
- comp = grow(comp);
- def.setOutput(comp, (int) def.getTotalOut(), comp.length - (int) def.getTotalOut());
- }
- final int rc = def.deflate(JZlib.Z_FINISH);
- if (rc == JZlib.Z_STREAM_END) {
- break;
- }
- if (rc != JZlib.Z_OK && rc != JZlib.Z_BUF_ERROR) {
- throw new IllegalStateException("deflate(Z_FINISH) error: " + rc);
- }
- }
-
- return Arrays.copyOf(comp, (int) def.getTotalOut());
+ return written == out.length ? out : Arrays.copyOf(out, written);
} finally {
- def.end();
- }
- }
-
- private static Deflater newDeflater(int level, boolean nowrap) {
- try {
- return new Deflater(level, nowrap);
- } catch (GZIPException e) {
- throw new IllegalStateException("Cannot create deflater.", e);
+ deflater.end();
}
}
-
- /**
- * Worst case for stored blocks plus the optional zlib wrapper. Each stored
- * block covers at most 65535 bytes and costs 5 bytes of header.
- */
- private static int worstCaseZlibSize(int n, boolean zlibWrapper) {
- final int blocks = (int) (((long) n + 65534) / 65535);
- final int header = zlibWrapper ? 2 + 4 : 0;
- return n + blocks * 5 + header + 16;
- }
-
- private static byte[] grow(byte[] comp) {
- return Arrays.copyOf(comp, Math.max(64, comp.length * 2));
- }
}
diff --git a/src/main/java/systems/crigges/jmpq3/compression/ZlibStore.java b/src/main/java/systems/crigges/jmpq3/compression/ZlibStore.java
deleted file mode 100644
index d8d36ab..0000000
--- a/src/main/java/systems/crigges/jmpq3/compression/ZlibStore.java
+++ /dev/null
@@ -1,85 +0,0 @@
-package systems.crigges.jmpq3.compression;
-
-/**
- * Emits a zlib stream made entirely of stored (uncompressed) deflate blocks.
- *
- * This is what {@code RecompressOptions.recompress == false} produces: the
- * sector is wrapped so it decodes as valid zlib without spending any time on
- * entropy coding. It replaces a jzlib level-0 round trip, which produced the
- * same bytes far more slowly.
- *
- * Stateless and safe for concurrent use. The previous implementation kept a
- * {@link ThreadLocal} scratch buffer; sizing the output exactly is both simpler
- * and cheaper than growing and caching a shared one.
- */
-final class ZlibStore {
- /** Maximum payload of a single stored deflate block. */
- private static final int MAX_BLOCK = 0xFFFF;
-
- /** Largest number of bytes that can be summed before Adler-32 overflows. */
- private static final int ADLER_CHUNK = 5552;
-
- private static final int ADLER_MODULUS = 65521;
-
- private ZlibStore() {
- }
-
- /**
- * @param in bytes to wrap.
- * @return a zlib stream that decodes back to {@code in}.
- */
- static byte[] storeLevel0(byte[] in) {
- final int len = in.length;
- // long arithmetic: len + MAX_BLOCK - 1 overflows for an input within
- // 64 KiB of Integer.MAX_VALUE.
- final int blocks = (int) Math.max(1, ((long) len + MAX_BLOCK - 1) / MAX_BLOCK);
- // 2 byte zlib header + 5 byte header per stored block + 4 byte Adler-32.
- final byte[] out = new byte[2 + len + blocks * 5 + 4];
-
- int o = 0;
- out[o++] = 0x78;
- out[o++] = 0x01;
-
- int off = 0;
- do {
- final int blockLen = Math.min(MAX_BLOCK, len - off);
- final boolean last = (off + blockLen) == len;
-
- out[o++] = (byte) (last ? 0x01 : 0x00);
- out[o++] = (byte) (blockLen & 0xFF);
- out[o++] = (byte) ((blockLen >>> 8) & 0xFF);
- final int nlen = (~blockLen) & 0xFFFF;
- out[o++] = (byte) (nlen & 0xFF);
- out[o++] = (byte) ((nlen >>> 8) & 0xFF);
-
- System.arraycopy(in, off, out, o, blockLen);
- o += blockLen;
- off += blockLen;
- // A zero length input still needs one (empty, final) block.
- } while (off < len);
-
- final int adler = adler32(in);
- out[o++] = (byte) ((adler >>> 24) & 0xFF);
- out[o++] = (byte) ((adler >>> 16) & 0xFF);
- out[o++] = (byte) ((adler >>> 8) & 0xFF);
- out[o] = (byte) (adler & 0xFF);
-
- return out;
- }
-
- private static int adler32(byte[] in) {
- int s1 = 1;
- int s2 = 0;
- int i = 0;
- while (i < in.length) {
- final int end = i + Math.min(ADLER_CHUNK, in.length - i);
- while (i < end) {
- s1 += in[i++] & 0xFF;
- s2 += s1;
- }
- s1 %= ADLER_MODULUS;
- s2 %= ADLER_MODULUS;
- }
- return (s2 << 16) | s1;
- }
-}
diff --git a/src/test/java/systems/crigges/jmpq3test/DegugHelperTests.java b/src/test/java/systems/crigges/jmpq3test/DebugHelperTests.java
similarity index 94%
rename from src/test/java/systems/crigges/jmpq3test/DegugHelperTests.java
rename to src/test/java/systems/crigges/jmpq3test/DebugHelperTests.java
index 0de7c56..18a76a4 100644
--- a/src/test/java/systems/crigges/jmpq3test/DegugHelperTests.java
+++ b/src/test/java/systems/crigges/jmpq3test/DebugHelperTests.java
@@ -7,7 +7,7 @@
/**
* Created by Frotty on 09.03.2017.
*/
-public class DegugHelperTests {
+public class DebugHelperTests {
@Test
public void testDebugHelper() {
diff --git a/src/test/java/systems/crigges/jmpq3test/GoldenManifest.java b/src/test/java/systems/crigges/jmpq3test/GoldenManifest.java
index 46dc18a..c31d3d8 100644
--- a/src/test/java/systems/crigges/jmpq3test/GoldenManifest.java
+++ b/src/test/java/systems/crigges/jmpq3test/GoldenManifest.java
@@ -20,7 +20,7 @@
* Regenerate with:
*
* python tools/mpqref.py manifest src/test/resources/mpqs \
- * --names src/main/resources/DefaultListfile.txt \
+ * --names src/test/resources/DefaultListfile.txt \
* -o src/test/resources/golden/fixtures.tsv
*
* Any diff in that file during review is a change in observable behaviour and
diff --git a/src/test/java/systems/crigges/jmpq3test/IssueSampleTests.java b/src/test/java/systems/crigges/jmpq3test/IssueSampleTests.java
new file mode 100644
index 0000000..e8ba319
--- /dev/null
+++ b/src/test/java/systems/crigges/jmpq3test/IssueSampleTests.java
@@ -0,0 +1,160 @@
+package systems.crigges.jmpq3test;
+
+import org.inwc3.jmpq.MpqArchive;
+import org.inwc3.jmpq.MpqOpenOptions;
+import org.testng.Assert;
+import org.testng.SkipException;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * The real maps named in the issue tracker.
+ *
+ * Why these are opt-in
+ * Two issues name a specific third-party map as the thing that must work:
+ * {@code #46} points at Forest Defense 0.21g and {@code #47} at Green TD Pro
+ * Deathmatch. Those maps are other people's work, so they are not committed
+ * here; a fixture directory is pointed at instead, and these tests skip when it
+ * is absent.
+ *
+ * That is a real limitation and worth stating plainly: CI does not run these,
+ * so the synthetic fixtures elsewhere are what protects the behaviour
+ * day to day. What these add is the one thing synthetic fixtures cannot — that
+ * the actual archives from the actual bug reports actually open.
+ *
+ *
Running them
+ *
+ * ./gradlew test -PissueSamples=/path/to/maps
+ *
+ * That is a Gradle project property. {@code -D} sets a system property
+ * on Gradle's own JVM rather than on the forked test JVM, so it would leave these
+ * tests skipping while looking like they had run — the build forwards it too, for
+ * exactly that reason, but the line above is the one to reach for.
+ *
+ * Any {@code .w3x} or {@code .mpq} in that directory is opened and every file it
+ * can name is extracted. Maps whose names are recognised get their specific
+ * pathology asserted as well.
+ */
+public class IssueSampleTests {
+
+ /** Directory holding the sample maps, if the runner supplied one. */
+ private static final String PROPERTY = "jmpq3.issueSamples";
+
+ private static List samples() {
+ final String configured = System.getProperty(PROPERTY);
+ if (configured == null || configured.isBlank()) {
+ throw new SkipException("Run with -PissueSamples= to test against the maps"
+ + " named in issues #46 and #47. They are third-party maps and are not"
+ + " committed to this repository.");
+ }
+ final Path directory = Path.of(configured);
+ if (!Files.isDirectory(directory)) {
+ throw new SkipException(PROPERTY + " is not a directory: " + directory);
+ }
+ try (var entries = Files.list(directory)) {
+ final List found = entries
+ .filter(Files::isRegularFile)
+ .filter(path -> {
+ final String name = path.getFileName().toString().toLowerCase();
+ return name.endsWith(".w3x") || name.endsWith(".w3m") || name.endsWith(".mpq");
+ })
+ .sorted()
+ .toList();
+ if (found.isEmpty()) {
+ throw new SkipException("No archives in " + directory);
+ }
+ return found;
+ } catch (IOException e) {
+ throw new SkipException("Cannot list " + directory + ": " + e.getMessage());
+ }
+ }
+
+ /**
+ * Every sample opens without {@code FORCE_V0}.
+ *
+ * That is the substance of issue #46. The old reader rejected a declared
+ * header size outside 32 to 208 outright, and protected maps carry garbage
+ * there — Forest Defense declares 2097410. Needing {@code FORCE_V0} to work
+ * around it was the symptom; not needing it is the fix.
+ */
+ @Test
+ public void everySampleOpensWithoutForcingVersion0() throws IOException {
+ for (Path sample : samples()) {
+ try (MpqArchive archive = MpqArchive.open(sample, MpqOpenOptions.defaults())) {
+ Assert.assertTrue(archive.blockCount() > 0,
+ sample.getFileName() + " opened but has no blocks");
+ }
+ }
+ }
+
+ /**
+ * Whatever a sample can name, it can extract, and both open modes agree on
+ * every byte.
+ *
+ * A protected map typically has no usable list file, so {@code names()} may
+ * be empty and nothing here requires otherwise. Extraction is still checked
+ * against known Warcraft III paths, which is how such a map is recovered in
+ * practice.
+ */
+ @Test
+ public void everySampleExtractsWhatItCanName() throws IOException {
+ final List knownPaths = List.of(
+ "war3map.j", "war3map.w3i", "war3map.w3e", "war3map.wts", "war3map.doo",
+ "war3map.shd", "war3map.wpm", "war3mapMap.blp", "war3mapExtra.txt");
+
+ for (Path sample : samples()) {
+ final List recovered = new ArrayList<>();
+ try (MpqArchive trusted = MpqArchive.open(sample, MpqOpenOptions.defaults());
+ MpqArchive forced = MpqArchive.open(sample, MpqOpenOptions.warcraft3())) {
+
+ for (String name : trusted.names()) {
+ Assert.assertEquals(trusted.read(name), forced.read(name),
+ sample.getFileName() + " / " + name + " differs between open modes");
+ recovered.add(name);
+ }
+
+ // Protected maps: nothing is enumerable, but the content is
+ // still there for a caller who knows the path.
+ for (String probe : knownPaths) {
+ if (trusted.contains(probe)) {
+ final byte[] content = trusted.read(probe);
+ Assert.assertTrue(content.length > 0,
+ sample.getFileName() + " / " + probe + " decoded to nothing");
+ Assert.assertEquals(content, forced.read(probe),
+ sample.getFileName() + " / " + probe + " differs between open modes");
+ recovered.add(probe);
+ }
+ }
+
+ Assert.assertFalse(recovered.isEmpty(),
+ sample.getFileName() + " yielded no files at all, by name or by list file");
+ }
+ }
+ }
+
+ /**
+ * Forest Defense from issue #46, if present: the header size really is
+ * garbage, and the archive really does open anyway.
+ */
+ @Test
+ public void forestDefenseHeaderIsRepairedRatherThanRejected() throws IOException {
+ final Path sample = samples().stream()
+ .filter(path -> path.getFileName().toString().toLowerCase().contains("forest"))
+ .findFirst()
+ .orElseThrow(() -> new SkipException("Forest Defense not among the samples"));
+
+ try (MpqArchive archive = MpqArchive.open(sample, MpqOpenOptions.defaults())) {
+ Assert.assertTrue(archive.header().malformed(),
+ "the whole point is that this header needs repair");
+ Assert.assertEquals(archive.header().headerSize(), 32,
+ "repaired to the size its version implies");
+ Assert.assertTrue(archive.contains("war3map.j"), "the map script must be reachable");
+ Assert.assertTrue(archive.read("war3map.j").length > 0);
+ }
+ }
+}
diff --git a/src/test/java/systems/crigges/jmpq3test/ReadmeExampleTests.java b/src/test/java/systems/crigges/jmpq3test/ReadmeExampleTests.java
new file mode 100644
index 0000000..ec50ad1
--- /dev/null
+++ b/src/test/java/systems/crigges/jmpq3test/ReadmeExampleTests.java
@@ -0,0 +1,119 @@
+package systems.crigges.jmpq3test;
+
+import org.inwc3.jmpq.MpqArchive;
+import org.inwc3.jmpq.MpqArchiveWriter;
+import org.inwc3.jmpq.MpqOpenOptions;
+import org.inwc3.jmpq.MpqWriteOptions;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+/**
+ * The examples in {@code Readme.md}, compiled and run.
+ *
+ * A readme that does not compile is worse than none: it is the first thing a
+ * consumer copies. The previous one had drifted far enough to claim that sparse
+ * and bzip2 were unsupported and that {@code (attributes)} could not be
+ * generated, both of which had stopped being true. This class is here so the
+ * next drift is a build failure rather than a surprise for whoever copies it.
+ *
+ * Keep these snippets and the readme in step. If a signature changes, both fail
+ * together, which is the point.
+ */
+public class ReadmeExampleTests {
+
+ /** The "Quick start" reading example. */
+ @Test
+ public void quickStartReads() throws IOException {
+ Path map = TestResources.mpqCopy("normalMap");
+
+ try (MpqArchive archive = MpqArchive.open(map, MpqOpenOptions.warcraft3())) {
+ Assert.assertTrue(archive.contains("war3map.j"));
+ byte[] script = archive.read("war3map.j");
+ Assert.assertTrue(new String(script, StandardCharsets.UTF_8).length() > 0);
+ Assert.assertFalse(archive.names().isEmpty());
+ }
+ }
+
+ /** The "Writing" example. */
+ @Test
+ public void writingCreatesAnArchive() throws IOException {
+ Path out = TestResources.scratchDir("readme-write").resolve("MyMap.w3x");
+ byte[] script = "function main takes nothing returns nothing\nendfunction"
+ .getBytes(StandardCharsets.UTF_8);
+
+ MpqArchiveWriter.create(MpqWriteOptions.defaults())
+ .put("war3map.j", script)
+ .put("war3mapImported/readme.txt", "generated by JMPQ3".getBytes(StandardCharsets.UTF_8))
+ .save(out);
+
+ try (MpqArchive archive = MpqArchive.open(out, MpqOpenOptions.defaults())) {
+ Assert.assertEquals(archive.read("war3map.j"), script);
+ Assert.assertTrue(archive.contains("war3mapImported/readme.txt"));
+ }
+ }
+
+ /** The rebuild-from-source example. */
+ @Test
+ public void rebuildingFromASourceArchive() throws IOException {
+ Path path = TestResources.mpqCopy("normalMap");
+ Path rebuilt = path.resolveSibling("Rebuilt.w3x");
+ byte[] newScript = "// replaced".getBytes(StandardCharsets.UTF_8);
+
+ try (MpqArchive source = MpqArchive.open(path, MpqOpenOptions.warcraft3())) {
+ MpqArchiveWriter writer = MpqArchiveWriter.from(source, MpqWriteOptions.defaults());
+ writer.remove("war3mapImported/old.blp");
+ writer.put("war3map.j", newScript);
+ writer.save(rebuilt);
+ }
+
+ try (MpqArchive archive = MpqArchive.open(rebuilt, MpqOpenOptions.warcraft3())) {
+ Assert.assertEquals(archive.read("war3map.j"), newScript);
+ }
+ Assert.assertTrue(Files.exists(rebuilt));
+ }
+
+ /** Every option named in the "Options worth knowing" block. */
+ @Test
+ public void everyDocumentedOptionExists() throws IOException {
+ MpqWriteOptions options = MpqWriteOptions.defaults()
+ .withFormatVersion(1)
+ .withSectorSizeShift(3)
+ .withListfile(false)
+ .withPrefix(true)
+ .withSectorChecksums(true)
+ .withAttributes(true)
+ .withAttributesTimestamp(1_600_000_000_000L)
+ .withHashTableCapacity(0x10000)
+ .withExtraBlockEntries(32);
+
+ byte[] image = MpqArchiveWriter.create(options)
+ .put("a.txt", "x".getBytes(StandardCharsets.UTF_8))
+ .toByteArray();
+
+ try (MpqArchive archive = MpqArchive.open(image, MpqOpenOptions.warcraft3())) {
+ Assert.assertEquals(archive.read("a.txt"), "x".getBytes(StandardCharsets.UTF_8));
+ Assert.assertEquals(archive.header().hashTableEntries(), 0x10000);
+ }
+ }
+
+ /** The integrity and rebuild-cost snippets. */
+ @Test
+ public void integrityAndRebuildCostAreReadable() throws IOException {
+ Path map = TestResources.mpqCopy("normalMap");
+
+ try (MpqArchive archive = MpqArchive.open(map, MpqOpenOptions.warcraft3())) {
+ Assert.assertTrue(archive.filesLostOnRebuild() >= 0);
+ Assert.assertNotNull(archive.integrity());
+ }
+
+ try (MpqArchive archive = MpqArchive.open(map,
+ MpqOpenOptions.defaults().withSectorChecksumVerification(false))) {
+ Assert.assertNotNull(archive.names());
+ }
+ }
+}
diff --git a/src/test/java/systems/crigges/jmpq3test/ReadmeSnippetCompilationTests.java b/src/test/java/systems/crigges/jmpq3test/ReadmeSnippetCompilationTests.java
new file mode 100644
index 0000000..49d0433
--- /dev/null
+++ b/src/test/java/systems/crigges/jmpq3test/ReadmeSnippetCompilationTests.java
@@ -0,0 +1,153 @@
+package systems.crigges.jmpq3test;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import javax.tools.Diagnostic;
+import javax.tools.DiagnosticCollector;
+import javax.tools.JavaCompiler;
+import javax.tools.JavaFileObject;
+import javax.tools.SimpleJavaFileObject;
+import javax.tools.StandardLocation;
+import javax.tools.ToolProvider;
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Compiles the Java blocks in {@code Readme.md} exactly as they are written.
+ *
+ *
Why this exists on top of {@link ReadmeExampleTests}
+ * That class runs the same code and so keeps the readme honest about
+ * behaviour. It cannot keep it honest about being copy-pasteable,
+ * because the imports live on the test class rather than in the snippet — which
+ * is exactly how a snippet referencing {@code Files}, {@code Path} and
+ * {@code StandardCharsets} with none of them imported passed review here and
+ * would have failed for the first person to copy it.
+ *
+ * So this compiles the text. A block that presents itself as complete, by
+ * carrying at least one {@code import}, has to compile with only the imports it
+ * declares. Blocks with no imports are fragments — the options builder, the
+ * two-line integrity examples — and are not compiled, because they legitimately
+ * reference variables the surrounding prose introduces.
+ */
+public class ReadmeSnippetCompilationTests {
+
+ /** Fewer than this and the extraction has silently stopped working. */
+ private static final int MINIMUM_COMPLETE_SNIPPETS = 3;
+
+ /** A Java block lifted out of the readme. */
+ private record Snippet(int number, int line, String imports, String body) {
+ boolean isComplete() {
+ return !imports.isBlank();
+ }
+ }
+
+ @Test
+ public void everyCompleteReadmeSnippetCompiles() throws IOException {
+ final Path readme = Path.of("Readme.md");
+ Assert.assertTrue(Files.exists(readme), "run from the project root; looked for " + readme);
+
+ final List snippets = extract(Files.readString(readme));
+ final List complete = snippets.stream().filter(Snippet::isComplete).toList();
+
+ Assert.assertTrue(complete.size() >= MINIMUM_COMPLETE_SNIPPETS,
+ "only found " + complete.size() + " self-contained snippets in the readme,"
+ + " which suggests the extraction below stopped matching its formatting");
+
+ final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
+ Assert.assertNotNull(compiler, "no system Java compiler; run tests on a JDK");
+
+ for (Snippet snippet : complete) {
+ final String className = "ReadmeSnippet" + snippet.number();
+ final String source = snippet.imports()
+ + "\npublic class " + className + " {\n"
+ + " @SuppressWarnings(\"unused\")\n"
+ + " void run() throws Exception {\n"
+ + snippet.body()
+ + "\n }\n}\n";
+
+ final DiagnosticCollector diagnostics = new DiagnosticCollector<>();
+ final boolean compiled = compile(compiler, className, source, diagnostics);
+
+ if (!compiled) {
+ final StringBuilder message = new StringBuilder("Readme.md snippet #")
+ .append(snippet.number()).append(" (line ").append(snippet.line())
+ .append(") does not compile as written:\n");
+ for (Diagnostic diagnostic : diagnostics.getDiagnostics()) {
+ if (diagnostic.getKind() == Diagnostic.Kind.ERROR) {
+ message.append(" ").append(diagnostic.getMessage(null)).append('\n');
+ }
+ }
+ Assert.fail(message.toString());
+ }
+ }
+ }
+
+ /**
+ * Splits each fenced {@code java} block into its imports and the rest.
+ *
+ * The imports have to be hoisted above the generated class declaration,
+ * which is the only rearranging done here — everything else is compiled
+ * verbatim, so a missing import stays missing.
+ */
+ private static List extract(String markdown) {
+ final List snippets = new ArrayList<>();
+ final String[] lines = markdown.split("\n", -1);
+
+ int number = 0;
+ for (int i = 0; i < lines.length; i++) {
+ if (!lines[i].strip().equals("```java")) {
+ continue;
+ }
+ final int startLine = i + 2;
+ final StringBuilder imports = new StringBuilder();
+ final StringBuilder body = new StringBuilder();
+
+ for (i++; i < lines.length && !lines[i].strip().equals("```"); i++) {
+ if (lines[i].startsWith("import ")) {
+ imports.append(lines[i]).append('\n');
+ } else {
+ body.append(" ").append(lines[i]).append('\n');
+ }
+ }
+ snippets.add(new Snippet(++number, startLine, imports.toString(), body.toString()));
+ }
+ return snippets;
+ }
+
+ private static boolean compile(JavaCompiler compiler, String className, String source,
+ DiagnosticCollector diagnostics)
+ throws IOException {
+ final JavaFileObject unit = new SimpleJavaFileObject(
+ URI.create("string:///" + className + ".java"), JavaFileObject.Kind.SOURCE) {
+ @Override
+ public CharSequence getCharContent(boolean ignoreEncodingErrors) {
+ return source;
+ }
+ };
+
+ final Path output = Files.createTempDirectory("readme-snippets");
+ try (var files = compiler.getStandardFileManager(diagnostics, null, null)) {
+ files.setLocation(StandardLocation.CLASS_OUTPUT, List.of(output.toFile()));
+ // The snippets reference this library, so they need the same
+ // classpath the tests run with.
+ final List options = List.of(
+ "-classpath", System.getProperty("java.class.path"));
+ return compiler.getTask(null, files, diagnostics, options, null, List.of(unit)).call();
+ } finally {
+ try (var walk = Files.walk(output)) {
+ walk.sorted(java.util.Comparator.reverseOrder()).forEach(path -> {
+ try {
+ Files.deleteIfExists(path);
+ } catch (IOException ignored) {
+ // A leftover temp file is not worth failing a test over.
+ }
+ });
+ }
+ }
+ }
+}
diff --git a/src/main/resources/DefaultListfile.txt b/src/test/resources/DefaultListfile.txt
similarity index 100%
rename from src/main/resources/DefaultListfile.txt
rename to src/test/resources/DefaultListfile.txt
diff --git a/src/test/resources/golden/fixtures.tsv b/src/test/resources/golden/fixtures.tsv
index 7cfaa83..c9a17f9 100644
--- a/src/test/resources/golden/fixtures.tsv
+++ b/src/test/resources/golden/fixtures.tsv
@@ -165,7 +165,7 @@ w3x2ini.w3x war3mapMap.blp 7753 3ccc826ae481dae9945b577178f967af COMPRESS|EXISTS
w3x2ini.w3x war3mapUnits.doo 238 549ce0ec6c6349232a9af8c6639fd541 COMPRESS|EXISTS zlib
wavTest.w3x (attributes) 56 6f67bd124a4552ad1a3f68fcdb5d56bb COMPRESS|ENCRYPTED|KEY_V2|EXISTS zlib
wavTest.w3x (listfile) 73 484bc94f0099608c4aa070d87c27bce4 COMPRESS|ENCRYPTED|KEY_V2|EXISTS zlib
-wavTest.w3x Abilities\Spells\NightElf\ReviveNightElf\ReviveNightElf.wav 144464 - COMPRESS|EXISTS unsupported:huffman
+wavTest.w3x Abilities\Spells\NightElf\ReviveNightElf\ReviveNightElf.wav 144464 6b131014f093fca5972bfc1a0477f1b1 COMPRESS|EXISTS huffman+adpcm_mono,zlib
wavTest.w3x config.txt 1676 d1d06dc8889f16d72ee542570455f3f8 COMPRESS|EXISTS zlib
ydwe.w3x (attributes) 512 af0e3e2caefe95ed6fa57dccde818556 EXISTS stored
ydwe.w3x (listfile) 201 b7c1d68cef90ed06dd12ec26da82a2d8 EXISTS stored
diff --git a/tools/huffman_tables.py b/tools/huffman_tables.py
new file mode 100644
index 0000000..0c9246c
--- /dev/null
+++ b/tools/huffman_tables.py
@@ -0,0 +1,191 @@
+"""Byte-to-weight tables, transcribed from StormLib src/huffman/huff.cpp.
+
+Order matches its DataDistributions[] pointer array, so the index is the
+data type taken from the first byte of a Huffman stream.
+"""
+
+DATA_DISTRIBUTIONS = (
+ # Sparse
+ (
+ 0x0A, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
+ 0x00, 0x00,
+ ),
+ # Binary
+ (
+ 0x54, 0x16, 0x16, 0x0D, 0x0C, 0x08, 0x06, 0x05, 0x06, 0x05, 0x06, 0x03, 0x04, 0x04, 0x03, 0x05,
+ 0x0E, 0x0B, 0x14, 0x13, 0x13, 0x09, 0x0B, 0x06, 0x05, 0x04, 0x03, 0x02, 0x03, 0x02, 0x02, 0x02,
+ 0x0D, 0x07, 0x09, 0x06, 0x06, 0x04, 0x03, 0x02, 0x04, 0x03, 0x03, 0x03, 0x03, 0x03, 0x02, 0x02,
+ 0x09, 0x06, 0x04, 0x04, 0x04, 0x04, 0x03, 0x02, 0x03, 0x02, 0x02, 0x02, 0x02, 0x03, 0x02, 0x04,
+ 0x08, 0x03, 0x04, 0x07, 0x09, 0x05, 0x03, 0x03, 0x03, 0x03, 0x02, 0x02, 0x02, 0x03, 0x02, 0x02,
+ 0x03, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x01, 0x01, 0x01, 0x02, 0x01, 0x02, 0x02,
+ 0x06, 0x0A, 0x08, 0x08, 0x06, 0x07, 0x04, 0x03, 0x04, 0x04, 0x02, 0x02, 0x04, 0x02, 0x03, 0x03,
+ 0x04, 0x03, 0x07, 0x07, 0x09, 0x06, 0x04, 0x03, 0x03, 0x02, 0x01, 0x02, 0x02, 0x02, 0x02, 0x02,
+ 0x0A, 0x02, 0x02, 0x03, 0x02, 0x02, 0x01, 0x01, 0x02, 0x02, 0x02, 0x06, 0x03, 0x05, 0x02, 0x03,
+ 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x03, 0x01, 0x01, 0x01,
+ 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x04, 0x04, 0x04, 0x07, 0x09, 0x08, 0x0C, 0x02,
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x03,
+ 0x04, 0x01, 0x02, 0x04, 0x05, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01,
+ 0x04, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x03, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x02, 0x01, 0x01, 0x02, 0x02, 0x02, 0x06, 0x4B,
+ 0x00, 0x00,
+ ),
+ # Text
+ (
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x27, 0x00, 0x00, 0x23, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xFF, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x02, 0x01, 0x01, 0x06, 0x0E, 0x10, 0x04,
+ 0x06, 0x08, 0x05, 0x04, 0x04, 0x03, 0x03, 0x02, 0x02, 0x03, 0x03, 0x01, 0x01, 0x02, 0x01, 0x01,
+ 0x01, 0x04, 0x02, 0x04, 0x02, 0x02, 0x02, 0x01, 0x01, 0x04, 0x01, 0x01, 0x02, 0x03, 0x03, 0x02,
+ 0x03, 0x01, 0x03, 0x06, 0x04, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x02, 0x01, 0x01,
+ 0x01, 0x29, 0x07, 0x16, 0x12, 0x40, 0x0A, 0x0A, 0x11, 0x25, 0x01, 0x03, 0x17, 0x10, 0x26, 0x2A,
+ 0x10, 0x01, 0x23, 0x23, 0x2F, 0x10, 0x06, 0x07, 0x02, 0x09, 0x01, 0x01, 0x01, 0x01, 0x01, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+ # General
+ (
+ 0xFF, 0x0B, 0x07, 0x05, 0x0B, 0x02, 0x02, 0x02, 0x06, 0x02, 0x02, 0x01, 0x04, 0x02, 0x01, 0x03,
+ 0x09, 0x01, 0x01, 0x01, 0x03, 0x04, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01,
+ 0x05, 0x01, 0x01, 0x01, 0x0D, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x02, 0x01, 0x01, 0x03, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x01,
+ 0x0A, 0x04, 0x02, 0x01, 0x06, 0x03, 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x03, 0x01, 0x01, 0x01,
+ 0x05, 0x02, 0x03, 0x04, 0x03, 0x03, 0x03, 0x02, 0x01, 0x01, 0x01, 0x02, 0x01, 0x02, 0x03, 0x03,
+ 0x01, 0x03, 0x01, 0x01, 0x02, 0x05, 0x01, 0x01, 0x04, 0x03, 0x05, 0x01, 0x03, 0x01, 0x03, 0x03,
+ 0x02, 0x01, 0x04, 0x03, 0x0A, 0x06, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x02, 0x02, 0x01, 0x0A, 0x02, 0x05, 0x01, 0x01, 0x02, 0x07, 0x02, 0x17, 0x01, 0x05, 0x01, 0x01,
+ 0x0E, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x06, 0x02, 0x01, 0x04, 0x05, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01,
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+ 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x07, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x01,
+ 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x02, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x11,
+ 0x00, 0x00,
+ ),
+ # ADPCM_4
+ (
+ 0xFF, 0xFB, 0x98, 0x9A, 0x84, 0x85, 0x63, 0x64, 0x3E, 0x3E, 0x22, 0x22, 0x13, 0x13, 0x18, 0x17,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+ # ADPCM_6
+ (
+ 0xFF, 0xF1, 0x9D, 0x9E, 0x9A, 0x9B, 0x9A, 0x97, 0x93, 0x93, 0x8C, 0x8E, 0x86, 0x88, 0x80, 0x82,
+ 0x7C, 0x7C, 0x72, 0x73, 0x69, 0x6B, 0x5F, 0x60, 0x55, 0x56, 0x4A, 0x4B, 0x40, 0x41, 0x37, 0x37,
+ 0x2F, 0x2F, 0x27, 0x27, 0x21, 0x21, 0x1B, 0x1C, 0x17, 0x17, 0x13, 0x13, 0x10, 0x10, 0x0D, 0x0D,
+ 0x0B, 0x0B, 0x09, 0x09, 0x08, 0x08, 0x07, 0x07, 0x06, 0x05, 0x05, 0x04, 0x04, 0x04, 0x19, 0x18,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+ # Stereo_3
+ (
+ 0xC3, 0xCB, 0xF5, 0x41, 0xFF, 0x7B, 0xF7, 0x21, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xBF, 0xCC, 0xF2, 0x40, 0xFD, 0x7C, 0xF7, 0x22, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x7A, 0x46, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+ # Stereo_4
+ (
+ 0xC3, 0xD9, 0xEF, 0x3D, 0xF9, 0x7C, 0xE9, 0x1E, 0xFD, 0xAB, 0xF1, 0x2C, 0xFC, 0x5B, 0xFE, 0x17,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xBD, 0xD9, 0xEC, 0x3D, 0xF5, 0x7D, 0xE8, 0x1D, 0xFB, 0xAE, 0xF0, 0x2C, 0xFB, 0x5C, 0xFF, 0x18,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x70, 0x6C, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+ # Stereo_5
+ (
+ 0xBA, 0xC5, 0xDA, 0x33, 0xE3, 0x6D, 0xD8, 0x18, 0xE5, 0x94, 0xDA, 0x23, 0xDF, 0x4A, 0xD1, 0x10,
+ 0xEE, 0xAF, 0xE4, 0x2C, 0xEA, 0x5A, 0xDE, 0x15, 0xF4, 0x87, 0xE9, 0x21, 0xF6, 0x43, 0xFC, 0x12,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xB0, 0xC7, 0xD8, 0x33, 0xE3, 0x6B, 0xD6, 0x18, 0xE7, 0x95, 0xD8, 0x23, 0xDB, 0x49, 0xD0, 0x11,
+ 0xE9, 0xB2, 0xE2, 0x2B, 0xE8, 0x5C, 0xDD, 0x15, 0xF1, 0x87, 0xE7, 0x20, 0xF7, 0x44, 0xFF, 0x13,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x5F, 0x9E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00,
+ ),
+)
+
+DATA_TYPE_SPARSE = 0
+
diff --git a/tools/mpqcodecs.py b/tools/mpqcodecs.py
new file mode 100644
index 0000000..ac3e198
--- /dev/null
+++ b/tools/mpqcodecs.py
@@ -0,0 +1,411 @@
+"""Huffman and ADPCM decompression, transcribed from StormLib.
+
+These are the two codecs a Warcraft III ``.wav`` is stored with, and the reason
+this file exists is issue #11: attributes generation was disabled years ago
+because the CRC32 of some wav files disagreed with StormLib's. A CRC32 over
+decoded content can only disagree if the decode disagrees, so the only way to
+settle it is to decode the same file twice, independently.
+
+"Independently" is the whole point. This is a transcription of
+``src/huffman/huff.cpp`` and ``src/adpcm/adpcm.cpp`` from StormLib, written from
+the C and not from the Java in this repository. If it agreed with the Java by
+construction it would prove nothing -- that trap has already cost this project
+two real bugs, both of which a reader and a writer sharing a misconception
+agreed about perfectly.
+
+Deliberately omitted: the quick-link cache in ``DecodeOneByte``. It is a lookup
+table that short-circuits the first seven levels of tree descent and cannot
+change the value produced, only how fast it is produced. Leaving it out keeps
+this readable, which matters more here than speed.
+"""
+
+from huffman_tables import DATA_DISTRIBUTIONS, DATA_TYPE_SPARSE
+
+MAX_DATA_TYPE = len(DATA_DISTRIBUTIONS)
+
+# Sentinels that live in the tree alongside the 256 byte values.
+END_OF_STREAM = 0x100
+NEW_BYTE_ESCAPE = 0x101
+
+
+class HuffmanError(Exception):
+ """The stream could not be decoded."""
+
+
+class _BitReader:
+ """StormLib's TInputStream: little-endian, least-significant bit first."""
+
+ def __init__(self, data):
+ self.data = data
+ self.at = 0
+ self.buffer = 0
+ self.count = 0
+
+ def bit(self):
+ if self.count == 0:
+ if self.at >= len(self.data):
+ return None
+ self.buffer = self.data[self.at]
+ self.at += 1
+ self.count = 8
+ value = self.buffer & 0x01
+ self.buffer >>= 1
+ self.count -= 1
+ return value
+
+ def byte(self):
+ if self.count < 8:
+ if self.at >= len(self.data):
+ return None
+ self.buffer |= self.data[self.at] << self.count
+ self.at += 1
+ self.count += 8
+ value = self.buffer & 0xFF
+ self.buffer >>= 8
+ self.count -= 8
+ return value
+
+
+class _Item:
+ """One node, which is simultaneously a tree node and a list element.
+
+ StormLib keeps every node in one weight-sorted doubly-linked list, and a
+ node's two children are ``child_lo`` and ``child_lo.prev``. The list order
+ *is* the tree structure; that is why the rebalancing below moves list links
+ around rather than swapping child pointers.
+ """
+
+ __slots__ = ("value", "weight", "parent", "child_lo", "prev", "next")
+
+ def __init__(self, value=0, weight=0):
+ self.value = value
+ self.weight = weight
+ self.parent = None
+ self.child_lo = None
+ self.prev = self
+ self.next = self
+
+ def unlink(self):
+ if self.prev is not None and self.next is not None:
+ self.prev.next = self.next
+ self.next.prev = self.prev
+ self.prev = None
+ self.next = None
+
+
+class HuffmanTree:
+ """StormLib's THuffmannTree, decompression side only."""
+
+ def __init__(self):
+ self.head = _Item()
+ self.by_byte = {}
+ self.sparse = False
+
+ # -- list plumbing, mirroring LinkTwoItems / InsertItem -----------------
+
+ def _link_after(self, anchor, item):
+ item.next = anchor.next
+ item.prev = anchor.next.prev
+ anchor.next.prev = item
+ anchor.next = item
+
+ def _insert(self, item, before, anchor):
+ item.unlink()
+ if anchor is None:
+ anchor = self.head
+ if before:
+ self._link_after(anchor.prev, item)
+ else:
+ self._link_after(anchor, item)
+
+ def _create(self, value, weight, before):
+ item = _Item(value, weight)
+ item.prev = item.next = None
+ self._insert(item, before, None)
+ return item
+
+ def _find_higher_or_equal(self, item, weight):
+ while item is not None and item is not self.head:
+ if item.weight >= weight:
+ return item
+ item = item.prev
+ return self.head
+
+ def _fixup_by_weight(self, item, max_weight):
+ if item.weight < max_weight:
+ higher = self._find_higher_or_equal(self.head.prev, item.weight)
+ item.unlink()
+ self._link_after(higher, item)
+ else:
+ max_weight = item.weight
+ return max_weight
+
+ # -- tree construction, mirroring BuildTree ----------------------------
+
+ def build(self, data_type):
+ data_type &= 0x0F
+ if data_type >= MAX_DATA_TYPE:
+ raise HuffmanError("data type %d has no weight table" % data_type)
+
+ weights = DATA_DISTRIBUTIONS[data_type]
+ self.by_byte = {}
+ max_weight = 0
+
+ for value in range(0x100):
+ if weights[value] != 0:
+ item = self._create(value, weights[value], before=False)
+ self.by_byte[value] = item
+ max_weight = self._fixup_by_weight(item, max_weight)
+
+ self.by_byte[END_OF_STREAM] = self._create(END_OF_STREAM, 1, before=True)
+ self.by_byte[NEW_BYTE_ESCAPE] = self._create(NEW_BYTE_ESCAPE, 1, before=True)
+
+ # Combine from the light end upwards, as BuildTree does.
+ child_lo = self.head.prev
+ while child_lo is not self.head:
+ child_hi = child_lo.prev
+ if child_hi is self.head:
+ break
+ parent = self._create(0, child_hi.weight + child_lo.weight, before=False)
+ child_lo.parent = parent
+ child_hi.parent = parent
+ parent.child_lo = child_lo
+ max_weight = self._fixup_by_weight(parent, max_weight)
+ child_lo = child_hi.prev
+
+ # -- the adaptive part, mirroring IncWeightsAndRebalance ---------------
+
+ def _inc_weights_and_rebalance(self, item):
+ while item is not None:
+ item.weight += 1
+ higher = self._find_higher_or_equal(item.prev, item.weight)
+ child_hi = higher.next
+
+ if child_hi is not item:
+ child_hi.unlink()
+ self._link_after(item, child_hi)
+ item.unlink()
+ self._link_after(higher, item)
+
+ child_lo = child_hi.parent.child_lo
+ parent = item.parent
+ if parent.child_lo is item:
+ parent.child_lo = child_hi
+ if child_lo is child_hi:
+ child_hi.parent.child_lo = item
+ parent = item.parent
+ item.parent = child_hi.parent
+ child_hi.parent = parent
+
+ item = item.parent
+
+ def _insert_branch_and_rebalance(self, value1, value2):
+ last = self.head.prev
+
+ child_hi = self._create(value1, last.weight, before=True)
+ child_hi.parent = last
+ self.by_byte[value1] = child_hi
+
+ child_lo = self._create(value2, 0, before=True)
+ child_lo.parent = last
+ last.child_lo = child_lo
+ self.by_byte[value2] = child_lo
+
+ self._inc_weights_and_rebalance(child_lo)
+
+ # -- decoding, mirroring DecodeOneByte / Decompress --------------------
+
+ def _decode_one(self, reader):
+ if self.head.next is self.head:
+ raise HuffmanError("empty tree")
+
+ item = self.head.next
+ while item.child_lo is not None:
+ bit = reader.bit()
+ if bit is None:
+ raise HuffmanError("stream ended mid-symbol")
+ # A set bit takes the higher-weight child, which is the list
+ # neighbour to the left of child_lo.
+ item = item.child_lo.prev if bit else item.child_lo
+ return item.value
+
+ def decompress(self, data, expected_size):
+ if expected_size == 0:
+ return b""
+
+ reader = _BitReader(data)
+ data_type = reader.byte()
+ if data_type is None:
+ raise HuffmanError("stream too short for a data type")
+ self.sparse = data_type == DATA_TYPE_SPARSE
+ self.build(data_type)
+
+ out = bytearray()
+ while True:
+ value = self._decode_one(reader)
+ if value == END_OF_STREAM:
+ break
+
+ if value == NEW_BYTE_ESCAPE:
+ value = reader.byte()
+ if value is None:
+ raise HuffmanError("escape with no byte after it")
+ self._insert_branch_and_rebalance(self.head.prev.value, value)
+ if not self.sparse:
+ self._inc_weights_and_rebalance(self.by_byte[value])
+
+ if len(out) >= expected_size:
+ break
+ out.append(value)
+
+ if self.sparse:
+ self._inc_weights_and_rebalance(self.by_byte[value])
+
+ return bytes(out)
+
+
+def huffman_decompress(data, expected_size):
+ """Decompresses one Huffman-compressed sector."""
+ return HuffmanTree().decompress(data, expected_size)
+
+
+# ---------------------------------------------------------------- ADPCM
+
+INITIAL_STEP_INDEX = 0x2C
+
+NEXT_STEP_TABLE = (
+ -1, 0, -1, 4, -1, 2, -1, 6,
+ -1, 1, -1, 5, -1, 3, -1, 7,
+ -1, 1, -1, 5, -1, 3, -1, 7,
+ -1, 2, -1, 4, -1, 6, -1, 8,
+)
+
+STEP_SIZE_TABLE = (
+ 7, 8, 9, 10, 11, 12, 13, 14,
+ 16, 17, 19, 21, 23, 25, 28, 31,
+ 34, 37, 41, 45, 50, 55, 60, 66,
+ 73, 80, 88, 97, 107, 118, 130, 143,
+ 157, 173, 190, 209, 230, 253, 279, 307,
+ 337, 371, 408, 449, 494, 544, 598, 658,
+ 724, 796, 876, 963, 1060, 1166, 1282, 1411,
+ 1552, 1707, 1878, 2066, 2272, 2499, 2749, 3024,
+ 3327, 3660, 4026, 4428, 4871, 5358, 5894, 6484,
+ 7132, 7845, 8630, 9493, 10442, 11487, 12635, 13899,
+ 15289, 16818, 18500, 20350, 22385, 24623, 27086, 29794,
+ 32767,
+)
+
+
+def _next_step_index(step_index, encoded):
+ step_index += NEXT_STEP_TABLE[encoded & 0x1F]
+ if step_index < 0:
+ return 0
+ if step_index > 88:
+ return 88
+ return step_index
+
+
+def _update_predicted(predicted, encoded, difference, bit_mask=0x40):
+ if encoded & bit_mask:
+ predicted -= difference
+ if predicted <= -32768:
+ predicted = -32768
+ else:
+ predicted += difference
+ if predicted >= 32767:
+ predicted = 32767
+ return predicted
+
+
+def _decode_sample(predicted, encoded, step_size, difference):
+ for shift in range(6):
+ if encoded & (1 << shift):
+ difference += step_size >> shift
+ return _update_predicted(predicted, encoded, difference)
+
+
+def _to_signed16(value):
+ value &= 0xFFFF
+ return value - 0x10000 if value & 0x8000 else value
+
+
+def adpcm_decompress(data, channels, expected_size):
+ """Decompresses one ADPCM-compressed sector.
+
+ Mirrors ``DecompressADPCM``. The output is capped at ``expected_size``, the
+ way the C is bounded by its output buffer: it stops writing when full and
+ reports how much it wrote.
+ """
+ predicted = [0, 0]
+ step_indexes = [INITIAL_STEP_INDEX, INITIAL_STEP_INDEX]
+
+ out = bytearray()
+ at = 0
+
+ def read_byte():
+ nonlocal at
+ if at >= len(data):
+ return None
+ value = data[at]
+ at += 1
+ return value
+
+ def read_word():
+ nonlocal at
+ if len(data) - at < 2:
+ return None
+ value = data[at] | (data[at + 1] << 8)
+ at += 2
+ return _to_signed16(value)
+
+ def write_word(sample):
+ if len(out) + 2 > expected_size:
+ return False
+ out.append(sample & 0xFF)
+ out.append((sample >> 8) & 0xFF)
+ return True
+
+ # The first byte is always zero; the second holds the bit shift.
+ read_byte()
+ bit_shift = read_byte()
+ if bit_shift is None:
+ return bytes(out)
+
+ for channel in range(channels):
+ initial = read_word()
+ if initial is None:
+ return bytes(out)
+ predicted[channel] = initial
+ if not write_word(initial):
+ return bytes(out)
+
+ channel_index = channels - 1
+
+ while True:
+ encoded = read_byte()
+ if encoded is None:
+ break
+
+ channel_index = (channel_index + 1) % channels
+
+ if encoded == 0x80:
+ if step_indexes[channel_index] != 0:
+ step_indexes[channel_index] -= 1
+ if not write_word(predicted[channel_index]):
+ return bytes(out)
+ elif encoded == 0x81:
+ step_indexes[channel_index] += 8
+ if step_indexes[channel_index] > 0x58:
+ step_indexes[channel_index] = 0x58
+ # Stay on the same channel for the next pass.
+ channel_index = (channel_index + 1) % channels
+ else:
+ step_index = step_indexes[channel_index]
+ step_size = STEP_SIZE_TABLE[step_index]
+ predicted[channel_index] = _to_signed16(_decode_sample(
+ predicted[channel_index], encoded, step_size, step_size >> bit_shift))
+ if not write_word(predicted[channel_index]):
+ break
+ step_indexes[channel_index] = _next_step_index(step_index, encoded)
+
+ return bytes(out)
diff --git a/tools/mpqref.py b/tools/mpqref.py
index 1a0dfaf..28cd94b 100644
--- a/tools/mpqref.py
+++ b/tools/mpqref.py
@@ -11,7 +11,7 @@
Fully decodes: stored, zlib, bzip2, sparse, and the multi-stage combinations of
those, applied in StormLib's ``dcmp_table`` order.
-Not decoded: PKWARE ("implode"), Huffman and ADPCM. Porting three more codecs
+Not decoded: PKWARE ("implode"). Porting one more codec
here would buy little: JMPQ3's *writer* only ever emits zlib or stored sectors,
so round-trip verification of anything JMPQ3 produces is fully covered. Files
using the other codecs are reported with ``codec=unsupported`` and their
@@ -35,6 +35,8 @@
import zlib
from pathlib import Path
+import mpqcodecs
+
# --------------------------------------------------------------------------
# Cryptography (StormLib InitializeMpqCryptography)
# --------------------------------------------------------------------------
@@ -191,9 +193,9 @@ def raiser(_data, _max_size):
(0x10, "bzip2", _bzip2),
(0x08, "pkware", _unsupported("pkware")),
(0x02, "zlib", _zlib),
- (0x01, "huffman", _unsupported("huffman")),
- (0x80, "adpcm_stereo", _unsupported("adpcm_stereo")),
- (0x40, "adpcm_mono", _unsupported("adpcm_mono")),
+ (0x01, "huffman", lambda data, size: mpqcodecs.huffman_decompress(data, size)),
+ (0x80, "adpcm_stereo", lambda data, size: mpqcodecs.adpcm_decompress(data, 2, size)),
+ (0x40, "adpcm_mono", lambda data, size: mpqcodecs.adpcm_decompress(data, 1, size)),
(0x20, "sparse", _sparse),
]