Phase 1 complete: compat facade, behaviour contracts, extension hooks

docs/migration-2.0.md

@@ -0,0 +1,117 @@
+# Migrating to JMPQ3 2.0
+
+The 1.x API still works. `JMpqEditor` is a deprecated facade over the new core
+and behaves as it did, so existing code compiles and runs unchanged. This
+describes what to move to when you are ready.
+
+## Coordinates and packages
+
+| | 1.x | 2.0 |
+|---|---|---|
+| Maven group | `systems.crigges` | `org.inwc3` |
+| New core | — | `org.inwc3.jmpq` |
+| Deprecated facade | `systems.crigges.jmpq3` | unchanged, still importable |
+| Java baseline | 11 | 25 |
+
+JitPack consumers using `com.github.inwc3:JMPQ3` are unaffected. Anything
+resolving the Maven coordinate directly needs the new group.
+
+## The shape of the new API
+
+Reading and writing are separate types, and writing happens when you ask for it:
+
+```java
+// read
+try (MpqArchive archive = MpqArchive.open(path, MpqOpenOptions.warcraft3())) {
+    byte[] script = archive.read("war3map.j");
+    for (String name : archive.names()) { ... }
+}
+
+// write
+byte[] image;
+try (MpqArchive source = MpqArchive.open(path, MpqOpenOptions.warcraft3())) {
+    image = MpqArchiveWriter.from(source, MpqWriteOptions.defaults())
+        .put("war3map.j", newScript)
+        .toByteArray();
+}
+Files.write(path, image);
+```
+
+The build happens while the source is open, because content is read from it, and
+the write happens after it is closed, because a mapped file cannot be replaced
+on Windows.
+
+## Method mapping
+
+| 1.x `JMpqEditor` | 2.0 |
+|---|---|
+| `new JMpqEditor(path, FORCE_V0)` | `MpqArchive.open(path, MpqOpenOptions.warcraft3())` |
+| `new JMpqEditor(path, READ_ONLY)` | `MpqArchive.open(path, MpqOpenOptions.defaults())` |
+| `hasFile(name)` | `MpqArchive.contains(name)` |
+| `hasFile(name, locale)` | `MpqArchive.contains(name, locale)` |
+| `extractFileAsBytes(name)` | `MpqArchive.read(name)` |
+| `extractFile(name, stream)` | `MpqArchive.readTo(name, stream)` |
+| `extractFileAsString(name)` | `new String(archive.read(name), UTF_8)` |
+| `getFileNames()` | `MpqArchive.names()` |
+| `getTotalFileCount()` | `MpqArchive.blockCount()` |
+| `getMpqFile(name)` | `MpqArchive.entry(name)` then `read(entry)` |
+| `getBlockTable()`, `getHashTable()` | `MpqArchive.entries()` |
+| `insertByteArray(name, bytes)` | `MpqArchiveWriter.put(name, bytes)` |
+| `insertFile(name, file)` | `MpqArchiveWriter.put(name, path)` |
+| `deleteFile(name)` | `MpqArchiveWriter.remove(name)` |
+| `close()` rebuilding the archive | `MpqArchiveWriter.save(path)` or `toByteArray()` |
+| `getOutputByteArray()` | `MpqArchiveWriter.toByteArray()` |
+| `setKeepHeaderOffset(false)` | `MpqWriteOptions.withPrefix(false)` |
+| `close(buildListfile, ...)` | `MpqWriteOptions.withListfile(...)` |
+| `RecompressOptions.newSectorSizeShift` | `MpqWriteOptions.withSectorSizeShift(...)` |
+
+## Behaviour differences worth knowing
+
+**Nothing is written unless you ask.** The facade rebuilds on `close()`. The core
+does not: `save` is a separate call. That is the single biggest difference, and
+the reason the split exists — a read can no longer rewrite a file.
+
+**The format version is chosen, not inherited.** `MpqWriteOptions` accepts
+version 0 or 1 and refuses the rest at construction. 1.x inherited whatever it
+read and then emitted a version 0 header body for it.
+
+**Sector size changes force a re-encode.** A file can only keep its stored bytes
+when the target archive keeps the source's sector size, because a sector offset
+table is expressed in that sector size. The writer works this out; 1.x copied
+regardless and silently corrupted the result.
+
+**Locales are first class.** One path can exist under several locales, and both
+sides know it: `MpqArchive.localesOf(name)`, and `put`/`remove`/`contains`
+overloads taking a locale. 1.x registered everything as neutral and a rebuild
+dropped all but one variant.
+
+**An archive that cannot enumerate itself says so.** `MpqArchive.isEnumerable()`
+and `filesLostOnRebuild()` replace a log warning, so you can find out before a
+rebuild how many files it would drop. Use `filesLostOnRebuild()`, not the
+deprecated `unnamedBlockCount()`: the latter counts only nameless blocks, so it
+reports zero for an archive whose `(attributes)` file a rebuild is about to
+discard.
+
+**Malformed headers are repaired, not rejected.** A garbage header size is
+replaced by the version's real size and the archive opens with
+`header().malformed()` set, matching what Storm.dll does. 1.x refused unless you
+passed `FORCE_V0`.
+
+**Hi-block tables are refused.** Archives placing file data beyond 4 GiB are
+rejected explicitly rather than misread. Support arrives with the version 2 to 4
+read work.
+
+## Protection tooling
+
+The `w3p` branch's `fakeFilesCount` and "maximise tables" mode are not in the
+core. The mechanisms they needed are:
+
+```java
+MpqWriteOptions.defaults()
+    .withHashTableCapacity(0x10000)   // maximised version 0 table
+    .withExtraBlockEntries(32)        // spare block slots
+    .withListfile(false)              // not enumerable by name
+```
+
+Decoy entries are ordinary `put` calls; what they are called is the tool's
+policy, not this library's.

src/main/java/org/inwc3/jmpq/MpqArchive.java

@@ -51,6 +51,21 @@ public final class MpqArchive implements AutoCloseable {
     /** Encryption key for block table data. */
     private static final int KEY_BLOCK_TABLE = tableKey("(block table)");
 
+    /**
+     * Files an archive holds by convention rather than by being listed. A list
+     * file does not name itself, so these have to be known rather than
+     * discovered.
+     */
+    private static final List<String> INTERNAL_NAMES =
+        List.of("(listfile)", "(attributes)", "(signature)");
+
+    /**
+     * Internal files a rebuild does not carry over. {@code (listfile)} is
+     * regenerated, so it is not lost; these two are simply dropped, which
+     * {@link #filesLostOnRebuild()} has to account for.
+     */
+    private static final List<String> LOST_ON_REBUILD = List.of("(attributes)", "(signature)");
+
     private static int tableKey(String name) {
         final MPQHashGenerator hasher = MPQHashGenerator.getFileKeyGenerator();
         hasher.process(name);
@@ -69,6 +84,25 @@ private static int tableKey(String name) {
     /** Names from the archive's list file, canonical name to spelling. */
     private final SequencedMap<String, String> names = new LinkedHashMap<>();
 
+    /**
+     * How much this archive knows about its own contents.
+     * <p>
+     * Three states that must not be conflated, and were: an archive with no
+     * list file, one whose list file will not parse, and one whose list file
+     * parsed and happens to be empty. The first two mean a rebuild would lose
+     * whatever it cannot name; the third is a perfectly healthy empty archive.
+     */
+    public enum Enumeration {
+        /** The list file parsed. {@link #names()} is authoritative. */
+        PARSED,
+        /** No {@code (listfile)} block at all. */
+        ABSENT,
+        /** A {@code (listfile)} block that could not be decoded. */
+        UNREADABLE
+    }
+
+    private Enumeration enumeration = Enumeration.ABSENT;
+
     private MpqArchive(MpqSource source, MpqOpenOptions options) throws IOException {
         this.source = source;
         this.defaultLocale = options.defaultLocale();
@@ -245,6 +279,14 @@ public List<MpqFileEntry> entries() {
         for (String name : names.values()) {
             namesByKey.put(MpqNames.fileKey(name), name);
         }
+        // The internal files are known by name even though a list file does not
+        // list itself. Without these they would count as unnameable, which is
+        // what unnamedBlockCount reports as data a rebuild would lose.
+        for (String internal : INTERNAL_NAMES) {
+            if (hashTable.hasFile(internal)) {
+                namesByKey.put(MpqNames.fileKey(internal), internal);
+            }
+        }
         final Map<Integer, HashTable.Mapping> byBlock = new LinkedHashMap<>();
         for (HashTable.Mapping mapping : hashTable.mappings()) {
             // Prefer a mapping whose name is known, so a block reachable by
@@ -418,17 +460,22 @@ private HashTable readHashTable() throws IOException {
      */
     private void readNames() {
         if (!contains("(listfile)")) {
+            enumeration = Enumeration.ABSENT;
             log.debug("{} has no (listfile); it cannot be enumerated.", source.origin());
             return;
         }
         final Listfile listfile;
         try {
             listfile = new Listfile(read("(listfile)"));
         } catch (IOException | RuntimeException e) {
+            enumeration = Enumeration.UNREADABLE;
             log.warn("Cannot read the (listfile) of {}; the archive is not enumerable.",
                 source.origin(), e);
             return;
         }
+        // Parsed, even if it turns out to name nothing: an empty list file is a
+        // valid list file, and a fresh archive has one.
+        enumeration = Enumeration.PARSED;
 
         for (String name : listfile.getFiles()) {
             if (contains(name)) {
@@ -440,6 +487,105 @@ private void readNames() {
         }
     }
 
+    /**
+     * Whether this archive can list its own contents.
+     * <p>
+     * An archive without a usable {@code (listfile)} cannot: the hash table
+     * stores hashes, not names, so there is nothing to enumerate. Its files are
+     * still readable by exact name. This is a queryable fact rather than a log
+     * line, because a caller about to rebuild needs to know that names it
+     * cannot see would be dropped.
+     *
+     * @return whether {@link #names()} reflects the whole archive.
+     */
+    public boolean isEnumerable() {
+        return enumeration == Enumeration.PARSED;
+    }
+
+    /**
+     * @return which of the three enumeration states this archive is in.
+     */
+    public Enumeration enumerationState() {
+        return enumeration;
+    }
+
+    /**
+     * How many live blocks no name resolves to.
+     * <p>
+     * A rebuild can only carry over files it can name, so this is exactly how
+     * many files a rebuild would discard. Non-zero means the archive's list file
+     * is incomplete, which protected archives do deliberately. Before 2.0 this
+     * was a log warning at open time and nothing a caller could act on.
+     *
+     * @return the number of unnameable live blocks.
+     */
+    public int filesLostOnRebuild() {
+        int lost = 0;
+        for (MpqFileEntry entry : entries()) {
+            if (entry.name().isEmpty() || LOST_ON_REBUILD.contains(entry.name())) {
+                lost++;
+            }
+        }
+        return lost;
+    }
+
+    /**
+     * @return the number of live blocks no name resolves to.
+     * @deprecated counts only nameless blocks, which understates what a rebuild
+     *             discards; use {@link #filesLostOnRebuild()}.
+     */
+    @Deprecated
+    public int unnamedBlockCount() {
+        int unnamed = 0;
+        for (MpqFileEntry entry : entries()) {
+            if (entry.name().isEmpty()) {
+                unnamed++;
+            }
+        }
+        return unnamed;
+    }
+
+    /**
+     * The hash table backing this archive.
+     * <p>
+     * Exposed for the deprecated {@code JMpqEditor} adapter, whose public API
+     * hands this object to callers. New code should use {@link #entry(String)}
+     * and {@link #localesOf(String)} instead, which do not require knowing how
+     * the format stores its index.
+     *
+     * @return the live hash table.
+     */
+    public HashTable hashTable() {
+        return hashTable;
+    }
+
+    /**
+     * A file's bytes exactly as stored, without decryption.
+     * <p>
+     * Exposed for the deprecated adapter, which constructs legacy
+     * {@code MpqFile} objects that do their own decoding. New code should use
+     * {@link #read(MpqFileEntry)}.
+     *
+     * @param entry the file.
+     * @return exactly {@link MpqFileEntry#compressedSize()} bytes.
+     * @throws IOException if the range lies outside the archive.
+     */
+    public byte[] rawBytes(MpqFileEntry entry) throws IOException {
+        return source.bytes(header.headerOffset() + entry.filePosition(), entry.compressedSize());
+    }
+
+    /**
+     * Every block table row, live or not, in table order.
+     * <p>
+     * Exposed for the deprecated adapter. {@link #entries()} is the supported
+     * form: it skips dead rows and attaches names and locales.
+     *
+     * @return the raw rows.
+     */
+    public List<MpqFileEntry> rawBlocks() {
+        return List.of(blocks);
+    }
+
     /**
      * The bytes preceding the archive header.
      * <p>

src/main/java/org/inwc3/jmpq/MpqArchiveWriter.java

@@ -53,8 +53,24 @@ public final class MpqArchiveWriter {
     private static final int KEY_HASH_TABLE = tableKey("(hash table)");
     private static final int KEY_BLOCK_TABLE = tableKey("(block table)");
 
-    /** Internal files the writer owns and callers must not supply. */
-    private static final List<String> RESERVED = List.of("(listfile)", "(attributes)", "(signature)");
+    /**
+     * Internal files the writer generates itself, so a caller cannot supply
+     * them: doing so would put two entries under one name.
+     * <p>
+     * Only {@code (listfile)} qualifies. {@code (attributes)} and
+     * {@code (signature)} are not generated, so a caller holding their bytes is
+     * free to write them as ordinary files -- which is the only way to preserve
+     * them until attributes generation lands.
+     */
+    private static final List<String> GENERATED = List.of("(listfile)");
+
+    /**
+     * Internal files carried over from a source archive is not attempted:
+     * {@code (listfile)} is regenerated, and the other two cannot be
+     * regenerated so a copy would be stale.
+     */
+    private static final List<String> NOT_CARRIED_OVER =
+        List.of("(listfile)", "(attributes)", "(signature)");
 
     private static int tableKey(String name) {
         final MPQHashGenerator hasher = MPQHashGenerator.getFileKeyGenerator();
@@ -140,8 +156,7 @@ public static MpqArchiveWriter from(MpqArchive source, MpqWriteOptions options)
             writer.prefix = source.prefixBytes();
         }
         for (String name : source.names()) {
-            if (RESERVED.contains(name)) {
-                // Regenerated rather than copied.
+            if (NOT_CARRIED_OVER.contains(name)) {
                 continue;
             }
             // Every locale variant, not just the one a lookup resolves.
@@ -154,6 +169,15 @@ public static MpqArchiveWriter from(MpqArchive source, MpqWriteOptions options)
                     new Pending(name, locale, new Content.Existing(source, entry)));
             }
         }
+        final int dropped = source.filesLostOnRebuild();
+        if (dropped > 0) {
+            // Stated plainly, because it is data loss the caller may not
+            // expect: these blocks exist but nothing names them, so the rebuilt
+            // archive cannot contain them.
+            log.warn("{} of the {} files in {} cannot be named and will not be carried over."
+                + " Supply a list file covering them if they matter.",
+                dropped, source.blockCount(), source);
+        }
         log.debug("Writer seeded with {} files from {}", writer.pending.size(), source);
         return writer;
     }
@@ -264,10 +288,10 @@ private void requireUsableName(String name) {
         if (name == null || name.isEmpty()) {
             throw new IllegalArgumentException("A file name is required.");
         }
-        for (String reserved : RESERVED) {
-            if (reserved.equalsIgnoreCase(name)) {
+        for (String generated : GENERATED) {
+            if (generated.equalsIgnoreCase(name)) {
                 throw new IllegalArgumentException(name + " is generated by the writer and cannot"
-                    + " be supplied. Use MpqWriteOptions to control it.");
+                    + " be supplied. Use MpqWriteOptions.withListfile to control it.");
             }
         }
     }