mergewarc

mergewarc combines small WARC-Zstd files into a larger WARC-Zstd file while preserving the compressed bytes of every input.

It is intended for dictionary-free WARC-Zstd objects. Inputs are validated as strict WARC-Zstd before publication. Dictionary frames and frames that reference an external dictionary are rejected instead of being silently combined into an invalid file.

Properties

• No decompression/recompression of stored output bytes.
• Strict WARC record, trailer, frame-content-size, checksum, and record-frame isolation validation through github.com/saveweb/unwarc.
• Algorithm-qualified checksums, byte offset, byte size, and WARC record count for every input.
• JSON Lines manifest suitable for streaming ingestion.
• Exact extraction and verification against every checksum recorded for an original input.
• CLI outputs are written through temporary files and are not published when validation fails.

CLI

go install github.com/saveweb/mergewarc/cmd/mergewarc@latest

mergewarc merge \
  -o collection.warc.zst \
  small-001.warc.zst small-002.warc.zst

# SHA-1, SHA-256, and BLAKE3 are all recorded by default. Use a comma-separated
# list to override the complete set.
mergewarc merge --checksum sha256,blake3 \
  -o collection-selected-checksums.warc.zst \
  small-001.warc.zst small-002.warc.zst

mergewarc extract \
  --manifest collection.warc.zst.manifest.jsonl \
  --name small-001.warc.zst \
  -o restored.warc.zst \
  collection.warc.zst

The merge command refuses to overwrite existing outputs. The default manifest path is the output path followed by .manifest.jsonl. If publishing the manifest fails after the merged output is published, the output is removed so the command can be retried without manual cleanup.

Library

inputs := []mergewarc.Input{
    mergewarc.FileInput("small-001.warc.zst", "./small-001.warc.zst"),
    mergewarc.FileInput("small-002.warc.zst", "./small-002.warc.zst"),
}

manifest, err := mergewarc.Merge(ctx, output, inputs, mergewarc.Options{
	Checksums: []mergewarc.ChecksumAlgorithm{
		mergewarc.ChecksumSHA1,
		mergewarc.ChecksumSHA256,
		mergewarc.ChecksumBLAKE3,
	},
})
if err != nil {
	return err
}
if err := mergewarc.WriteJSONL(manifestOutput, manifest); err != nil {
	return err
}

Input.Open is called once when its input is processed. The same stream is validated, copied, and hashed in one pass, so object-storage or HTTP sources do not need to be downloaded twice or materialized locally.

Manifest

The manifest begins with a versioned header, contains one input object per source, and ends with an output summary:

{"type":"mergewarc","version":1,"format":"warc-zstd"}
{"type":"input","name":"small-001.warc.zst","offset":0,"size":1234,"checksums":["sha1:...","sha256:...","blake3:..."],"records":4}
{"type":"output","size":1234,"checksums":["sha1:...","sha256:...","blake3:..."],"inputs":1}

Offsets and sizes address encoded bytes in the merged object. They can be used directly with local ReaderAt implementations or HTTP/object-storage range requests.

Each value in checksums uses the self-describing algorithm:lowercase-hex form. Version 1 supports blake3:<64 hex characters>, sha1:<40 hex characters>, and sha256:<64 hex characters>.

Dictionaries

This version intentionally rejects dictionaries. A WARC-Zstd file may contain only one dictionary frame at its beginning, so independently self-contained dictionary-bearing files cannot be concatenated byte-for-byte. A future merge mode would need to group inputs by dictionary content and rewrite their prefix frames; that is a different operation from the lossless merge implemented here.

License

CC0 1.0 Universal.