Skip to content

verification.md

Latest commit

 

History

History
77 lines (55 loc) · 1.83 KB

verification.md

File metadata and controls

77 lines (55 loc) · 1.83 KB
Raw

File Integrity Verification

For debugging and verification purposes across all projects.

SHA256-Chunked Verification

0Mb Files

sha256sum ./FILE | xxd -r -p | base64

<= 8Mb Files

sha256sum ./FILE | xxd -r -p | sha256sum | xxd -r -p | base64

> 8Mb Files

split -b 8388608 ./FILE --filter='sha256sum' | xxd -r -p | \
  sha256sum | xxd -r -p | base64

Verify Packages

split -l 1 ~/MANIFEST.jsonl --filter="jq -cSM 'del(.physical_keys)'" | \
  tr -d '\n' | sha256sum

Note: If your JSONL manifest contains "meta": null entries, you need to convert them to "meta": {} first to match the quilt3 implementation's hashing behavior:

split -l 1 ~/MANIFEST.jsonl \
  --filter="jq -cSM 'if .meta == null then .meta = {} else . end | \
    del(.physical_keys)'" | \
  tr -d '\n' | sha256sum

Caveat: Use this meta-null form only when rows actually carry "meta": null. Applied to a header line (which has no meta), jq reads the absent .meta as null and injects meta: {}, corrupting the header hash — use the plain del(.physical_keys) form otherwise.

CRC64/NVMe Verification

CRC64-NVMe is a whole-file checksum (no chunking). The digest is 8 bytes, base64-encoded for storage.

Remote objects (AWS CLI v2.22+)

S3 stores CRC64-NVMe checksums automatically for new objects. Retrieve it with:

aws s3api head-object \
  --bucket BUCKET --key KEY \
  --checksum-mode ENABLED

Note: Requires AWS CLI v2.22+ (or v1.36+). Older versions do not support CRC64-NVMe headers.

Local files

crc-fast provides a SIMD-accelerated CLI tool. Its output is hex, so convert to base64 to match the S3 format:

cargo install crc-fast --features cli
checksum -a CRC-64/NVME -f ./FILE | xxd -r -p | base64