Skip to content

fix(upload): send a whole-file SHA1 checksum with every upload - #3531

Open
Mortimer-RR wants to merge 2 commits into
opencloud-eu:mainfrom
Mortimer-RR:fix/upload-checksum
Open

Mortimer-RR wants to merge 2 commits into
opencloud-eu:mainfrom
Mortimer-RR:fix/upload-checksum

Conversation

@Mortimer-RR

Copy link
Copy Markdown

Description

Browser uploads now carry a SHA1 of the bytes that are actually sent, so the server rejects
damaged or spliced uploads instead of storing them.

  • New Uppy pre-processor UploadChecksumPlugin (packages/web-pkg/src/services/uppy/checksum/),
    registered by UppyService for the tus and the XHR uploader. It hashes file.data in a
    web worker, reading 8 MiB Blob.slices, and sets file.meta.checksum = 'sha1 <hex>'.
    Folders, remote (companion) files and files that already have a checksum (retries) are
    skipped. If hashing fails, that file fails (upload-error) instead of being uploaded
    without a checksum; other files continue.
  • Hashing uses hash-wasm (incremental; crypto.subtle.digest can't hash incrementally).
    Compiling its WebAssembly needs 'wasm-unsafe-eval' in the CSP's script-src, which
    OpenCloud's default CSP doesn't allow, so the worker falls back to the pure JavaScript
    js-sha1 when WebAssembly can't be compiled. Same SHA1; JavaScript is about 7脳 slower.
  • checksum is added to TUS_ALLOWED_META_FIELDS (a hash of the transmitted bytes reveals
    no path or cleartext). The XHR fallback sends OC-Checksum: SHA1:<hex>, the desktop
    client's format.
  • Vault uploads: HandleUpload.applyVaultEncryption replaces file.data with the
    ciphertext before uppy.upload() runs the pre-processors, so the hash covers the
    ciphertext that is sent.
  • UploadInfo.vue shows "Calculating checksum..." for a file while it is hashed.

New dependencies: hash-wasm, js-sha1.

Related Issue

How Has This Been Tested?

  • test environment: Vitest (happy-dom, @vitest/web-worker); headless Chromium against an
    OpenCloud server with the matching server fix (460 for checksum mismatches,
    fix(tus): return 460 for checksum mismatches instead of 500聽reva#843), default CSP
  • test case 1: checksum.spec.ts: known SHA1s, slice-by-slice hashing, the meta field is set
    before the upload, ciphertext is hashed when file.data is replaced, folders skipped, an
    unreadable file fails alone, and the JavaScript fallback when WebAssembly is blocked
  • test case 2: uppyService.spec.ts: the tus Upload-Metadata fields are exactly name,
    mtime, checksum, with no path fields
  • test case 3: useUpload.spec.ts: XHR uploads send OC-Checksum
  • test case 4: UploadInfo.spec.ts: "Calculating checksum..." while hashing
  • test case 5: headless browser against the server: files from 0 B to 5 GB uploaded intact;
    an upload with a tampered checksum and the same-fingerprint resume splice were both
    rejected (460) and stored nothing
  • test case 6: hashing time with WebAssembly, Chrome 154 on Windows: 3.0 s per 1 GB, 29.7 s
    per 10 GB, UI fully responsive; Firefox 155 (Linux container): 5.1 s / 50.6 s. JavaScript
    fallback in the same browser: about 7脳 slower
  • pnpm check:types, pnpm lint and Prettier pass; 6 unit tests unrelated to this change
    fail on main as well (CreateFolderModal, ResourcePreview, conflictDialog,
    resourcesTransfer)

Types of changes

  • Bugfix
  • Enhancement (a change that doesn't break existing code or deployments)
  • Breaking change (a modification that affects current functionality)
  • Technical debt (addressing code that needs refactoring or improvements)
  • Tests (adding or improving tests)
  • Documentation (updates or additions to documentation)
  • Maintenance (like dependency updates or tooling adjustments)

Browser uploads carried no checksum, so the server could not tell a
damaged upload from a good one: overlapping or spliced chunks were
stored silently. tus-js-client can also resume a different file's
partial upload when name, type, size, mtime and endpoint all match,
splicing the two files together.

Add an Uppy pre-processor that computes the SHA1 of the bytes that are
actually sent (after vault encryption) in a web worker, reading the file
in 8 MiB slices with hash-wasm since crypto.subtle cannot hash
incrementally. The checksum goes into the tus Upload-Metadata as
`checksum: sha1 <hex>`, and into the `OC-Checksum: SHA1:<hex>` header for
plain PUT uploads. The server verifies it and rejects mismatching
uploads.

If a file cannot be hashed, that file fails instead of being uploaded
without a checksum. The upload info shows "Calculating checksum..." for
a file while it is hashed.
hash-wasm needs 'wasm-unsafe-eval' in the Content-Security-Policy's
script-src to compile its WebAssembly module. OpenCloud's default CSP
does not allow it, so computing the checksum failed and every browser
upload was rejected before it started.

The checksum worker now tries hash-wasm once and, if WebAssembly cannot
be compiled, uses the pure JavaScript js-sha1 implementation instead.
Both produce the same SHA1; the JavaScript one is about 7 times slower.
Deployments that add 'wasm-unsafe-eval' to script-src get the fast path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser uploads carry no checksum, so damaged or spliced uploads are stored silently

1 participant